Retrieving Stored HTML and Screenshots
What attachments are
Most crawl data is exposed as metrics — numbers, strings and booleans on the CrawlUrl type. Some containers
additionally capture whole files for each crawled URL: the page's HTML source, a screenshot of the rendered page,
a raw Lighthouse report. Those files are called attachments.
Attachments are exposed on CrawlUrl through two fields:
| Field | Type | Use it to |
|---|---|---|
attachmentNames | [String!] | Discover which files exist for a URL. Cheap to select, and filterable — see Filtering. |
attachments | [CrawlUrlAttachment!] | Get the file metadata, including signedLocation — the temporary link you download the file from. |
Signed download links are only produced by the query below. Report downloads and the raw parquet exports carry the attachment names at most, never the files themselves.
Before you start
Attachments only exist for a crawl if the container that produces them was enabled on the project at the time of that crawl. Stored HTML and screenshots come from the Store HTML and Store Screenshots extensions — see Extensions for how to list, link and enable them. Enabling an extension has no effect on crawls that already finished.
Storing a body and a screenshot for every URL of a large crawl produces a lot of data, and retrieving it is one request per URL. Filter down to the URLs you actually need before you start downloading, and keep the rate limits in mind.
Attachment names
Every attachment is named <container name>/<file name>, so the prefix tells you which container wrote it. The ones
Lumar's own containers produce are:
| Name | Written by | Contents |
|---|---|---|
HtmlStoring/static-body.html | Store HTML | The HTML as returned by the server, before any JavaScript ran. |
HtmlStoring/rendered-body.html | Store HTML | The DOM after rendering. Only written when the crawl rendered JavaScript. |
ScreenshotStoring/screenshot.jpeg | Store Screenshots | A screenshot of the rendered page. |
AccessibilityIssues/a11y-full-page-screenshot.png | Accessibility module | The full-page screenshot behind accessibility issue examples. |
Other containers add their own — the Site Speed module, for example, stores each URL's raw Lighthouse report as
lighthouse.json, and custom metric containers can store anything they like via
context.storeAttachment().
Treat file extensions as an implementation detail: match on the container prefix (ScreenshotStoring/) or on
contentType rather than hard-coding screenshot.jpeg.
Retrieving attachments for a URL
Query the report the URL appears in, filter crawlUrls down to that URL, and select attachments. all_pages is the
report that contains every crawled page; any report the URL appears in works.
query GetCrawlUrlAttachments($crawlId: ObjectID!, $reportTemplateCode: String!, $url: String!) {
getReportStat(input: { crawlId: $crawlId, reportTemplateCode: $reportTemplateCode }) {
crawlUrls(filter: { url: { eq: $url } }, first: 1) {
nodes {
url
attachments {
name
contentType
sizeBytes
expiresAt
signedLocation
}
}
}
}
}
Each entry of attachments has:
| Field | Type | Description |
|---|---|---|
name | String! | <container name>/<file name>, as listed above. |
contentType | String! | MIME type of the stored file, e.g. text/html or image/jpeg. |
sizeBytes | Int! | Size of the stored file. |
expiresAt | DateTime! | When the stored file itself is deleted. See Retention. |
signedLocation | String! | A pre-signed link the file can be downloaded from. Generated per request, valid for 24 hours. |
Filtering by url requires the exact crawled URL — protocol, trailing slash, query string and casing all have to
match. If you already have a urlDigest from a report row, filter on that instead: it is the stable identity of a URL
within a crawl and avoids any normalisation surprises.
query GetCrawlUrlAttachmentsByDigest($crawlId: ObjectID!, $urlDigest: String!) {
getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
crawlUrls(filter: { urlDigest: { eq: $urlDigest } }, first: 1) {
nodes {
url
attachments {
name
contentType
sizeBytes
expiresAt
signedLocation
}
}
}
}
}
Downloading the file
signedLocation is a pre-signed storage link, not a Lumar API endpoint. Send none of your Lumar
authentication headers with it — no x-auth-token, no x-api-key — the credentials it needs are
already in the query string. If you download through the same HTTP client you use for the API, make sure its default
headers are not applied here:
curl -o rendered-body.html "https://ds-storage-s3-attachments-...s3.amazonaws.com/attachments/...?X-Amz-Signature=..."
Because the link carries its own credentials, treat it as a secret: do not log it or paste it into shared tickets. It stops working 24 hours after the API returned it, so generate links as you download rather than collecting them all up front.
Check the HTTP status before you read the body. A rejected download — expired signature, deleted file — answers with an error status and a short XML error document rather than an empty response, so code that reads the body unconditionally will treat that document as the page's content.
Finding the URLs that have attachments
attachmentNames is filterable, which makes it the cheap way to narrow a crawl down to the pages worth downloading.
arrayContains matches one entry of the array exactly; arrayContainsLike matches an entry that contains the value,
case-insensitively. Neither takes wildcards — * and ? are matched literally — so select a whole container's output
with the name prefix on its own: "HtmlStoring/" for stored bodies, "ScreenshotStoring/" for screenshots.
query ListUrlsWithAttachments($crawlId: ObjectID!, $namePrefix: String!, $after: String) {
getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
crawlUrls(
filter: { attachmentNames: { arrayContainsLike: $namePrefix } }
first: 100
after: $after
) {
nodes {
url
urlDigest
attachmentNames
}
pageInfo {
hasNextPage
endCursor
}
totalCount
}
}
}
namePrefix is what decides which attachment you get back, so set it to the container you actually want — the variables
above ask for stored HTML.
Combine it with any other CrawlUrl filter to scope the work, for example only indexable product pages:
filter: {
attachmentNames: { arrayContainsLike: "HtmlStoring/" }
url: { beginsWith: "https://www.example.com/products/" }
indexable: { eq: true }
}
When you want one specific file rather than everything a container wrote, match it exactly:
filter: { attachmentNames: { arrayContains: "HtmlStoring/rendered-body.html" } }
Page through the result 100 nodes at a time using pageInfo.endCursor — see Pagination.
Comparing two crawls
Content and screenshot diffing is the usual reason to pull attachments out. The shape of it is:
- List the URLs that carry the attachment you are comparing, in each crawl, and intersect the two sets of
urlvalues. Use the discovery query above with the prefix for that attachment —"HtmlStoring/"for bodies,"ScreenshotStoring/"for screenshots. Leaving it on"HtmlStoring/"while comparing screenshots returns an empty set on a project that only enabled Store Screenshots. - For each URL in the intersection, fetch its attachments in both crawls.
- Download the two
signedLocations and diff the bodies (or the images) locally.
The example below compares stored HTML. For screenshots, swap the discovery prefix as above and select
ScreenshotStoring/screenshot. by prefix instead of the two HtmlStoring/ names — then compare
await download.arrayBuffer() rather than .text().
Fetch the attachment metadata immediately before you download it, so the 24-hour link lifetime is never the thing that breaks a long-running job:
const API = "https://api.lumar.io/graphql";
async function graphql<T>(query: string, variables: Record<string, unknown>): Promise<T> {
const response = await fetch(API, {
method: "POST",
headers: {
"Content-Type": "application/json",
"x-auth-token": process.env.LUMAR_TOKEN!,
},
body: JSON.stringify({ query, variables }),
});
const { data, errors } = await response.json();
if (errors) throw new Error(JSON.stringify(errors));
return data as T;
}
const ATTACHMENTS = `
query GetCrawlUrlAttachments($crawlId: ObjectID!, $url: String!) {
getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
crawlUrls(filter: { url: { eq: $url } }, first: 1) {
nodes {
attachments {
name
expiresAt
signedLocation
}
}
}
}
}`;
async function fetchStoredHtml(crawlId: string, url: string): Promise<string | undefined> {
const data = await graphql<any>(ATTACHMENTS, { crawlId, url });
const attachments = data.getReportStat?.crawlUrls?.nodes[0]?.attachments ?? [];
// Prefer the rendered DOM, fall back to the pre-JavaScript body.
const attachment =
attachments.find((a: any) => a.name === "HtmlStoring/rendered-body.html") ??
attachments.find((a: any) => a.name === "HtmlStoring/static-body.html");
if (!attachment) return undefined;
if (new Date(attachment.expiresAt) < new Date()) return undefined; // deleted from storage
const download = await fetch(attachment.signedLocation);
// Storage answers a rejected download with an XML error document and a 2xx-less status.
// `fetch` resolves either way, so a body read without this check diffs the error as content.
if (!download.ok) {
throw new Error(`Download failed for ${url} (${download.status} ${download.statusText})`);
}
return await download.text();
}
const [before, after] = await Promise.all([
fetchStoredHtml("TjAwNUNyYXdsMTc5MDQ3NQ", "https://www.example.com/products/"),
fetchStoredHtml("TjAwNUNyYXdsMTc5MTU0Ng", "https://www.example.com/products/"),
]);
Both bodies are raw HTML, so a plain text diff will report every whitespace and ordering change. For a content diff,
parse and extract the parts you care about first — Lumar already exposes many of them as metrics
(pageTitle, description, h1Tag, wordCount, contentSize), and comparing those across two crawls with
Get URL Data is far cheaper than downloading every body.
Retention
Stored files are deleted some time after the crawl — currently around 72 days. expiresAt is the authoritative value:
check it before downloading, because a URL keeps listing the attachment after that point and signedLocation will
fail once the underlying file is gone. Deleting a crawl deletes its attachments with it.
expiresAt and the 24-hour lifetime of signedLocation are two different clocks: the first is how long Lumar keeps
the file, the second is how long the link you were just handed works for.
Attachments are read through a report, so they follow the crawl's availability. Crawls older than 60 days are archived automatically and have to be unarchived before their rows — and therefore their attachments — can be queried again.