# Retrieving Stored HTML and Screenshots

https://api-docs.lumar.io/docs/graphql/get-attachments

## What attachments are

Most crawl data is exposed as metrics — numbers, strings and booleans on the `CrawlUrl` type. Some containers
additionally capture whole **files** for each crawled URL: the page's HTML source, a screenshot of the rendered page,
a raw Lighthouse report. Those files are called **attachments**.

Attachments are exposed on `CrawlUrl` through two fields:

| Field             | Type                    | Use it to                                                                                           |
| ----------------- | ----------------------- | --------------------------------------------------------------------------------------------------- |
| `attachmentNames` | `[String!]`             | Discover which files exist for a URL. Cheap to select, and filterable — see [Filtering](filtering). |
| `attachments`     | `[CrawlUrlAttachment!]` | Get the file metadata, including `signedLocation` — the temporary link you download the file from.  |

Signed download links are only produced by the query below. Report downloads and the
[raw parquet exports](get-raw-data) carry the attachment names at most, never the files themselves.

## Before you start

Attachments only exist for a crawl if the container that produces them was enabled on the project **at the time of that
crawl**. Stored HTML and screenshots come from the **Store HTML** and **Store Screenshots** extensions — see
[Extensions](/docs/extensions.md) for how to list, link and enable them. Enabling an extension has no effect on crawls that
already finished.

Storing a body and a screenshot for every URL of a large crawl produces a lot of data, and retrieving it is one request
per URL. Filter down to the URLs you actually need before you start downloading, and keep the
[rate limits](rate-limits) in mind.

## Attachment names

Every attachment is named `<container name>/<file name>`, so the prefix tells you which container wrote it. The ones
Lumar's own containers produce are:

| Name                                                | Written by           | Contents                                                                  |
| --------------------------------------------------- | -------------------- | ------------------------------------------------------------------------- |
| `HtmlStoring/static-body.html`                      | Store HTML           | The HTML as returned by the server, before any JavaScript ran.            |
| `HtmlStoring/rendered-body.html`                    | Store HTML           | The DOM after rendering. Only written when the crawl rendered JavaScript. |
| `ScreenshotStoring/screenshot.jpeg`                 | Store Screenshots    | A screenshot of the rendered page.                                        |
| `AccessibilityIssues/a11y-full-page-screenshot.png` | Accessibility module | The full-page screenshot behind accessibility issue examples.             |

Other containers add their own — the Site Speed module, for example, stores each URL's raw Lighthouse report as
`lighthouse.json`, and [custom metric containers](/docs/custom-metrics.md) can store anything they like via
`context.storeAttachment()`.

Treat file extensions as an implementation detail: match on the container prefix (`ScreenshotStoring/`) or on
`contentType` rather than hard-coding `screenshot.jpeg`.

## Retrieving attachments for a URL

Query the report the URL appears in, filter `crawlUrls` down to that URL, and select `attachments`. `all_pages` is the
report that contains every crawled page; any report the URL appears in works.

```graphql
query GetCrawlUrlAttachments($crawlId: ObjectID!, $reportTemplateCode: String!, $url: String!) {
  getReportStat(input: { crawlId: $crawlId, reportTemplateCode: $reportTemplateCode }) {
    crawlUrls(filter: { url: { eq: $url } }, first: 1) {
      nodes {
        url
        attachments {
          name
          contentType
          sizeBytes
          expiresAt
          signedLocation
        }
      }
    }
  }
}
```

**Variables:**
```json
{
  "crawlId": "TjAwNUNyYXdsMTc5MDQ3NQ",
  "reportTemplateCode": "all_pages",
  "url": "https://www.example.com/products/"
}
```

**Response:**
```json
{
  "data": {
    "getReportStat": {
      "crawlUrls": {
        "nodes": [
          {
            "url": "https://www.example.com/products/",
            "attachments": [
              {
                "name": "HtmlStoring/static-body.html",
                "contentType": "text/html",
                "sizeBytes": 142387,
                "expiresAt": "2026-10-21T09:14:22.000Z",
                "signedLocation": "https://ds-storage-s3-attachments-resources-prod-use1.s3.us-east-1.amazonaws.com/attachments/1790475/...?X-Amz-Signature=..."
              },
              {
                "name": "HtmlStoring/rendered-body.html",
                "contentType": "text/html",
                "sizeBytes": 210554,
                "expiresAt": "2026-10-21T09:14:22.000Z",
                "signedLocation": "https://ds-storage-s3-attachments-resources-prod-use1.s3.us-east-1.amazonaws.com/attachments/1790475/...?X-Amz-Signature=..."
              },
              {
                "name": "ScreenshotStoring/screenshot.jpeg",
                "contentType": "image/jpeg",
                "sizeBytes": 88213,
                "expiresAt": "2026-10-21T09:14:22.000Z",
                "signedLocation": "https://ds-storage-s3-attachments-resources-prod-use1.s3.us-east-1.amazonaws.com/attachments/1790475/...?X-Amz-Signature=..."
              }
            ]
          }
        ]
      }
    }
  }
}
```

Each entry of `attachments` has:

| Field            | Type        | Description                                                                                       |
| ---------------- | ----------- | ------------------------------------------------------------------------------------------------- |
| `name`           | `String!`   | `<container name>/<file name>`, as listed above.                                                  |
| `contentType`    | `String!`   | MIME type of the stored file, e.g. `text/html` or `image/jpeg`.                                   |
| `sizeBytes`      | `Int!`      | Size of the stored file.                                                                          |
| `expiresAt`      | `DateTime!` | When the stored file itself is deleted. See [Retention](#retention).                              |
| `signedLocation` | `String!`   | A pre-signed link the file can be downloaded from. Generated per request, **valid for 24 hours**. |

Filtering by `url` requires the exact crawled URL — protocol, trailing slash, query string and casing all have to
match. If you already have a `urlDigest` from a report row, filter on that instead: it is the stable identity of a URL
within a crawl and avoids any normalisation surprises.

```graphql
query GetCrawlUrlAttachmentsByDigest($crawlId: ObjectID!, $urlDigest: String!) {
  getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
    crawlUrls(filter: { urlDigest: { eq: $urlDigest } }, first: 1) {
      nodes {
        url
        attachments {
          name
          contentType
          sizeBytes
          expiresAt
          signedLocation
        }
      }
    }
  }
}
```

**Variables:**
```json
{
  "crawlId": "TjAwNUNyYXdsMTc5MDQ3NQ",
  "urlDigest": "0f2a1c7e9b4d5a6380f1c2d3e4b5a697"
}
```

## Downloading the file

`signedLocation` is a pre-signed storage link, not a Lumar API endpoint. Send **none** of your Lumar
[authentication](authentication) headers with it — no `x-auth-token`, no `x-api-key` — the credentials it needs are
already in the query string. If you download through the same HTTP client you use for the API, make sure its default
headers are not applied here:

```bash
curl -o rendered-body.html "https://ds-storage-s3-attachments-...s3.amazonaws.com/attachments/...?X-Amz-Signature=..."
```

Because the link carries its own credentials, treat it as a secret: do not log it or paste it into shared tickets. It
stops working 24 hours after the API returned it, so generate links as you download rather than collecting them all up
front.

Check the HTTP status before you read the body. A rejected download — expired signature, deleted file — answers with an
error status and a short XML error document rather than an empty response, so code that reads the body unconditionally
will treat that document as the page's content.

## Finding the URLs that have attachments

`attachmentNames` is filterable, which makes it the cheap way to narrow a crawl down to the pages worth downloading.
`arrayContains` matches one entry of the array exactly; `arrayContainsLike` matches an entry that _contains_ the value,
case-insensitively. Neither takes wildcards — `*` and `?` are matched literally — so select a whole container's output
with the name prefix on its own: `"HtmlStoring/"` for stored bodies, `"ScreenshotStoring/"` for screenshots.

```graphql
query ListUrlsWithAttachments($crawlId: ObjectID!, $namePrefix: String!, $after: String) {
  getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
    crawlUrls(
      filter: { attachmentNames: { arrayContainsLike: $namePrefix } }
      first: 100
      after: $after
    ) {
      nodes {
        url
        urlDigest
        attachmentNames
      }
      pageInfo {
        hasNextPage
        endCursor
      }
      totalCount
    }
  }
}
```

**Variables:**
```json
{
  "crawlId": "TjAwNUNyYXdsMTc5MDQ3NQ",
  "namePrefix": "HtmlStoring/",
  "after": null
}
```

**Response:**
```json
{
  "data": {
    "getReportStat": {
      "crawlUrls": {
        "nodes": [
          {
            "url": "https://www.example.com/products/",
            "urlDigest": "0f2a1c7e9b4d5a6380f1c2d3e4b5a697",
            "attachmentNames": [
              "HtmlStoring/static-body.html",
              "HtmlStoring/rendered-body.html",
              "ScreenshotStoring/screenshot.jpeg"
            ]
          },
          {
            "url": "https://www.example.com/products/shoes/",
            "urlDigest": "7d3b9e0a1f2c4658a9b0c1d2e3f40512",
            "attachmentNames": ["HtmlStoring/static-body.html"]
          }
        ],
        "pageInfo": {
          "hasNextPage": true,
          "endCursor": "MTAw"
        },
        "totalCount": 2186
      }
    }
  }
}
```

`namePrefix` is what decides which attachment you get back, so set it to the container you actually want — the variables
above ask for stored HTML.

Combine it with any other `CrawlUrl` filter to scope the work, for example only indexable product pages:

```graphql
filter: {
  attachmentNames: { arrayContainsLike: "HtmlStoring/" }
  url: { beginsWith: "https://www.example.com/products/" }
  indexable: { eq: true }
}
```

When you want one specific file rather than everything a container wrote, match it exactly:

```graphql
filter: { attachmentNames: { arrayContains: "HtmlStoring/rendered-body.html" } }
```

Page through the result 100 nodes at a time using `pageInfo.endCursor` — see [Pagination](pagination).

## Comparing two crawls

Content and screenshot diffing is the usual reason to pull attachments out. The shape of it is:

1. List the URLs that carry the attachment you are comparing, in each crawl, and intersect the two sets of `url` values.
   Use the discovery query above with the prefix for that attachment — `"HtmlStoring/"` for bodies,
   `"ScreenshotStoring/"` for screenshots. Leaving it on `"HtmlStoring/"` while comparing screenshots returns an empty
   set on a project that only enabled Store Screenshots.
2. For each URL in the intersection, fetch its attachments in both crawls.
3. Download the two `signedLocation`s and diff the bodies (or the images) locally.

The example below compares stored HTML. For screenshots, swap the discovery prefix as above and select
`ScreenshotStoring/screenshot.` by prefix instead of the two `HtmlStoring/` names — then compare
`await download.arrayBuffer()` rather than `.text()`.

Fetch the attachment metadata immediately before you download it, so the 24-hour link lifetime is never the thing that
breaks a long-running job:

```ts
const API = "https://api.lumar.io/graphql";

async function graphql<T>(query: string, variables: Record<string, unknown>): Promise<T> {
  const response = await fetch(API, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-auth-token": process.env.LUMAR_TOKEN!,
    },
    body: JSON.stringify({ query, variables }),
  });
  const { data, errors } = await response.json();
  if (errors) throw new Error(JSON.stringify(errors));
  return data as T;
}

const ATTACHMENTS = `
  query GetCrawlUrlAttachments($crawlId: ObjectID!, $url: String!) {
    getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
      crawlUrls(filter: { url: { eq: $url } }, first: 1) {
        nodes {
          attachments {
            name
            expiresAt
            signedLocation
          }
        }
      }
    }
  }`;

async function fetchStoredHtml(crawlId: string, url: string): Promise<string | undefined> {
  const data = await graphql<any>(ATTACHMENTS, { crawlId, url });
  const attachments = data.getReportStat?.crawlUrls?.nodes[0]?.attachments ?? [];

  // Prefer the rendered DOM, fall back to the pre-JavaScript body.
  const attachment =
    attachments.find((a: any) => a.name === "HtmlStoring/rendered-body.html") ??
    attachments.find((a: any) => a.name === "HtmlStoring/static-body.html");

  if (!attachment) return undefined;
  if (new Date(attachment.expiresAt) < new Date()) return undefined; // deleted from storage

  const download = await fetch(attachment.signedLocation);
  // Storage answers a rejected download with an XML error document and a 2xx-less status.
  // `fetch` resolves either way, so a body read without this check diffs the error as content.
  if (!download.ok) {
    throw new Error(`Download failed for ${url} (${download.status} ${download.statusText})`);
  }

  return await download.text();
}

const [before, after] = await Promise.all([
  fetchStoredHtml("TjAwNUNyYXdsMTc5MDQ3NQ", "https://www.example.com/products/"),
  fetchStoredHtml("TjAwNUNyYXdsMTc5MTU0Ng", "https://www.example.com/products/"),
]);
```

Both bodies are raw HTML, so a plain text diff will report every whitespace and ordering change. For a content diff,
parse and extract the parts you care about first — Lumar already exposes many of them as metrics
(`pageTitle`, `description`, `h1Tag`, `wordCount`, `contentSize`), and comparing those across two crawls with
[Get URL Data](get-url-data) is far cheaper than downloading every body.

## Retention

Stored files are deleted some time after the crawl — currently around 72 days. `expiresAt` is the authoritative value:
check it before downloading, because a URL keeps listing the attachment after that point and `signedLocation` will
fail once the underlying file is gone. Deleting a crawl deletes its attachments with it.

`expiresAt` and the 24-hour lifetime of `signedLocation` are two different clocks: the first is how long Lumar keeps
the file, the second is how long the link you were just handed works for.

Attachments are read through a report, so they follow the crawl's availability. Crawls older than 60 days are
[archived automatically](crawl-archiving) and have to be unarchived before their rows — and therefore their
attachments — can be queried again.