Get URL Data
Using the correct query
There are two ways to retrieve raw URL (/link/sitemap/etc) data from Lumar:
- This page describes how to retrieve defined metrics for URLs in the crawl. This query can be filtered, sorted, etc. but requires you to paginate URLs 100 at a time. This is perfect for getting a sample of the available data, but is not well suited to getting all data for a crawl.
- The Download Raw Data query allows you to download all data from a datasource in a single request, however this cannot be filtered or sorted. This is the most efficient way to access all data.
Neither returns the files a crawl captured for a URL — its stored HTML body or screenshot. Those are retrieved separately, see Stored HTML & Screenshots.
Using the getReportStats query to access Crawl URL data
The sample query below will return 5 properties (fetchTime, pageTitle, responsive, url, wordCount) from the crawled URL but hundreds are available - for the comprehensive list, inspect type CrawlUrl.
GetUrlDataTry in Explorer
GraphQL
query GetUrlData($crawlId: ObjectID!) {
getReportStat(input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }) {
crawlUrls(reportType: Basic, first: 3) {
nodes {
fetchTime
pageTitle
responsive
url
wordCount
}
totalCount
}
}
}