Skip to main content

URL-based task coverage

A URL-based task (type: TaggedURLs) follows a fixed set of URLs. When you create one with createReportStatLegacyTask, Lumar records the URLs the report holds for that crawl, and later crawls tag only those URLs as new, resolved or unresolved.

The URL limit​

A URL-based task records at most taggedTaskReportDownloadRowsLimit URLs, a per-account setting you can read on the Account object. It is 100,000 unless your account has a different limit. When the report holds more URLs than that, the task tracks only that many of them.

The limit that applies is the one in force when the task was created. Changing the account's limit later does not change what an existing task tracks.

Reading a task's coverage​

Query the task's urlSnapshot field:

Operation: query getLegacyTaskUrlSnapshot($id: ObjectID!) { getLegacyTask(id: $id) { id type identified remaining urlSnapshot { status sourceUrlCount urlLimit trackedUrlCount truncated } } }Variables: { "id": "TjAxMExlZ2FjeVRhc2s1NzI2OQ" }Response Example: { "data": { "getLegacyTask": { "id": "TjAxMExlZ2FjeVRhc2s1NzI2OQ", "type": "TaggedURLs", "identified": 817604, "remaining": 817604, "urlSnapshot": { "status": "Generated", "sourceUrlCount": 817604, "urlLimit": 100000, "trackedUrlCount": 100000, "truncated": true } } } }
getLegacyTaskUrlSnapshotTry in Explorer
GraphQL
query getLegacyTaskUrlSnapshot($id: ObjectID!) {
getLegacyTask(id: $id) {
id
type
identified
remaining
urlSnapshot {
status
sourceUrlCount
urlLimit
trackedUrlCount
truncated
}
}
}
FieldMeaning
sourceUrlCountURLs the report held when the task was created. It does not change with later crawls.
urlLimitThe account's URL limit when the task was created.
trackedUrlCountURLs the task tracks: urlLimit when the report held more URLs than that, otherwise sourceUrlCount.
truncatedtrue when the report held more URLs than urlLimit, so the task tracks only some of them.
statusProgress of recording the task's URLs: Pending, Generated or Unavailable.

In the example, the task identified 817,604 URLs and tracks 100,000 of them.

identified and remaining keep their meaning: they count the URLs the task found, and they are updated after each crawl. urlSnapshot counts the URLs the task tracks, so identified can be larger than trackedUrlCount.

While the URLs are being recorded​

The URLs are recorded shortly after the task is created. Until then status is Pending and trackedUrlCount is null, but sourceUrlCount, urlLimit and truncated are already set, so you can tell a user up front that the task will track only some of its URLs. If recording does not finish, status stays Pending.

status is Unavailable when the recorded URLs no longer exist, for example because they were deleted.

Older tasks and other task types​

Tasks created before Lumar recorded coverage return null for sourceUrlCount, urlLimit, trackedUrlCount and truncated, because the limit in force when they were created is not known. Do not substitute the account's current limit: it may differ from the one the task was created with.

urlSnapshot is null for tasks that are not URL-based, because they re-evaluate the report filter on every crawl instead of tracking a fixed set of URLs.