URL-based task coverage
A URL-based task (type: TaggedURLs) follows a fixed set of URLs. When you create one with
createReportStatLegacyTask, Lumar records the URLs the report holds for that crawl, and later crawls tag only those
URLs as new, resolved or unresolved.
The URL limit
A URL-based task records at most taggedTaskReportDownloadRowsLimit URLs, a per-account setting you can read on the
Account object. It is 100,000 unless your account has a different limit. When the report holds more URLs than that,
the task tracks only that many of them.
The limit that applies is the one in force when the task was created. Changing the account's limit later does not change what an existing task tracks.
Reading a task's coverage
Query the task's urlSnapshot field:
query getLegacyTaskUrlSnapshot($id: ObjectID!) {
getLegacyTask(id: $id) {
id
type
identified
remaining
urlSnapshot {
status
sourceUrlCount
urlLimit
trackedUrlCount
truncated
}
}
}
| Field | Meaning |
|---|---|
sourceUrlCount | URLs the report held when the task was created. It does not change with later crawls. |
urlLimit | The account's URL limit when the task was created. |
trackedUrlCount | URLs the task tracks: urlLimit when the report held more URLs than that, otherwise sourceUrlCount. |
truncated | true when the report held more URLs than urlLimit, so the task tracks only some of them. |
status | Progress of recording the task's URLs: Pending, Generated or Unavailable. |
In the example, the task identified 817,604 URLs and tracks 100,000 of them.
identified and remaining keep their meaning: they count the URLs the task found, and they are updated after each
crawl. urlSnapshot counts the URLs the task tracks, so identified can be larger than trackedUrlCount.
While the URLs are being recorded
The URLs are recorded shortly after the task is created. Until then status is Pending and trackedUrlCount is
null, but sourceUrlCount, urlLimit and truncated are already set, so you can tell a user up front that the task
will track only some of its URLs. If recording does not finish, status stays Pending.
status is Unavailable when the recorded URLs no longer exist, for example because they were deleted.
Older tasks and other task types
Tasks created before Lumar recorded coverage return null for sourceUrlCount, urlLimit, trackedUrlCount and
truncated, because the limit in force when they were created is not known. Do not substitute the account's current
limit: it may differ from the one the task was created with.
urlSnapshot is null for tasks that are not URL-based, because they re-evaluate the report filter on every crawl
instead of tracking a fixed set of URLs.