Tutorial: Run a Crawl and Get Results
This tutorial walks through the complete workflow of authenticating, creating a project, running a crawl, waiting for it to finish, and fetching the results.
Step 1: Verify authentication
Before doing anything, confirm your token is valid by querying your user details and available accounts.
query VerifyAuth {
me {
id
username
accounts(first: 1) {
nodes {
id
name
}
}
}
}
If this returns an error, see Authentication to obtain a valid token.
Step 2: Create a project (if needed)
If you do not already have a project, create one. See How to Create a Project for the full guide. Here is a quick example:
mutation CreateSEOProject($input: CreateProjectInput!) {
createSEOProject(input: $input) {
project {
...ProjectDetails
}
}
}
fragment ProjectDetails on Project {
id
name
primaryDomain
# ...other fields you want to retrieve
}
Step 3: Run a crawl
Trigger a crawl for your project using the runCrawlForProject mutation:
mutation RunCrawl($input: RunCrawlForProjectInput!) {
runCrawlForProject(input: $input) {
crawl {
id
statusEnum
createdAt
}
}
}
The crawl starts in Queued status and progresses through Crawling, Finalizing, and finally Finished.
Optional: Stop and finalize the crawl early
Use cancelCrawling to stop fetching new pages and finalize the data collected so far. The crawl cannot be resumed after this action, but it remains in the project history with its partial results.
The crawl must be in Crawling or Paused status. A Queued crawl cannot be stopped yet, while a crawl in Finalizing is already completing and does not need this mutation.
This mutation requires the Editor role for the crawl's account. Requests authenticated with a Viewer role return an INSUFFICIENT_PERMISSIONS error.
mutation StopCrawl($input: CancelCrawlingInput!) {
cancelCrawling(input: $input) {
crawl {
id
statusEnum
canceledAt
}
}
}
Step 4: Poll for crawl completion
Query the crawl status periodically until it reaches Finished. A polling interval of 30--60 seconds is reasonable for most crawls.
query PollCrawlStatus($crawlId: ObjectID!) {
getCrawl(id: $crawlId) {
id
status
createdAt
finishedAt
crawlUrlsTotal
}
}
async function waitForCrawl(crawlId: string): Promise<void> {
while (true) {
const result = await executeQuery(POLL_QUERY, { crawlId });
const status = result.data.getCrawl.status;
if (status === "Finished") {
console.log("Crawl finished!");
return;
}
if (status === "Archived" || status === "Paused") {
throw new Error(`Crawl ended with status: ${status}`);
}
console.log(`Crawl status: ${status}. Checking again in 30s...`);
await new Promise(resolve => setTimeout(resolve, 30000));
}
}
Step 5: Fetch the results
Once the crawl is finished, query the report data:
query GetCrawlResults($crawlId: ObjectID!) {
getReportStat(
input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }
) {
basic
crawlUrls(first: 5) {
nodes {
url
httpStatusCode
pageTitle
}
totalCount
}
}
}
You can request different report templates by changing the reportTemplateCode parameter. See Report Templates Overview for how to discover available templates.
Next steps
- Export Crawl Data -- learn how to bulk-export results.
- Track SEO Health -- monitor health scores over time.
- Setup Automated Monitoring -- automate crawls with schedules and alerts.