Skip to main content

Tutorial: Run a Crawl and Get Results

This tutorial walks through the complete workflow of authenticating, creating a project, running a crawl, waiting for it to finish, and fetching the results.

Step 1: Verify authentication​

Before doing anything, confirm your token is valid by querying your user details and available accounts.

query VerifyAuth {
  me {
    id
    username
    accounts(first: 1) {
      nodes {
        id
        name
      }
    }
  }
}

Try in explorer

If this returns an error, see Authentication to obtain a valid token.

Step 2: Create a project (if needed)​

If you do not already have a project, create one. See How to Create a Project for the full guide. Here is a quick example:

mutation CreateSEOProject($input: CreateProjectInput!) {
  createSEOProject(input: $input) {
    project {
      ...ProjectDetails
    }
  }
}

fragment ProjectDetails on Project {
  id
  name
  primaryDomain
  # ...other fields you want to retrieve
}

Try in explorer

Step 3: Run a crawl​

Trigger a crawl for your project using the runCrawlForProject mutation:

mutation RunCrawl($input: RunCrawlForProjectInput!) {
  runCrawlForProject(input: $input) {
    crawl {
      id
      status
      createdAt
    }
  }
}

Try in explorer

The crawl starts in Queued status and progresses through Crawling, Finalizing, and finally Finished.

Step 4: Poll for crawl completion​

Query the crawl status periodically until it reaches Finished. A polling interval of 30--60 seconds is reasonable for most crawls.

query PollCrawlStatus($crawlId: ObjectID!) {
  getCrawl(id: $crawlId) {
    id
    status
    createdAt
    finishedAt
    crawlUrlsTotal
  }
}

Try in explorer

async function waitForCrawl(crawlId: string): Promise<void> {
  while (true) {
    const result = await executeQuery(POLL_QUERY, { crawlId });
    const status = result.data.getCrawl.status;

    if (status === "Finished") {
      console.log("Crawl finished!");
      return;
    }

    if (status === "Archived" || status === "Paused") {
      throw new Error(`Crawl ended with status: ${status}`);
    }

    console.log(`Crawl status: ${status}. Checking again in 30s...`);
    await new Promise(resolve => setTimeout(resolve, 30000));
  }
}

Step 5: Fetch the results​

Once the crawl is finished, query the report data:

query GetCrawlResults($crawlId: ObjectID!) {
  getReportStat(
    input: { crawlId: $crawlId, reportTemplateCode: "all_pages" }
  ) {
    basic
    crawlUrls(first: 5) {
      nodes {
        url
        httpStatusCode
        pageTitle
      }
      totalCount
    }
  }
}

Try in explorer

You can request different report templates by changing the reportTemplateCode parameter. See Report Templates Overview for how to discover available templates.

Next steps​