Skip to main content

AI Bot Not Blocked

Priority: Critical

Impact: Positive

What issues it may cause​

Greater likelihood of inclusion in AI-generated answers and summaries, improving brand presence within AI discovery channels.

How do you fix it​

Maintain access for AI crawlers where appropriate. Periodically validate robots.txt and meta tags, and monitor logs to ensure AI user-agents can reach key content.

What is the positive impact​

Positive signal: content is eligible to be processed, referenced, and cited in AI and generative search experiences.

How to fetch the data for this report template​

You will need to run a crawl for report template to generate report. When report has been generated and you have crawl id you can fetch data for the report using the following query:

query GetReportStatForCrawl(
    $crawlId: ObjectID!
    $reportTemplateCode: String!
    $after: String
   ) {
      getReportStat(
        input: {crawlId: $crawlId, reportTemplateCode: $reportTemplateCode}
      ) {
        crawlUrls(after: $after, reportType: Basic) {
          nodes {
            pageTitle
            url
            description
            foundAtUrl
            deeprank
            level
            aiBotBlocked
            httpStatusCode
            indexable
            foundInGoogleAnalytics
            foundInGoogleSearchConsole
            foundInBacklinks
            foundInList
            foundInLogSummary
            foundInWebCrawl
            foundInSitemap
          }
          totalCount
          pageInfo {
            endCursor
            hasNextPage
          }
        }
     }
   }

Try in explorer