Disallowed URLs (Uncrawled)

All URLs which were disallowed in the robots.txt file on the live site, or from a custom robots.txt file applied in Advanced Settings.

These URLs were not crawled by Lumar. Disallowed URLs can be crawled by enabling the 'Check disallowed links' setting in Advanced Settings > Scope > Link Validation, and will appear in the Disallowed Pages report.

Priority: None

Impact: Neutral

How to fetch the data for this report template

You will need to run a crawl for report template to generate report. When report has been generated and you have crawl id you can fetch data for the report using the following query:

Query

query GetReportStatForCrawl(
    $crawlId: ObjectID!
    $reportTemplateCode: String!
    $after: String
   ) {
      getReportStat(
        input: {crawlId: $crawlId, reportTemplateCode: $reportTemplateCode}
      ) {
        crawlUncrawledUrls(after: $after, reportType: Basic) {
          nodes {
            url
            foundAtUrl
            foundAtSitemap
            level
            restrictedReason
            robotsTxtRuleMatch
            foundInWebCrawl
            foundInGoogleAnalytics
            foundInGoogleSearchConsole
            foundInBacklinks
            foundInList
            foundInLogSummary
            foundInSitemap
          }
          totalCount
          pageInfo {
            endCursor
            hasNextPage
          }
        }
     }
   }

Variables

{"crawlId":"TjAwNUNyYXdsNDAwMA","reportTemplateCode":"disallowed_urls"}

cURL

curl -X POST -H "Content-Type: application/json" -H "apollographql-client-name: docs-example-client" -H "apollographql-client-version: 1.0.0" -H "x-auth-token: YOUR_API_SESSION_TOKEN" --data '{"query":"query GetReportStatForCrawl( $crawlId: ObjectID! $reportTemplateCode: String! $after: String ) { getReportStat( input: {crawlId: $crawlId, reportTemplateCode: $reportTemplateCode} ) { crawlUncrawledUrls(after: $after, reportType: Basic) { nodes { url foundAtUrl foundAtSitemap level restrictedReason robotsTxtRuleMatch foundInWebCrawl foundInGoogleAnalytics foundInGoogleSearchConsole foundInBacklinks foundInList foundInLogSummary foundInSitemap } totalCount pageInfo { endCursor hasNextPage } } } }","variables":{"crawlId":"TjAwNUNyYXdsNDAwMA","reportTemplateCode":"disallowed_urls"}}' https://api.lumar.io/graphql

Try in explorer