Disallowed Page

TRUE if the page is disallowed based on the robots.txt file and user-agent value used for the crawl

Code: disallowedPage