Google Dorking

Prompts and answers

• Name the key term of what a "Crawler" is used to do
  Index
• What is the name of the technique that "Search Engines" use to retrieve this information about websites?
  Crawling
• What is an example of the type of contents that could be gathered from a website?
  Keywords
• Where would "robots.txt" be located on the domain "ablog.com"
  ablog.com/robots.txt
• If a website was to have a sitemap, where would that be located?
  /sitemap.xml
• How would we only allow "Bingbot" to index the website?
  User-agent: Bingbot
• How would we prevent a "Crawler" from indexing the directory "/dont-index-me/"?
  Disallow: /dont-index-me/
• What is the extension of a Unix/Linux system configuration file that we might want to hide from "Crawlers"?
  .conf
• What is the typical file structure of a "Sitemap"?
  XML
• What real life example can "Sitemaps" be compared to?
  Map
• Name the keyword for the path taken for content on a website
  Route
• What would be the format used to query the site bbc.co.uk about flood defences
  site: bbc.co.uk flood defences
• What term would you use to search by file type?
  filetype:
• What term can we use to look for login pages?
  intitle: login