WebgrepBot

WebgrepBot is the crawler behind this search engine. It downloads the public HTML of websites, mostly homepages, so that people can search their source code: which sites use a library, a theme, an analytics tag or a server.

How to recognise it

Every request it makes sends this user agent:

WebgrepBot/1.0 (+https://webgrep.78.47.41.66.sslip.io/bot)

What it does

• By default it reads robots.txt before anything else on a site and follows the rules for webgrepbot, or for * when there are none for it by name. If robots.txt cannot be fetched because of a server error, it assumes it may not crawl the site at all.
• By default it also skips pages marked <meta name="robots" content="noindex">.
• It makes one request at a time to a site, with a pause between them, usually of a second or more.
• It downloads HTML only. Images, video, downloads and anything over a few megabytes are left alone.
• It does not run JavaScript, submit forms, or sign in anywhere.
• It only connects to public addresses, never to private or internal networks.

How to stop it

Add this to your site's robots.txt:

User-agent: webgrepbot
Disallow: /

It takes effect the next time the crawler visits: robots.txt is read again at the start of every crawl. Pages collected before then stay in the index.

What is kept

For each page: the address, the response status line and headers, and the HTML as served, with each version kept as it changes. That is what the search syntax runs against. For each site, once a week at most: its robots.txt, ads.txt, app-ads.txt, security.txt, sitemap and favicon, its public DNS records, and the details of the TLS certificate it presents. Nothing the crawler keeps is anything a visitor could not see for themselves.

Removing a site

Blocking the crawler stops future visits. To have what was already collected deleted as well, send a removal request: once granted, the site's pages, history and records are deleted and it is never crawled again. Anything else: hello@example.com.