VibekingBot

Recognising it

Every request from VibekingBot has this user agent:

VibekingBot/1.0 (+https://vibekingdevelopment.com/bot)

A user agent can be imitated, so check the signature below.

Verification

Every VibekingBot request carries Web Bot Auth headers: Signature-Agent with the value "https://vibekingdevelopment.com", Signature-Input and Signature. Every request, including every redirect hop, gets its own signature.

The public keys are at:

https://vibekingdevelopment.com/.well-known/http-message-signatures-directory

Site owners and firewalls can verify the signature instead of trusting the user agent.

There is still no IP list: VibekingBot runs on a cloud network with changing addresses.

Is your site behind QUIC.cloud? QUIC.cloud does not put VibekingBot on its good bots list; allow it in your QUIC.cloud dashboard under CDN, CDN Config, Security, Access Control, User Agent Allowlist.

Shopify store owners can create a crawler signature for VibekingBot in their Shopify admin and enter it in Vibeking. While it is valid, requests to that domain carry Shopify's signature instead of Vibeking's.

What it requests

• Scan: weekly on Monday from 02:00 UTC, when a customer starts a scan and as soon as a customer adds their first site. VibekingBot reads robots.txt, the sitemap, llms.txt and the pages of the site (HTML, up to 2 MB per page). For linked files such as PDFs and images, for internal links the scan does not read itself and for a sample from the sitemap and from llms.txt it only requests the status, first with HEAD and with GET if needed. It also checks that the http and www variants of the homepage redirect correctly.
• Check for pages that do not exist: one request per scan to a made-up address on the site (for example /vk-niet-bestaand-…), only when robots.txt allows it and after the Crawl-delay. This shows whether the site returns a proper 404 for unknown addresses.
• Availability: every 5 minutes the site's address, following any redirects. Only the status counts; the content is not read.
• Changes: once a day, one after another, the homepage, robots.txt, sitemap.xml and llms.txt.
• PageSpeed: daily for sites with a paid subscription, for each measured page first only the status. The measurement itself is done by Google PageSpeed Insights; those requests come from Google.
• Proof after a fix a customer orders: a few pages and files to check the result.

Speed

A scan makes one request at a time and waits at least one second between two requests.

If robots.txt has a Crawl-delay in the group for VibekingBot, or otherwise in the group for all bots (*), the scan waits that long between two requests. There is no maximum. With a Crawl-delay above 60 seconds a scan checks fewer pages, so it finishes within about a day.

The daily check for changes makes its four requests one after another, with the Crawl-delay in between (at least one second). The other checks above do not follow Crawl-delay.

robots.txt

The scan follows Disallow and Allow for pages and links on the site, from the group for VibekingBot or otherwise from the group for all bots (*). A group of its own for VibekingBot replaces the group for all bots, so add your Disallow rules to it as well.

The scan always requests robots.txt, the sitemap, llms.txt, the homepage (including the http and www variants) and the target of a redirect. The availability, change, PageSpeed and proof checks do not look at robots.txt.

User-agent: VibekingBot
Crawl-delay: 5
Disallow: /admin/

User-agent: VibekingBot
Disallow: /

With Disallow: / the scan skips all pages, except what is listed above as always requested. That also applies when the site owner is a customer.

VibekingRadar

Vibeking also follows news and changes from sources such as Google, OpenAI, Bing and schema.org. VibekingRadar fetches those RSS feeds and pages every 30 minutes, with this user agent:

VibekingRadar/1.0 (+https://vibekingdevelopment.com/bot)

Questions

Email info@vibekingdevelopment.com.