Bot

BizKitHubBot is an automated web crawler developed by BizKitHub for the technical analysis of the web environment, with emphasis on stability, security, and optimization.

About BizKitHubBot

BizKitHubBot is designed to collect publicly available information from the internet with emphasis on stability, security, and the optimization of web services.

What BizKitHubBot Is For

• Technical analysis — analyzes the technical state of websites and their availability.
• Problem detection — detects outages, error states, and anomalies.
• Structure mapping — maps the structure of websites and their URLs.
• Technical history — builds a technical history of web resources.
• Security analysis — identifies security threats.
• SEO optimization — improves SEO based on the real topology of the internet.

Services built on BizKitHubBot data help both website operators and end users by uncovering technical problems and supporting a faster, safer web.

Technical Behavior

BizKitHubBot accesses public websites under the following conditions:

• Respects robots.txt — first fetches and evaluates the robots.txt file if it exists.
• Minimal load — waits at least 5 seconds between two requests.
• Public sites only — does not attempt to access non-publicly available pages.
• Prioritizes sitemaps — URLs listed in sitemap.xml receive higher crawling priority.

Crawl Budget

Each website has a monthly limit on the number of crawled URLs (default: 50) that changes dynamically based on the quality of the site.

Identifying BizKitHubBot

User-Agent:

BizKitHubBot/1.0 (+https://bizkithub.com/bot)

To verify the authenticity of a request, a reverse DNS lookup can be used.

Robots.txt Rules

BizKitHubBot follows the standard robots.txt specification. A rule disallowing crawling can look, for example, like this:

User-agent: BizKitHubBot
Disallow: /

Note: The User-agent is case-insensitive, so BizKitHubBot is equivalent to bizkithubbot. Paths (e.g. /Private/) are however case-sensitive.

Limitations and Specifics

• File size — BizKitHubBot loads at most the first 15 MB of each HTML or text file (in its uncompressed form).
• Request count — the number of requests is limited so that the server is not overloaded.
• Broken robots.txt — if robots.txt is malformed or missing, access is considered allowed.

How to Disallow or Limit Crawling

To disallow crawling of your site:

User-agent: BizKitHubBot
Disallow: /

• Automatic deletion — data is removed after robots.txt is re-fetched, typically within a week.
• Expedited removal — an expedited removal can be requested via BizKitHub Cloud.

Data Processing and Protection

All data is:

• Collected only from publicly accessible sources
• Anonymized for public outputs
• Made available to the specific operators of their own websites
• Protected against misuse in accordance with the law

Crawl Budget Optimization

The amount of URLs crawled is determined by the crawl budget, which can be influenced by the following factors.

Increasing the crawl budget

• Fast and stable server responses
• Correctly set status codes
• Absence of duplicate content
• Structured sitemaps
• Registration of the site in BizKitHub Cloud

Penalizing the crawl budget

• Frequent server outages
• Invalid or missing HTTP headers
• Redirects to non-existent pages
• Substantially duplicate content

Note: BizKitHub reserves the right to decide on the prioritization of crawling and the scope of data collection.

Developer Tools

PHP library for custom crawling

composer require baraja-core/webcrawler

Guides and examples are available in the PHP manual for developers.

Part of the BizKitHub Ecosystem

BizKitHubBot is part of the BizKitHub Cloud ecosystem and contributes to technically improving the quality of web services in the Czech Republic and globally. More information at bizkithub.com/bot.