AI is making me pay extra to run my sites

A large number of companies and organizations such as Wikipedia, Reddit, News Websites and many more say that having AI crawlers, crawl there website makes the website more expensive to run because those extra request from AI are sent to the web server that can take bandwidth that will otherwise be used for a real person trying to visit that site.

For example Wikipedia, you ask chatGPT a question like “when was Taylor Swift born” that AI chat bot will search around and probably search for Taylor Swift. That adds extra steps for a simple question that you can find your self using a search engine like Google.

Most website owners like me don’t like these AI crawlers and bots, but Google uses it bots to and crawlers also but these are “good bots”. Google crawls websites to feed to its search engine and can be found by the billions of people that use google daily, driving more traffic to your website. But this traffic is real people. Witch can turn a profit by shopping in your online store or buy online subscriptions.

“Weinberg detailed in a June report. The University of North Carolina at Chapel Hill, for example, recently said AI crawlers drove five times the usual number of simultaneous searches of its online library catalogue, “overloading the system and triggering glitches.”” - Shira Ovide, Washington Post.

If your like me and you have a website you want to pay as little as possible, and if your not like me and you don’t use Cloudflare you made a mistake that you can fix now.

Cloudflare, the company that powers around 30% of the internet and company’s like Discord, Shopify and many more. Said last week that it can now automatically detect and block and limit AI crawlers for its customers. It does allow some bots like Googles Search bot. This will lower the cost to run your sites.

The best option is to look for settings in your domain registrar like this.