AI crawlers destroying websites in hunger for content

Traffic on my website has plunged over the past few years. In 2019, my site attracted almost 1.5 million visitors. The content on my site has not gotten this bad, this quickly!

Back in 2016, a single popular page on my website attracted almost 1 million visits — about as many visits as my entire website got in the years 2022, 2024 and 2024 combined. 

I’ve never spent a dime promoting my website. The big drop in 2017 came after a homework cheating service purchased ads that Google ran above the organic search results, grabbing traffic that used to click through to my free resources.

But GenZ has little direct contact with web pages, preferring instead to get their content through social media apps and AI tools. 

The content on my website is just as good as it has been for decades. But people searching for writing tips or info on niche topics that I write about are satisfied by the AI-driven summaries that appear along with search results, instead of clicking the link and actually reading my resources for themselves.

[W]hen AI searchbots, with Meta (52% of AI searchbot traffic), Google (23%), and OpenAI (20%) leading the way, clobber websites with as much as 30 Terabits in a single surge, they’re damaging even the largest companies’ site performance.

Now, if that were traffic that I could monetize, it would be one thing. It’s not. It used to be when search indexing crawler, Googlebot, came calling, I could always hope that some story on my site would land on the magical first page of someone’s search results so they’d visit me, they’d read the story, and two or three times out of a hundred visits, they’d click on an ad, and I’d get a few pennies of income. Or, if I had a business site, I might sell a widget or get someone to do business with me.

AI searchbots? Not so much. AI crawlers don’t direct users back to the original sources. They kick our sites around, return nothing, and we’re left trying to decide how we’re to make a living in the AI-driven web world.

[…]

Remember the open web? I do. I can see our kids on the Internet, where you must pay cash money to access almost anything. I don’t think anyone wants a Balkanized Internet, but I fear that’s exactly where we’re going. —The Register