Op-Ed: Stop the Bad Bots
News publishers across the country are struggling with a uniquely modern problem – their webpages are being overrun with bots. Content that is written for human readers is being consumed by millions of these automated crawlers that comb the Internet and often go out of their way to disguise their identity and steal content. These uninvited “bad bots,” many of whom come from China or other countries, are slowing down publishers’ pages for users and undermining a healthy content licensing market that would support continued community coverage, ultimately impacting countless newsrooms’ ability to serve their readers.
Publishers have tried to stop these bots on their own, but they need help. Thankfully, Representatives Laurel Lee (FL-15), Valerie Foushee (NC-4), and Gus Bilirakis (FL-12) recently introduced the Stealth Bot Prohibition Act, a new piece of bipartisan transparency legislation that would require bad bots to disclose their identity and purpose, giving content providers the information they need to protect their work. This is a simple, common-sense solution that is already receiving wide support from members of Congress on both sides of the aisle, and it couldn’t come at a more urgent time.
Content providers all across the internet, including web archives, libraries, health and financial institutions, and even responsible AI companies themselves, have reported a dramatic increase in the number of bad bots weighing down their websites. It’s a problem so widespread that experts now say there are more bots than people on the internet. This new flood of traffic is changing the makeup of the internet itself, and in the process, creating a significant financial burden for newsrooms both large and small.
With normal human activity, the costs of hosting a website – servers, electricity, bandwidth, etc. – are offset by users paying for subscriptions or viewing or clicking on ads. But bad bots don’t click on ads, and they don’t pay for subscriptions. They don’t even browse a website like a human, with a few page views a day. They hit pages over and over again, sometimes millions of times in a single day. All that traffic can slow a website down for human readers, and if it happens enough, can trigger bandwidth overage charges, slower pages, or even total outages. Many publishers, especially local news providers, are already operating at razor-thin margins and can’t afford the higher technical costs that come with this deluge of bad bots.
To make matters worse, many of these bad bots take news content just to put it on a multi-billion-dollar reseller market, one where AI developers buy news content from the owners of the bots, not the creators of the content. They then use that content to create substitutive products that directly compete with the publisher websites themselves. Users have started getting their news and questions answered by AI chatbots, rather than clicking on links to the original sources of this information. As a result, web traffic for news websites continues to fall. Reversing this trend is the first step in establishing a healthy licensing marketplace where content is rewarded as a valuable input for AI, just like investments in energy and chips.
The news industry – from the largest corporations to the smallest local newsrooms – has tried to block these bots using technical fixes, but bad bots are so advanced that they’re able to disguise themselves to look like good bots or even humans. Malicious foreign actors are now hijacking devices and servers in ordinary people’s homes to act as “residential proxies,” masking not only the identity of their bots, but their source as well. These bots are so effective at disguising themselves that publishers simply can’t keep up, and are losing precious resources trying to play bot Whac-a-Mole. Instead, we need lawmakers to step up to protect quality content that requires tremendous investment to create.
The Stealth Bot Prohibition Act offers a simple solution. It doesn’t ban bots outright, but forces them to identify themselves, a common sense transparency measure which gives news publishers the information they need to block bad bots themselves, with Federal Trade Commission and other enforcement mechanisms to ensure compliance. Representatives Lee, Foushee, and Bilirakis are offering a reasonable, modest standard, and it’s one that their colleagues should get behind to keep the internet clean and protect the quality content that the American people and the world rely on.