How The Rapid Growth Of Bots Could Change The Open Web

So far, bots have collected information that makes apps better for humans. But a new species grows in population: agentic bots.

For two decades, we optimized sites for GoogleBot. Soon, we could focus on AI helpers that act as intermediaries between humans and the open web.

We have already optimized for the BotNet with Schema and product feeds in Google’s Merchant Center. XML sitemaps have become table stakes decades ago.

At the next level, we could have separate websites or APIs for agentic bots with a whole new marketing playing field.

Agentic Web

As we’re getting close to getting more bot than human traffic on the open web, it’s essential to keep in mind that about 65% of bot traffic is estimated to be malicious.

Good bots include scrapers from search engines, SEO tools, security defense, and, of course, AI crawlers. Both types of bots are growing in population, but only the good ones are really useful.

Comparing the last with the previous six months, AI crawlers grew by 18% while search engines slowed down by 10%. GoogleBot specifically reduced activity by -1.6%.

However, Google’s AI crawler made up for it with an increase of 1.4%. GPT Bot was the most active AI crawler, with 3.8% of all requests – and grew 12%.

Image Credit: Kevin Indig

Today, AI bots have three goals:

1. Collect training data.
2. Build a search index to ground LLM answers (RAG).
3. Collect real-time data for prompts that demand freshness.

But currently, all big AI developers work on agents that browse the web and take action for users:

I see three possible outcomes:

The reason is that human attention shifts to social platforms, especially for younger generations. As the bots take the open web, humans flee to engagement retreats.

Marketing

Image Credit: Kevin Indig

In the distant future, good bots could become equal rights citizens of the open web nation as more humans spend time on closed social platforms and agentic LLMs increase the already fast growth of bot traffic. If this theory plays out, what does this mean for marketing and specifically SEO?

Imagine booking a trip. Instead of browsing Google or Booking, you tell your agent where you think about going and when.

Based on what it knows about your preferences, your agent curates three options of flights and hotels from your platform of choice.

When you pick a flight, it’s added to your calendar, and the tickets are in your inbox. You don’t need to checkout. The agent does everything for you. You could apply the same scenario to ecommerce or software.

Given that companies like Google already have the capabilities to build this today, it’s worth thinking about what would remain constant, what would change, and what would become more and less important in this vision.

Constant

Bots don’t need CSS or hero images. There is no downside to cloaking your site for LLM crawlers, so there is a chance that websites show bots a barebones version.

A skill set that remains constant in this future is technical SEO: crawlability, (server) speed, internal linking, and structured data.

Change

An agentic open web offers much better ad targeting capabilities since bots know their owners inside out.

Humans will make purchase decisions a lot faster since their agents give them all the information they need and know their preferences.

Advertising costs drop significantly and offer even higher returns than already today.

Since bots can translate anything in seconds, localization and international sales are no longer a problem. Humans can buy from anyone, anywhere – only constraints by shipping and inventory. The global economy opens up even further.

If we play it right, agents could be the ultimate safe-keepers of privacy: No one has as much data about you as they do, but you can control how much they share.

Agents know you but don’t need to share that information with others. We might share even more data with them, spinning the value flywheel of data → understanding → results → data → value → results → etc.

On the downside, we need to build defenses against rogue bots by redefining what bots are allowed to do in a robots.txt 2.0-like format. Cybersecurity has become even more important, but also more complex since bad bots can mimic good ones so much better.

We’ll need to figure out the environmental impact of more energy consumption by higher bot traffic.

Hopefully, bots will be more efficient and, therefore, cause less total web traffic than humans. This would at least somewhat offset the energy gorge LLMs are already causing.

Important

The most bot-friendly format of information is raw and structured: XML, RSS, and API feeds. We have already sent product feeds and XML sitemaps to Google, but agents will want more.

Web design will be less important in the future, and maybe that’s okay since most sites look very similar anyway.

Feed design becomes more important: what information to include in feeds, how much, how often to update it, and what requests to send back to bots.

Marketers will spend a lot more time reverse-engineering what conversations with chatbots look like. Bots will likely be a black box like Google’s algorithm, but advertising could bring light into what people ask the most.

Relationships

In the agentic future, it’s hard to get customers to switch once customers have settled on a brand they like until they have a bad experience.

As a result, an important marketing lever will be getting customers to try your brand with campaigns like discounts and exclusive offers.

Once you have conviction and signal that your product is better, the quest becomes persuading users to try it out. Obviously, that already works today.

With so many opportunities to advertise and sway users towards a specific product organically before they buy, we have a lot more influence on the purchase.