AI and the future of web searches

ChatGPT has the potential to revolutionize the way we browse and generate content for the internet. Having used it for a couple of weeks, I believe that it has implications for the websites that ChatGPT crawls to obtain information. In this article, I will specifically discuss the Bing Chat feature, which uses OpenAI technology.

Unfortunately, much of the content on the web today is not primarily created for human consumption. Many blog owners and media outles write articles with the sole purpose of pleasing search engines, hoping to have their content appear on the first page of search results. To achieve this, they often add superfluous content that is irrelevant to the main topic, in the hopes of triggering a search term.

This is where AI search results can make a significant difference in productivity. When prompted with a question, the AI runs a web search, crawls for information from multiple sources and compiles an answer. This is similar to conducting a traditional web search, where one would need to click on links, skim through articles, and avoid the fluff – a time-consuming endeavour. However, the Bing AI performs this task in seconds and provides a list of citations of the websites it crawled for information.

But this process poses an interesting problem for the websites that the bots are crawling to get their information. In a traditional web search, users have to navigate through the content themselves. This exposes them to other services or products offered by the site or advertising spots placed next to the content to generate income. When interacting with a chat, none of this organic interaction happens, and sites could potentially lose a significant amount of income when this way of searching becomes mainstream. A large portion of the organic traffic they would have received will simply disappear, and they may never know their content even appeared in a chat search.

To prevent search engines from indexing their content, sites can add a noindex tag to the meta head of an HTML document. Presumably, the AIs are also respecting this flag. However, if sites want their content indexed traditionally but not for the consumption of bots, a distinction needs to be made on exactly what automated systems can present and crawl what information. Alternatively, Microsoft may need to pay for the content it is presenting, much in the way advertisers pay per view and click of an ad. Do chat engines need to pay for content they’re consuming an rehashing?

In either case, there is going to be a turning point in how content will be created and consumed on the internet. Similar to how Google changed the way content is generated to appeal to search engines, OpenAI’s impact on content generation will be felt through chat results it generates. The question now is whether a simple citation is enough to give companies and individuals adequate brand exposure. Will they continue to receive enough clicks to make producing content viable?