AI companies have promised that next-generation search engines will drive more visitors to websites. However, a report from content licensing platform TollBit suggests otherwise. OpenAI and Perplexity have claimed their AI-powered search engines would generate new revenue streams for site owners, but the report, shared with Forbes, found that AI search engines send 96% less referral traffic to news sites and blogs compared to traditional Google search. Meanwhile, the volume of information AI collects from websites has more than doubled.
According to TollBit, OpenAI, Perplexity, Meta, and other AI companies crawled websites around 2 million times in Q4 2024. The data sample includes 160 sites, such as national and local news platforms, consumer technology sites, and shopping blogs. On average, each page was crawled seven times. TollBit provides tools to track AI crawls and charges AI platform developers fees when they access registered sites. OpenAI, Meta, and Perplexity did not comment on the report, though Perplexity stated that their system follows robots.txt directives, which define allowed areas of websites for crawlers.
The Impact of AI on Search Traffic
In February, Gartner analysts predicted that by 2026, traditional search engine traffic would decline by 25% due to AI chatbots and agents. The trend has already begun. The educational technology company Chegg recently sued Google over AI-generated summaries appearing in search results. In the second quarter of last year, when these summaries first appeared, Chegg’s traffic fell by 8%. By January this year, traffic had plummeted by 49%. The lawsuit claims that Google’s AI summaries included Chegg’s materials without attribution. As a result, the company is considering going private or selling its assets. Google called the lawsuit “baseless,” arguing that search result diversity has increased.
A further complication arises from AI crawlers misidentifying their “User-Agent” values, making it difficult for site owners to track data collection. Some reports suggest that Google uses the same bots for both indexing and AI training, while Perplexity’s actions are even more unpredictable. Although the company claims to respect robots.txt directives, blocked resources still report unexpected AI-related referral traffic. One website was officially scanned 500 times but received 10,000 visitors from Perplexity, raising concerns about unregistered bots.
AI Content Use and Copyright Issues
Perplexity has previously faced criticism for copying and reusing content from paid news outlets like Forbes, CNBC, and Bloomberg without proper attribution. Additionally, its service frequently links to low-quality AI-generated blogs and inaccurate social media posts. The New York Post and Dow Jones have sued Perplexity for copyright violations. Website owners also face increased costs due to uncontrolled AI data scraping, as it forces them to invest more in server resources. Meanwhile, OpenAI and Perplexity continue developing AI agents that autonomously gather online information and generate reports, which could further intensify these challenges.
One potential solution is direct content licensing, notes NIXSOLUTIONS. Organizations like the Associated Press, Axel Springer, and the Financial Times have already signed agreements with OpenAI. New companies are also emerging to charge AI platforms whenever they extract content from websites—a model used by TollBit, which compiled the recent report.
The situation continues to evolve, and we’ll keep you updated as more companies address these challenges.