TLDR Tech / Marketing
OpenAI Has Tripled Their Crawl of the Web: An Analysis of 7B+ Log Files
OpenAI is scaling crawl activity, so brands need to treat AI bots as a discovery channel alongside search. Total crawl volume has tripled since August 2025, while direct user-triggered visits dropped 28%. This signals greater reliance on cached indexes and automated retrieval. Post GPT-5, OpenAI has leaned more on search than training, making real-time visibility more important than static training data inclusion. Crawl behavior varies by industry, requiring vertical-specific optimization. Google still dominates at 18.2B monthly crawl events versus OpenAI's 887M, but OpenAI has grown from 1.38% to 4% of Google's crawl share in a year.
Read the full story