On October 13, we read the public robots.txt files of 11 news sites that cover the Coachella Valley and the Inland Empire. Seven ask AI companies’ crawlers to stay out. Three do not. One, the Press-Enterprise, has no robots.txt at all. Of the 10 sites we could check, none publishes the file that a new publisher standard uses to say where reports on how AI systems used a site’s journalism should be sent.
That standard is Content Telemetry 1.0, published October 2 by SPUR, the publisher coalition. It defines how AI agents can report when they retrieve, ground an answer in, cite, display or send readers to a publisher’s content. Our brief on SPUR has the background. The spec says a content owner publishes a manifest at /.well-known/content-telemetry.json that names where reports about its content should be sent. We checked for that file on every site. It returned “not found” at 10 sites, and the server at the 11th, Palm Springs Life, refused the request, so we could not tell. The standard was 11 days old.
The Inland Valley Daily Bulletin, the San Bernardino Sun and the Redlands Daily Facts each tell all 12 AI crawlers we tested to keep out of the whole site. The list: OpenAI’s GPTBot, ChatGPT-User and OAI-SearchBot; Anthropic’s ClaudeBot and anthropic-ai; Google-Extended; Common Crawl’s CCBot; PerplexityBot; Applebot-Extended; ByteDance’s Bytespider; Meta’s meta-externalagent; and Amazonbot. The Desert Sun’s file also disallows all 12, with narrow exceptions: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot and Google-Extended are allowed into branded and contributor content paths.
The Desert Review and the Hi-Desert Star block 11 of the 12, all but Amazonbot. Palm Springs Life blocks nine: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent and Amazonbot. It does not block ChatGPT-User, OAI-SearchBot or PerplexityBot.
KESQ, the Coachella Valley Independent and NBC Palm Springs, whose KMIR address leads to the same site, name no AI crawler in their files. The Press-Enterprise’s robots.txt returned “not found,” so it states no crawler rules.
A robots.txt file is a request. Crawlers follow it voluntarily, and the file does not show whether any AI company has used a site’s reporting. The 12 crawlers are the ones we chose to test; a site could restrict others. These 11 sites are the outlets we could identify that report on the region, not a census, and we did not ask any of them why their files read as they do.
For the record, AICV.news’s own robots.txt disallows nothing and explicitly names OpenAI’s, Perplexity’s and Anthropic’s crawlers as allowed, and we do not publish a manifest either.
How we checked: a script that identified itself as an AICV News check fetched each site’s robots.txt and manifest path twice on October 13, and the results matched for the nine sites that answered it. The Desert Sun and NBC Palm Springs refused scripted requests, so we read their files once, in a browser. We counted a crawler as blocked when a group naming it contains “Disallow: /”. The spec’s description of the manifest is in section 8 of its specification.