Perplexity published a response to Cloudflare’s claims that it disrespects robots.txt and engages in stealth crawling. Perplexity argues that Cloudflare is mischaracterizing AI Assistants as web crawlers, saying that they should not be subject to the same restrictions since they are user-initiated assistants. According to Perplexity, its system does not store or index content ahead of time. Instead, it fetches webpages only in response to specific user questions.
Source: Perplexity Says Cloudflare Is Blocking Legitimate AI Assistants