TLDR Tech / Dev
Crawl web content
The Cloudflare `/crawl` endpoint allows users to scrape web content by providing a starting URL and following links up to a configurable depth or page limit. The process involves initiating a crawl job with a POST request to receive a job ID, then using that ID in a GET request to poll for job status and retrieve the results. Content can be returned in HTML, Markdown, or JSON.
Read the full story