Back
07 Mar, 2024

Fetch Zendesk pages using proxy

As Zendesk has implemented Cloudflare to protect knowledge base articles from being accessed by bots, we've implemented a little logic so that a rotating proxy is used when scraping an article's content. We've seen some good results, but every now and then a page is still not scraped.

You can verify this by looking at the page title and using the View option to check the content that was actually scraped.