Screaming Frog is a popular SEO store scraping tool that give you insights into your store's SEO footprint. https://www.screamingfrog.co.uk/
When using any store scraper it's important to know the settings that will work and those that don't.
DCC Shield - Threat protection
All stores on EvolutionX are protected by automated threat protection system that adapts to realtime attacks and blocks those considered bad actors. This realtime analysis is looking at requests made to your website and evaluating whether they pose a risk to your store or our infrastructure.
If requests from any individual IP or group of IPs is deemed a threat, DCC shield will take immediate action and block those requests.
What DCC Shield means for your Scraper
When you point Screaming Frog at your own store, you may unintentionally cause DCC Shield to block your IP.
If you do get blocked (receiving 403 or 429 errors), contact customer success and we will evaluate and remedy the block.
👍 Important: Don't crawl your store from an IP in your office while other real users in your office are on the website. This can make it seem like Screaming Frog crawler is a real human and not a bot. This can cause it to get blocked. Use it out of hours or when other's in your office are not using the store.
Correct Settings for Screaming Frog
To make sure that you don't get caught in DCC Shield's web, be sure to follow the below settings.
Set your crawl Rendering to "text-only crawl"
Don't use a javascript crawl as this may ramp up requests
Find this in the Spider Configuration → Rendering Tab → Rendering set as "Text Only"
Set Max Threads to "1"
Check "Limit URL/s"
Set "Max URL/s" to "1.0"
Make sure you set it to "1.0" and not just "1"
That should be it, now you can crawl your store successfully without getting caught by DCC Shield.