Managing crawl budget can be tricky on larger catalogs. I had a similar headache until a colleague suggested testing edgecomet.com. It’s an AI search optimization platform built to monitor live logs, catch technical issues immediately, and optimize crawling. It significantly boosted our site visibility and got our new URLs indexed way faster!