r/SEO_LLM•
JobOk233

Why Does Myntra’s robots.txt Have Weird Parameters Like amazon com, reddit com, xnxx, flipkart, etc.?

Was analyzing Myntra’s robots.txt and noticed they are blocking hundreds of unusual patterns/keywords like competitor domains, spam terms, adult keywords, and autogenerated-looking URLs.

Example:

https://preview.redd.it/m4mdoozkzu3h1.png?width=731&format=png&auto=webp&s=73e138206ce915c1cad4d22abb70c295123c1371

Now I’m curious:

Are these there because:

  • large ecommerce sites face massive spam URL generation?
  • Google discovers weird URLs through backlinks/crawler traps?
  • dynamic routing creates crawlable junk pages?
  • they’re trying to save crawl budget aggressively?
  • or this is mainly for hacked/parasite SEO prevention?

Would love to know how enterprise SEO teams handle this at scale.

Posted May 28, 2026, 10:54 AM UTC

01
Loading comments…