Hey everyone! I'm excited to share SpiderLock, an open-source Python web crawler I built specifically for security reconnaissance and site mapping. It's designed to give pentesters, bug bounty hunters, and security researchers a focused tool for understanding target structure.

Key Features:

🔹 Supports both Breadth-First Search (BFS) and Depth-First Search (DFS) crawling strategies

🔹 Respects robots.txt before starting any crawl 🔹 Configurable depth limits for controlled exploration

🔹 Stores results in SQLite for easy querying and integration

🔹 Generates a 2D graphical map of site structure (UML-style diagrams showing links and page relationships)

Use Cases:

• Pentesters performing reconnaissance during engagements
• Security researchers exploring target structures
• Developers/learners studying how crawlers work

The project is fully open-source and available here: 👉 GitHub – SpiderLock (https://github.com/sherlock2215/SpiderLock)

Seeking Feedback! 🙏

As I develop this further, I'd really appreciate your thoughts on:

1. Workflow Enhancements: What features would make it more practical for your penetration testing or bug bounty workflows?
2. Integrations: Any suggestions for other tools it should integrate with (e.g., Nmap, Gobuster, or vulnerability parsers)?
3. Data & Visualization: Improvements to the visualization or other data export formats you'd find useful.

Looking forward to your thoughts and pull requests! Happy crawling!