A high-performance, production-ready web crawler written in Rust. Designed for maximum index purity and efficiency.

Key Features

• Zero-spam indexing: Intelligent filtering of <script> and <style> tags.
• Anti-Dupe System: Utilizes SHA-256 URL hashing for automatic content overwriting in OpenSearch.
• TTL (Time-To-Live): Automatic re-indexing of pages after a set duration (default is 7 days).
• Pause/Resume: Ability to pause and resume data collection without losing queue state.
• Asynchronous Performance: Built on tokio and axum for lightning-fast request processing.

Installation

1. Clone the repository:

   git clone [https://github.com/AICrox2025/SuperCrawl.git](https://github.com/AICrox2025/SuperCrawl.git)

2. Configuration: Configure your OpenSearch credentials in main.rs or Cargo.toml.

3. Run the project:

Built with Rust for speed and safety.