🕷️ Advanced Go Web Scraper & Crawler API

A high-performance, concurrent web scraping and deep crawling API built in Go (Golang) using the Gin Web Framework and Go's native recursive HTML parsing nodes.

---

🚀 Features

• Concurrency-Safe Visited Registry: Uses a mutex-locked tracker to prevent infinite loops and duplicate scraping passes.
• Depth-Based BFS Crawler: Implements a Breadth-First Search (BFS) spider queue using goroutines and wait-groups, parallelizing requests up to a configurable concurrency limit.
• Recursive HTML Node Parsing: Walks Go's tokenized HTML element nodes to extract page titles, meta descriptions, meta keywords, headings (h1 through h6), paragraph text blocks, links, and image sources recursively.
• CORS Enabled: Out-of-the-box support for Cross-Origin Resource Sharing, allowing frontend single-page apps (SPA) to query endpoints directly.
• Robust Relative URL Resolution: Resolves relative paths automatically to absolute URLs relative to the base domain.

---

📐 Architecture & Layout

• main.go: Entry point, registers middlewares (CORS), initiates routers, and launches the HTTP server.
• models/models.go: Defines structured JSON schemas for scraping requests, crawled metrics, link shapes, and responses.
• controller/handler.go: Decodes JSON requests, calls service APIs, and formats HTTP response codes.
• service/scraper.go: Declares the ScraperService interface and its concrete implementation GoScraperService, encapsulating concurrent crawler threads and low-level HTML walk engines.

---

🏃 Running the Server

Start standard Go server (runs on port 8000):

go build -o server main.go
./server

---

📡 API Reference

1. Health Status

Check if the crawler service is operational.

• Request: GET /health
• Response:

  {
    "engine": "Golang net/html recursive parser + BFS Concurrent Crawler",
    "message": "Scraper API in Go is fully operational",
    "status": "healthy"
  }

---

2. Single-Page Scrape

Extract meta tags, headings, text, media, and link relationships from a single URL.

• Request: POST /api/scrape
• Headers: Content-Type: application/json
• Body:

  {
    "url": "https://example.com",
    "timeout": 10,
    "follow_redirects": true,
    "headers": {
      "Custom-Token": "optional-key-here"
    }
  }

• Response:

  {
    "url": "https://example.com",
    "status_code": 200,
    "title": "Example Domain",
    "meta_description": "A placeholder domain used for illustrative examples.",
    "meta_keywords": "examples, test",
    "headings": {
      "h1": ["Example Domain"]
    },
    "paragraphs": [
      "This domain is for use in illustrative examples in documents."
    ],
    "links": [
      {
        "text": "More information...",
        "url": "https://www.iana.org/domains/reserved",
        "external": true
      }
    ],
    "images": [],
    "raw_text_length": 61
  }

---

3. Deep BFS Crawling

Search recursively through child pages under a given domain base.

• Request: POST /api/crawl
• Headers: Content-Type: application/json
• Body:

  {
    "url": "https://example.com",
    "max_depth": 2,
    "max_pages": 15,
    "concurrency": 4,
    "polite_delay": 200,
    "same_domain": true
  }

• Response:

  {
    "start_url": "https://example.com",
    "total_pages_scraped": 2,
    "pages": [
      {
        "url": "https://example.com",
        "status_code": 200,
        "title": "Example Domain",
        ...
      }
    ]
  }