crawlengine-mcp

CrawlEngine MCP is a Model Context Protocol (2024-11-05) web extraction server built on httpx and beautifulsoup4 that converts HTML pages into clean Markdown, resolves relative links, extracts JSON-LD and HTML tables, and rotates browser headers and HTTP/SOCKS5 proxies.

Listed in TensorBlock/awesome-mcp-servers.

Why CrawlEngine MCP?

1. Context Window Efficiency: Strips <script>, <style>, <nav>, <footer>, cookie banners, and ads before returning Markdown to the model.
2. Resolved Canonical Links: Converts relative <a href="/docs/start"> paths into full URLs using the page's base_url.
3. Outbound SSRF Protection: Blocks loopback, RFC1918 private networks, and cloud metadata (169.254.169.254) on both initial requests and HTTP redirects.

Features

• Semantic Markdown Extraction: Preserves headings, lists, code blocks, blockquotes, and resolved inline links without duplicating nested list paragraphs.
• Browser Header Rotation: Rotates realistic Chrome, Firefox, and Safari User-Agent and Sec-Ch-Ua client-hint headers.
• HTTP & SOCKS5 Proxy Pool: Round-robin rotation via CRAWLENGINE_PROXIES (httpx[socks]) with automatic ejection after 3 consecutive failures.
• Structured Data & Metadata: Extracts application/ld+json schemas, HTML <table> rows, OpenGraph tags, canonical URLs, and filtered internal/external links.

Quickstart

Run with uvx

uvx crawlengine-mcp

Claude Desktop / Cursor Configuration

{
  "mcpServers": {
    "crawlengine-mcp": {
      "command": "uvx",
      "args": ["crawlengine-mcp"]
    }
  }
}

CLI Usage

crawlengine-cli scrape https://example.com
crawlengine-cli links https://example.com --internal-only
crawlengine-cli meta https://example.com

MCP Tools Available

Tool	Parameters	Description
scrape_url	url, extract_markdown, use_proxy, timeout	Fetch an HTTP(S) page and convert its main content into Markdown
extract_structured_data	url	Parse application/ld+json blocks and HTML tables into JSON records
extract_links	url, internal_only	Return deduplicated absolute hyperlinks with anchor text
extract_metadata	url	Extract page title, meta description, OpenGraph tags, and canonical URL
get_proxy_status	(none)	Inspect total, healthy, and ejected proxy counts in CRAWLENGINE_PROXIES

Genesis MCP Ecosystem

Module	Description	Repository
CrawlEngine MCP	HTTP web extraction to clean Markdown, JSON-LD parsing, and proxy rotation	GitHub
BrowserBridge MCP	Lightweight HTTP/DOM session automation with SHA-256 action receipts	GitHub
PgSafe MCP	Read-only PostgreSQL/SQLite query guard, PII masking, and dry-run mutations	GitHub
OmniBridge MCP	Multi-channel notification bus with SQLite retry queue	GitHub
SysFlow MCP	Linux systemd inspection, confined log tailing, and port auditing	GitHub
FastMCP Forge	Zero-dependency FastMCP framework, CLI scaffolder, and SSE middleware	GitHub

Support

If you find this project useful, you can support maintenance on Ko-fi.

License

MIT (c) 2026 Johann Schiffer