Website to RAG Chunks: Vector-DB Ready Markdown
Crawl a website, docs portal or blog and get LLM-ready chunks for RAG: clean Markdown split by headings to your token budget, with overlap, heading path, chunk IDs, content hashes and page metadata. Drop straight into Pinecone, Qdrant, Weaviate, pgvector or LangChain. Pay per page.
Overview
A tool that crawls websites and documentation portals, extracts main content using a readability parser, and converts it into clean, LLM-ready Markdown chunks optimized for retrieval-augmented generation (RAG). It preserves GFM tables and code blocks, splits content by heading boundaries into chunks packed to a token budget with configurable overlap, and produces per-chunk metadata for indexing: a stable chunk identifier, a content hash for change detection, token counts, heading path and optional context prefixes, plus page metadata (title, description, language, dates, URL, crawl depth). The actor uses a plain HTTP crawler with an in-memory frontier for fast, non-browser crawling; chunking logic preserves code fences, merges tiny fragments, and drops extraneous tracking links to improve retrieval quality for embeddings and vector search.
Use cases
- Turn documentation portals, help centers, blogs or product pages into LLM-ready knowledge sources for chatbots and support assistants.
- Build RAG knowledge bases and vector indexes for Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or LangChain/LlamaIndex workflows.
- Maintain fresh indexes by re-crawling on a schedule and re-embedding only chunks that changed using content-hash-based change detection.
- Produce clean, heading-aware Markdown datasets with token counts suitable for LLM fine-tuning, evaluation sets, or embedding pipelines.
Pricing
| Page scraped | $0.003 per event (3 USD per 1,000) |
|---|---|
| Billing | Charged by Apify to your Apify account; platform usage may apply |
How to use
Open on Apify Store
Sign in to Apify or create a free account.
Configure the input
Fill in the input form; each field is documented on the Apify page.
Run and export
Start the run and download results as JSON, CSV or Excel, or use the API.
