TORIXSoftware & digital products
SoftwarePay per result

Website to RAG Chunks: Vector-DB Ready Markdown

Crawl a website, docs portal or blog and get LLM-ready chunks for RAG: clean Markdown split by headings to your token budget, with overlap, heading path, chunk IDs, content hashes and page metadata. Drop straight into Pinecone, Qdrant, Weaviate, pgvector or LangChain. Pay per page.

Overview

A tool that crawls websites and documentation portals, extracts main content using a readability parser, and converts it into clean, LLM-ready Markdown chunks optimized for retrieval-augmented generation (RAG). It preserves GFM tables and code blocks, splits content by heading boundaries into chunks packed to a token budget with configurable overlap, and produces per-chunk metadata for indexing: a stable chunk identifier, a content hash for change detection, token counts, heading path and optional context prefixes, plus page metadata (title, description, language, dates, URL, crawl depth). The actor uses a plain HTTP crawler with an in-memory frontier for fast, non-browser crawling; chunking logic preserves code fences, merges tiny fragments, and drops extraneous tracking links to improve retrieval quality for embeddings and vector search.

Use cases

  • Turn documentation portals, help centers, blogs or product pages into LLM-ready knowledge sources for chatbots and support assistants.
  • Build RAG knowledge bases and vector indexes for Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or LangChain/LlamaIndex workflows.
  • Maintain fresh indexes by re-crawling on a schedule and re-embedding only chunks that changed using content-hash-based change detection.
  • Produce clean, heading-aware Markdown datasets with token counts suitable for LLM fine-tuning, evaluation sets, or embedding pipelines.

Pricing

Page scraped$0.003 per event (3 USD per 1,000)
BillingCharged by Apify to your Apify account; platform usage may apply

How to use

Open on Apify Store

Sign in to Apify or create a free account.

Configure the input

Fill in the input form; each field is documented on the Apify page.

Run and export

Start the run and download results as JSON, CSV or Excel, or use the API.