Document parsing and data ingestion
Data ingestion tools turn PDFs, Office files, scans, images, and web pages into clean text, Markdown, or structured JSON that a RAG pipeline can chunk, embed, and cite. Parsing quality sets a ceiling on retrieval quality: a table read in the wrong order or a dropped footnote cannot be recovered later in the pipeline.
The category spans open document parsers and OCR toolkits, such as Docling, Marker, MinerU, and PaddleOCR; managed document APIs, such as LlamaParse, Reducto, Mistral OCR, and the Azure, AWS, and Google Cloud services; and web extraction tools, such as Firecrawl, Crawl4AI, Jina Reader, and Apify, that crawl sites and return LLM-ready Markdown.
Many open-source options use copyleft licenses such as AGPL-3.0, and several ship model weights under separate terms, so check both the code and the model licenses before deploying.
Document parsing and data ingestion compared
| Platform | Open source | Deployment | Best for |
|---|---|---|---|
| Amazon Textract | No | Managed cloud, API | Document extraction inside AWS workflows |
| Apache Tika | Yes (Apache-2.0) | Library, Self-hosted, CLI | Broad file-format coverage in Java pipelines |
| Apify | No | Managed cloud, API | Ready-made scrapers for specific websites |
| Azure Document Intelligence | No | Managed cloud, API | Form and document extraction on Azure |
| Chunkr | Yes (AGPL-3.0) | API, Self-hosted | Self-hosted layout analysis and chunking |
| Crawl4AI | Yes (Apache-2.0) | Library, Self-hosted, Managed cloud | Self-hosted crawling for LLM pipelines |
| Docling | Yes (MIT) | Library, Self-hosted, Local | Self-hosted parsing that preserves document structure |
| Extend | No | API, Managed cloud | Production document extraction pipelines |
| Firecrawl | Yes (AGPL-3.0) | Managed cloud, API, Self-hosted | Turning websites into LLM-ready Markdown |
| Google Document AI | No | Managed cloud, API | Document processing on Google Cloud |
| Jina Reader | Yes (Apache-2.0) | API, Self-hosted | Quick URL-to-Markdown conversion with no setup |
| LandingAI Agentic Document Extraction | No | API, Managed cloud | Schema-based extraction with visual grounding |
| LiteParse | Yes (Apache-2.0) | Library, CLI, Local | Local parsing without GPUs or API calls |
| LlamaParse | No | Managed cloud, API | Complex layouts, tables, and charts in RAG pipelines |
| Marker | Yes (Apache-2.0 (code); modified OpenRAIL-M (model weights)) | Library, CLI, API, Managed cloud | Converting PDFs to Markdown locally or by API |
| MarkItDown | Yes (MIT) | Library, CLI | Lightweight conversion of Office files to Markdown |
| MinerU | Yes (MinerU Open Source License (Apache-2.0 with additional terms)) | Library, CLI, Self-hosted, Managed cloud | Self-hosted PDF conversion with formula and table support |
| Mistral OCR | No | API, Self-hosted | API-based OCR for scanned and handwritten documents |
| olmOCR | Yes (Apache-2.0) | Library, Self-hosted | Batch OCR of large PDF collections on your own GPUs |
| PaddleOCR | Yes (Apache-2.0) | Library, Self-hosted | Multilingual OCR that runs on your own hardware |
| PyMuPDF4LLM | Yes (AGPL-3.0) | Library | Text-layer PDF extraction in Python |
| Reducto | No | API, Managed cloud | Parsing complex enterprise documents through an API |
| ScrapeGraphAI | Yes (MIT) | Library, API, Managed cloud | Prompt-based extraction of structured web data |
| Spider | Yes (MIT) | Library, API, Managed cloud | Large crawls where throughput matters |
| Trafilatura | Yes (Apache-2.0) | Library, CLI | Extracting article text from web pages at scale |
| Unstract | Yes (AGPL-3.0) | Self-hosted, Managed cloud | Prompt-defined extraction deployed as APIs or ETL jobs |
| Unstructured | Yes (Apache-2.0) | Library, Managed cloud, API | Converting mixed document types into RAG-ready elements |
| Xberg | Yes (MIT) | Library, CLI, Self-hosted, Managed cloud | Polyglot document extraction in self-hosted pipelines |
Profiles

Amazon Textract
Amazon Web Services · aws.amazon.com
Amazon Textract is an AWS service that uses OCR and machine learning to extract printed text, handwriting, forms, tables, and answers to natural-language queries from scanned documents, returning results with confidence scores and coordinates.

Apache Tika
Apache Software Foundation · tika.apache.org
Apache Tika is a Java toolkit that detects and extracts text and metadata from more than a thousand file types, available as a library, a REST server, and a command-line app, and often used as the parsing stage in search and RAG ingestion.
Apify
apify.com
Apify is a web scraping and automation platform with a marketplace of ready-made scrapers called Actors, an MCP server for agents, and the Apache-2.0 Crawlee library for building crawlers in Node.js and Python.
Azure Document Intelligence
Microsoft · azure.microsoft.com
Azure Document Intelligence extracts text, key-value pairs, tables, and fields from documents with prebuilt and custom models, and Azure Content Understanding extends it with LLM-based analyzers for documents, images, audio, and video.
Microsoft now presents Document Intelligence as part of Azure Content Understanding in Foundry Tools, the new name for Azure AI services.
Chunkr
Lumina AI · chunkr.ai
Chunkr is a document intelligence API and AGPL-3.0 self-hostable service for layout analysis, OCR with bounding boxes, and semantic chunking that converts PDFs, Office files, and images into HTML, Markdown, or JSON.
Chunkr is being acquired by Extend; the hosted Chunkr API shuts down on November 15, 2026, while the AGPL-3.0 repository remains available.
Crawl4AI
crawl4ai.com
Crawl4AI is an Apache-2.0 Python crawler and scraper that drives a headless browser to turn pages into clean Markdown or structured JSON for LLMs, with Docker deployment and a hosted Crawl4AI Cloud API.
Docling
LF AI & Data Foundation · docling.ai
Docling is an MIT-licensed document conversion toolkit, started at IBM Research and hosted by the LF AI & Data Foundation, that converts PDFs, Office files, HTML, and images into a unified document model with layout, reading order, tables, and formulas.
Extend
extend.ai
Extend is a document processing platform for AI agents that parses, extracts, and splits documents with specialized vision models, and it is acquiring Chunkr, whose hosted API customers are migrating to Extend.

Firecrawl
firecrawl.dev
Firecrawl is a web data API for AI agents that searches, scrapes, crawls, and maps websites into clean Markdown or structured JSON, available as a hosted service or self-hosted from its AGPL-3.0 repository.

Google Document AI
Google Cloud · cloud.google.com
Document AI is Google Cloud's document processing suite with pretrained processors for OCR, layout parsing, forms, and common document types, Workbench for training custom extractors, and a layout parser that Google's RAG Engine can use for ingestion.

Jina Reader
Elastic · jina.ai
Jina Reader converts a web page to LLM-friendly Markdown when its URL is prefixed with r.jina.ai, and its s.jina.ai endpoint returns cleaned content from web search results, with Apache-2.0 source available for self-hosting.
Elastic acquired Jina AI in October 2025; Reader remains available at jina.ai.
LandingAI Agentic Document Extraction
LandingAI · landing.ai
LandingAI's Agentic Document Extraction is an API that parses documents into Markdown and structured chunks with visual grounding, then extracts fields against a schema, with an Apache-2.0 command-line client.

LiteParse
LlamaIndex · developers.llamaindex.ai
LiteParse is LlamaIndex's open-source document parser that runs locally without a vision-language model, extracting text with bounding boxes and page screenshots from PDFs, Office files, and images, with optional Tesseract OCR and a command-line tool.

LlamaParse
LlamaIndex · llamaindex.ai
LlamaParse is LlamaIndex's agentic document processing platform that parses PDFs, spreadsheets, images, and slides into structured output with vision-language models, with Extract and Index services for schema-based extraction and retrieval.
LlamaIndex renamed its LlamaCloud platform to LlamaParse in February 2026, and the llama-cloud SDKs replaced the deprecated llama-cloud-services packages.
Marker
Datalab · datalab.to
Marker is Datalab's open-source converter that turns PDFs, images, and Office and EPUB files into Markdown, JSON, or HTML using its Surya OCR and layout models, with a hosted Datalab API that also serves the Chandra OCR model.

MarkItDown
Microsoft · github.com
MarkItDown is Microsoft's MIT-licensed Python tool and command-line utility that converts PDF, Word, PowerPoint, Excel, images, audio, HTML, and other files to Markdown for LLM input, with an optional MCP server package.
MinerU
OpenDataLab · mineru.net
MinerU is an open-source tool from OpenDataLab that converts PDFs, Office documents, images, and web pages into Markdown or JSON with layout, table, and formula recognition, offered as a CLI, Python library, and hosted service.
Mistral OCR
Mistral AI · docs.mistral.ai
Mistral OCR is Mistral AI's document understanding model, at version 3 since December 2025, that extracts text, tables as HTML, embedded images, and handwriting into structured output through the API and the Document AI playground in Mistral AI Studio.

olmOCR
Ai2 · github.com
olmOCR is Ai2's Apache-2.0 toolkit for converting PDFs and images into clean Markdown with olmOCR-2, a fine-tuned 7B vision-language model, run through vLLM for batch processing of large document collections.

PaddleOCR
Baidu (PaddlePaddle) · paddleocr.com
PaddleOCR is an Apache-2.0 OCR and document parsing toolkit from Baidu's PaddlePaddle team, with PP-OCRv5 text recognition, PP-StructureV3 for layout and tables, and the PaddleOCR-VL vision-language model for converting documents to structured data.

PyMuPDF4LLM
Artifex · pymupdf.readthedocs.io
PyMuPDF4LLM is a Python package built on PyMuPDF that extracts PDF and other document content as Markdown, JSON, or text for LLM and RAG pipelines, licensed under AGPL-3.0 with commercial licenses available from Artifex.

Reducto
reducto.ai
Reducto is a document processing API that parses complex layouts and tables with bounding-box citations, and adds extraction, splitting, classification, and editing endpoints for building document pipelines.
ScrapeGraphAI
scrapegraphai.com
ScrapeGraphAI is an MIT-licensed Python library that builds LLM-driven scraping pipelines from a natural-language prompt and a URL and returns structured data, with a hosted API from the same team for production use.
Spider
spider.cloud
Spider is an MIT-licensed web crawler written in Rust for high-concurrency crawling and scraping, with the spider.cloud hosted API returning pages as Markdown, text, or HTML for agents and RAG pipelines.

Trafilatura
trafilatura.readthedocs.io
Trafilatura is an Apache-2.0 Python package and command-line tool that crawls web pages and extracts main text, comments, and metadata while removing boilerplate, with output as plain text, Markdown, JSON, XML, or CSV.

Unstract
Zipstack · unstract.com
Unstract is an AGPL-3.0 platform for LLM-driven document extraction where teams define prompts in Prompt Studio and deploy them as APIs or ETL pipelines, alongside the LLMWhisperer text extraction service and an MCP server.
Unstructured
unstructured.io
Unstructured provides an Apache-2.0 Python library that partitions PDFs, Office files, HTML, images, and other formats into typed elements, plus a managed platform that runs parsing, chunking, and embedding pipelines through an API or MCP server.

Xberg
xberg.io
Xberg is an MIT-licensed document intelligence engine with a Rust core and bindings for Python, Node.js, Go, Java, and other languages that extracts text, tables, and metadata from PDFs, Office files, images, email, and archives, with built-in OCR.
Xberg is the renamed successor to Kreuzberg, which was previously published under the kreuzberg package name and keeps a separate v4 long-term support repository.
How to choose
- 01Input coverage
- Supported formats, from native and scanned PDFs to Office files, images, HTML, email, and audio.
- 02Layout fidelity
- Handling of reading order, multi-column pages, tables, formulas, charts, and handwriting, plus bounding boxes for citations.
- 03Output format
- Markdown, HTML, or typed JSON elements with coordinates, and whether chunking and schema-based extraction are built in.
- 04Deployment
- A managed API, a self-hosted service, or a local library, which determines data residency and whether documents leave your network.
- 05Compute
- Text-layer and classic OCR pipelines run on CPUs, while vision-language parsers usually need GPUs or a hosted API.
- 06License
- Permissive, copyleft such as AGPL-3.0, or proprietary terms, and any separate license that covers model weights.
Frequently asked questions
What is document parsing in RAG?
- Document parsing in RAG is the step that converts source files, such as PDFs, slides, and scans, into text and structure a retrieval pipeline can use. A parser identifies headings, paragraphs, tables, figures, and reading order, then outputs Markdown or JSON that is split into chunks and embedded. Errors at this stage, such as merged columns or lost tables, carry through to every answer.
How do you parse PDFs for an LLM?
- For PDFs with a text layer, a fast extractor such as PyMuPDF4LLM, MarkItDown, or LiteParse can produce Markdown directly. Scanned pages, complex tables, and charts need OCR or layout models, which tools such as Docling, Marker, MinerU, and olmOCR run locally, and which LlamaParse, Reducto, Mistral OCR, and the cloud document services offer as APIs. Compare outputs on a sample of your own documents.
What is the best OCR for LLMs?
- No single OCR option suits every document set. Vision-language OCR models, such as Mistral OCR, olmOCR, PaddleOCR-VL, and Datalab's Chandra, read layouts and tables into Markdown, while classic engines such as Tesseract and PP-OCR are lighter and run on CPUs. Cloud services from Azure, AWS, and Google add forms and key-value extraction. Test candidates on representative scans, including handwriting and tables.
What is the difference between Docling and LlamaParse?
- Docling is an MIT-licensed toolkit that runs on your own machines and converts documents into a structured document model using local layout, table, and OCR models. LlamaParse is LlamaIndex's managed platform that parses documents through an API using vision-language models, with added extraction and indexing services. Docling keeps data in your environment; LlamaParse removes the need to operate models and GPUs.
What does a web scraper for LLMs do?
- A web scraper for LLMs fetches pages, often with a headless browser to render JavaScript, removes navigation, ads, and boilerplate, and returns the main content as Markdown or structured JSON sized for a model's context window. Tools such as Firecrawl, Crawl4AI, Jina Reader, Spider, and ScrapeGraphAI also crawl whole sites, follow links, and expose APIs or MCP servers that agents can call.
Related guides
RAG and vector search hubOther categories
Product details change quickly; check each vendor’s site for current features, licensing and pricing.