Document parsing and data ingestion

Data ingestion tools turn PDFs, Office files, scans, images, and web pages into clean text, Markdown, or structured JSON that a RAG pipeline can chunk, embed, and cite. Parsing quality sets a ceiling on retrieval quality: a table read in the wrong order or a dropped footnote cannot be recovered later in the pipeline.

The category spans open document parsers and OCR toolkits, such as Docling, Marker, MinerU, and PaddleOCR; managed document APIs, such as LlamaParse, Reducto, Mistral OCR, and the Azure, AWS, and Google Cloud services; and web extraction tools, such as Firecrawl, Crawl4AI, Jina Reader, and Apify, that crawl sites and return LLM-ready Markdown.

Many open-source options use copyleft licenses such as AGPL-3.0, and several ship model weights under separate terms, so check both the code and the model licenses before deploying.

Document parsing and data ingestion compared

PlatformOpen sourceDeploymentBest for
Amazon TextractNoManaged cloud, APIDocument extraction inside AWS workflows
Apache TikaYes (Apache-2.0)Library, Self-hosted, CLIBroad file-format coverage in Java pipelines
ApifyNoManaged cloud, APIReady-made scrapers for specific websites
Azure Document IntelligenceNoManaged cloud, APIForm and document extraction on Azure
ChunkrYes (AGPL-3.0)API, Self-hostedSelf-hosted layout analysis and chunking
Crawl4AIYes (Apache-2.0)Library, Self-hosted, Managed cloudSelf-hosted crawling for LLM pipelines
DoclingYes (MIT)Library, Self-hosted, LocalSelf-hosted parsing that preserves document structure
ExtendNoAPI, Managed cloudProduction document extraction pipelines
FirecrawlYes (AGPL-3.0)Managed cloud, API, Self-hostedTurning websites into LLM-ready Markdown
Google Document AINoManaged cloud, APIDocument processing on Google Cloud
Jina ReaderYes (Apache-2.0)API, Self-hostedQuick URL-to-Markdown conversion with no setup
LandingAI Agentic Document ExtractionNoAPI, Managed cloudSchema-based extraction with visual grounding
LiteParseYes (Apache-2.0)Library, CLI, LocalLocal parsing without GPUs or API calls
LlamaParseNoManaged cloud, APIComplex layouts, tables, and charts in RAG pipelines
MarkerYes (Apache-2.0 (code); modified OpenRAIL-M (model weights))Library, CLI, API, Managed cloudConverting PDFs to Markdown locally or by API
MarkItDownYes (MIT)Library, CLILightweight conversion of Office files to Markdown
MinerUYes (MinerU Open Source License (Apache-2.0 with additional terms))Library, CLI, Self-hosted, Managed cloudSelf-hosted PDF conversion with formula and table support
Mistral OCRNoAPI, Self-hostedAPI-based OCR for scanned and handwritten documents
olmOCRYes (Apache-2.0)Library, Self-hostedBatch OCR of large PDF collections on your own GPUs
PaddleOCRYes (Apache-2.0)Library, Self-hostedMultilingual OCR that runs on your own hardware
PyMuPDF4LLMYes (AGPL-3.0)LibraryText-layer PDF extraction in Python
ReductoNoAPI, Managed cloudParsing complex enterprise documents through an API
ScrapeGraphAIYes (MIT)Library, API, Managed cloudPrompt-based extraction of structured web data
SpiderYes (MIT)Library, API, Managed cloudLarge crawls where throughput matters
TrafilaturaYes (Apache-2.0)Library, CLIExtracting article text from web pages at scale
UnstractYes (AGPL-3.0)Self-hosted, Managed cloudPrompt-defined extraction deployed as APIs or ETL jobs
UnstructuredYes (Apache-2.0)Library, Managed cloud, APIConverting mixed document types into RAG-ready elements
XbergYes (MIT)Library, CLI, Self-hosted, Managed cloudPolyglot document extraction in self-hosted pipelines

Profiles

  • Amazon Textract logo

    Amazon Textract

    Amazon Web Services · aws.amazon.com

    Amazon Textract is an AWS service that uses OCR and machine learning to extract printed text, handwriting, forms, tables, and answers to natural-language queries from scanned documents, returning results with confidence scores and coordinates.

    • Managed cloud
    • API
  • Apache Tika logo

    Apache Tika

    Apache Software Foundation · tika.apache.org

    Apache Tika is a Java toolkit that detects and extracts text and metadata from more than a thousand file types, available as a library, a REST server, and a command-line app, and often used as the parsing stage in search and RAG ingestion.

    • Open source · Apache-2.0
    • Library
    • Self-hosted
    • CLI
    • Java
  • Apify logo

    Apify

    apify.com

    Apify is a web scraping and automation platform with a marketplace of ready-made scrapers called Actors, an MCP server for agents, and the Apache-2.0 Crawlee library for building crawlers in Node.js and Python.

    • Managed cloud
    • API
    • TypeScript
    • Python
  • Azure Document Intelligence logo

    Azure Document Intelligence

    Microsoft · azure.microsoft.com

    Azure Document Intelligence extracts text, key-value pairs, tables, and fields from documents with prebuilt and custom models, and Azure Content Understanding extends it with LLM-based analyzers for documents, images, audio, and video.

    Microsoft now presents Document Intelligence as part of Azure Content Understanding in Foundry Tools, the new name for Azure AI services.

    • Managed cloud
    • API
  • Chunkr logo

    Chunkr

    Lumina AI · chunkr.ai

    Chunkr is a document intelligence API and AGPL-3.0 self-hostable service for layout analysis, OCR with bounding boxes, and semantic chunking that converts PDFs, Office files, and images into HTML, Markdown, or JSON.

    Chunkr is being acquired by Extend; the hosted Chunkr API shuts down on November 15, 2026, while the AGPL-3.0 repository remains available.

    • Open source · AGPL-3.0
    • API
    • Self-hosted
    • Rust
  • Crawl4AI logo

    Crawl4AI

    crawl4ai.com

    Crawl4AI is an Apache-2.0 Python crawler and scraper that drives a headless browser to turn pages into clean Markdown or structured JSON for LLMs, with Docker deployment and a hosted Crawl4AI Cloud API.

    • Open source · Apache-2.0
    • Library
    • Self-hosted
    • Managed cloud
    • Python
  • Docling logo

    Docling

    LF AI & Data Foundation · docling.ai

    Docling is an MIT-licensed document conversion toolkit, started at IBM Research and hosted by the LF AI & Data Foundation, that converts PDFs, Office files, HTML, and images into a unified document model with layout, reading order, tables, and formulas.

    • Open source · MIT
    • Library
    • Self-hosted
    • Local
    • Python
  • Extend logo

    Extend

    extend.ai

    Extend is a document processing platform for AI agents that parses, extracts, and splits documents with specialized vision models, and it is acquiring Chunkr, whose hosted API customers are migrating to Extend.

    • API
    • Managed cloud
  • Firecrawl logo

    Firecrawl

    firecrawl.dev

    Firecrawl is a web data API for AI agents that searches, scrapes, crawls, and maps websites into clean Markdown or structured JSON, available as a hosted service or self-hosted from its AGPL-3.0 repository.

    • Open source · AGPL-3.0
    • Managed cloud
    • API
    • Self-hosted
    • TypeScript
    • Python
  • Google Document AI logo

    Google Document AI

    Google Cloud · cloud.google.com

    Document AI is Google Cloud's document processing suite with pretrained processors for OCR, layout parsing, forms, and common document types, Workbench for training custom extractors, and a layout parser that Google's RAG Engine can use for ingestion.

    • Managed cloud
    • API
  • Jina Reader logo

    Jina Reader

    Elastic · jina.ai

    Jina Reader converts a web page to LLM-friendly Markdown when its URL is prefixed with r.jina.ai, and its s.jina.ai endpoint returns cleaned content from web search results, with Apache-2.0 source available for self-hosting.

    Elastic acquired Jina AI in October 2025; Reader remains available at jina.ai.

    • Open source · Apache-2.0
    • API
    • Self-hosted
    • TypeScript
  • LandingAI Agentic Document Extraction

    LandingAI · landing.ai

    LandingAI's Agentic Document Extraction is an API that parses documents into Markdown and structured chunks with visual grounding, then extracts fields against a schema, with an Apache-2.0 command-line client.

    • API
    • Managed cloud
  • LiteParse logo

    LiteParse

    LlamaIndex · developers.llamaindex.ai

    LiteParse is LlamaIndex's open-source document parser that runs locally without a vision-language model, extracting text with bounding boxes and page screenshots from PDFs, Office files, and images, with optional Tesseract OCR and a command-line tool.

    • Open source · Apache-2.0
    • Library
    • CLI
    • Local
  • LlamaParse logo

    LlamaParse

    LlamaIndex · llamaindex.ai

    LlamaParse is LlamaIndex's agentic document processing platform that parses PDFs, spreadsheets, images, and slides into structured output with vision-language models, with Extract and Index services for schema-based extraction and retrieval.

    LlamaIndex renamed its LlamaCloud platform to LlamaParse in February 2026, and the llama-cloud SDKs replaced the deprecated llama-cloud-services packages.

    • Managed cloud
    • API
    • Python
    • TypeScript
  • Marker

    Datalab · datalab.to

    Marker is Datalab's open-source converter that turns PDFs, images, and Office and EPUB files into Markdown, JSON, or HTML using its Surya OCR and layout models, with a hosted Datalab API that also serves the Chandra OCR model.

    • Open source · Apache-2.0 (code); modified OpenRAIL-M (model weights)
    • Library
    • CLI
    • API
    • Managed cloud
    • Python
  • MarkItDown logo

    MarkItDown

    Microsoft · github.com

    MarkItDown is Microsoft's MIT-licensed Python tool and command-line utility that converts PDF, Word, PowerPoint, Excel, images, audio, HTML, and other files to Markdown for LLM input, with an optional MCP server package.

    • Open source · MIT
    • Library
    • CLI
    • Python
  • MinerU logo

    MinerU

    OpenDataLab · mineru.net

    MinerU is an open-source tool from OpenDataLab that converts PDFs, Office documents, images, and web pages into Markdown or JSON with layout, table, and formula recognition, offered as a CLI, Python library, and hosted service.

    • Open source · MinerU Open Source License (Apache-2.0 with additional terms)
    • Library
    • CLI
    • Self-hosted
    • Managed cloud
    • Python
  • Mistral OCR logo

    Mistral OCR

    Mistral AI · docs.mistral.ai

    Mistral OCR is Mistral AI's document understanding model, at version 3 since December 2025, that extracts text, tables as HTML, embedded images, and handwriting into structured output through the API and the Document AI playground in Mistral AI Studio.

    • API
    • Self-hosted
  • olmOCR logo

    olmOCR

    Ai2 · github.com

    olmOCR is Ai2's Apache-2.0 toolkit for converting PDFs and images into clean Markdown with olmOCR-2, a fine-tuned 7B vision-language model, run through vLLM for batch processing of large document collections.

    • Open source · Apache-2.0
    • Library
    • Self-hosted
    • Python
  • PaddleOCR logo

    PaddleOCR

    Baidu (PaddlePaddle) · paddleocr.com

    PaddleOCR is an Apache-2.0 OCR and document parsing toolkit from Baidu's PaddlePaddle team, with PP-OCRv5 text recognition, PP-StructureV3 for layout and tables, and the PaddleOCR-VL vision-language model for converting documents to structured data.

    • Open source · Apache-2.0
    • Library
    • Self-hosted
    • Python
  • PyMuPDF4LLM logo

    PyMuPDF4LLM

    Artifex · pymupdf.readthedocs.io

    PyMuPDF4LLM is a Python package built on PyMuPDF that extracts PDF and other document content as Markdown, JSON, or text for LLM and RAG pipelines, licensed under AGPL-3.0 with commercial licenses available from Artifex.

    • Open source · AGPL-3.0
    • Library
    • Python
  • Reducto logo

    Reducto

    reducto.ai

    Reducto is a document processing API that parses complex layouts and tables with bounding-box citations, and adds extraction, splitting, classification, and editing endpoints for building document pipelines.

    • API
    • Managed cloud
    • Python
    • TypeScript
  • ScrapeGraphAI logo

    ScrapeGraphAI

    scrapegraphai.com

    ScrapeGraphAI is an MIT-licensed Python library that builds LLM-driven scraping pipelines from a natural-language prompt and a URL and returns structured data, with a hosted API from the same team for production use.

    • Open source · MIT
    • Library
    • API
    • Managed cloud
    • Python
  • Spider logo

    Spider

    spider.cloud

    Spider is an MIT-licensed web crawler written in Rust for high-concurrency crawling and scraping, with the spider.cloud hosted API returning pages as Markdown, text, or HTML for agents and RAG pipelines.

    • Open source · MIT
    • Library
    • API
    • Managed cloud
    • Rust
  • Trafilatura logo

    Trafilatura

    trafilatura.readthedocs.io

    Trafilatura is an Apache-2.0 Python package and command-line tool that crawls web pages and extracts main text, comments, and metadata while removing boilerplate, with output as plain text, Markdown, JSON, XML, or CSV.

    • Open source · Apache-2.0
    • Library
    • CLI
    • Python
  • Unstract logo

    Unstract

    Zipstack · unstract.com

    Unstract is an AGPL-3.0 platform for LLM-driven document extraction where teams define prompts in Prompt Studio and deploy them as APIs or ETL pipelines, alongside the LLMWhisperer text extraction service and an MCP server.

    • Open source · AGPL-3.0
    • Self-hosted
    • Managed cloud
    • Python
  • Unstructured logo

    Unstructured

    unstructured.io

    Unstructured provides an Apache-2.0 Python library that partitions PDFs, Office files, HTML, images, and other formats into typed elements, plus a managed platform that runs parsing, chunking, and embedding pipelines through an API or MCP server.

    • Open source · Apache-2.0
    • Library
    • Managed cloud
    • API
    • Python
  • Xberg logo

    Xberg

    xberg.io

    Xberg is an MIT-licensed document intelligence engine with a Rust core and bindings for Python, Node.js, Go, Java, and other languages that extracts text, tables, and metadata from PDFs, Office files, images, email, and archives, with built-in OCR.

    Xberg is the renamed successor to Kreuzberg, which was previously published under the kreuzberg package name and keeps a separate v4 long-term support repository.

    • Open source · MIT
    • Library
    • CLI
    • Self-hosted
    • Managed cloud
    • Rust
    • Python
    • TypeScript

How to choose

01Input coverage
Supported formats, from native and scanned PDFs to Office files, images, HTML, email, and audio.
02Layout fidelity
Handling of reading order, multi-column pages, tables, formulas, charts, and handwriting, plus bounding boxes for citations.
03Output format
Markdown, HTML, or typed JSON elements with coordinates, and whether chunking and schema-based extraction are built in.
04Deployment
A managed API, a self-hosted service, or a local library, which determines data residency and whether documents leave your network.
05Compute
Text-layer and classic OCR pipelines run on CPUs, while vision-language parsers usually need GPUs or a hosted API.
06License
Permissive, copyleft such as AGPL-3.0, or proprietary terms, and any separate license that covers model weights.

Frequently asked questions

What is document parsing in RAG?

Document parsing in RAG is the step that converts source files, such as PDFs, slides, and scans, into text and structure a retrieval pipeline can use. A parser identifies headings, paragraphs, tables, figures, and reading order, then outputs Markdown or JSON that is split into chunks and embedded. Errors at this stage, such as merged columns or lost tables, carry through to every answer.

How do you parse PDFs for an LLM?

For PDFs with a text layer, a fast extractor such as PyMuPDF4LLM, MarkItDown, or LiteParse can produce Markdown directly. Scanned pages, complex tables, and charts need OCR or layout models, which tools such as Docling, Marker, MinerU, and olmOCR run locally, and which LlamaParse, Reducto, Mistral OCR, and the cloud document services offer as APIs. Compare outputs on a sample of your own documents.

What is the best OCR for LLMs?

No single OCR option suits every document set. Vision-language OCR models, such as Mistral OCR, olmOCR, PaddleOCR-VL, and Datalab's Chandra, read layouts and tables into Markdown, while classic engines such as Tesseract and PP-OCR are lighter and run on CPUs. Cloud services from Azure, AWS, and Google add forms and key-value extraction. Test candidates on representative scans, including handwriting and tables.

What is the difference between Docling and LlamaParse?

Docling is an MIT-licensed toolkit that runs on your own machines and converts documents into a structured document model using local layout, table, and OCR models. LlamaParse is LlamaIndex's managed platform that parses documents through an API using vision-language models, with added extraction and indexing services. Docling keeps data in your environment; LlamaParse removes the need to operate models and GPUs.

What does a web scraper for LLMs do?

A web scraper for LLMs fetches pages, often with a headless browser to render JavaScript, removes navigation, ads, and boilerplate, and returns the main content as Markdown or structured JSON sized for a model's context window. Tools such as Firecrawl, Crawl4AI, Jina Reader, Spider, and ScrapeGraphAI also crawl whole sites, follow links, and expose APIs or MCP servers that agents can call.

Other categories

Product details change quickly; check each vendor’s site for current features, licensing and pricing.