Chunking strategies and metadata parsing for complex structured PDFs

A PDF isn't really "text with a fixed size" — it's titles, paragraphs, tables, and list items laid out on a page, and a chunker that ignores that structure will regularly slice a table in half or merge a footer into the middle of a paragraph. The Unstructured library handles this by first detecting document elements (title, narrative text, table, list item) with real metadata — page number, coordinates, element type — and only chunking within or across those elements afterward.

A structure-aware chunking pipeline

1. Partition the document into elements first. Unstructured's partition_pdf function returns a list of typed elements (titles, tables, narrative text) instead of one undifferentiated text blob.

2. Chunk by element boundaries, not raw character counts. Combine small consecutive elements up to your target chunk size, but never split a table element in the middle — tables should stay intact or be summarized as a unit.

3. Carry metadata through to the vector store. Page number and element type should be stored alongside the chunk's text, so a later query can filter to "just tables" or cite the exact page a claim came from.

# Structure-aware PDF parsing and chunking with Unstructured
from unstructured.partition.pdf import partition_pdf
from unstructured.chunking.title import chunk_by_title

# 1. Partition into typed elements (Title, NarrativeText, Table, ...)
elements = partition_pdf(filename="quarterly_report.pdf")

# 2. Chunk by title boundaries, respecting table/element structure
chunks = chunk_by_title(elements, max_characters=1000)

for chunk in chunks:
    # Each chunk carries page number and element metadata
    print(chunk.metadata.page_number, chunk.text[:80])

The metadata is what makes this worth the extra setup — once every chunk carries its page number and element type, your RAG system can cite "page 14, table 2" instead of just returning a wall of retrieved text with no provenance.