Multithreading support

[abhishek-gh7]

Hey, Just thinking if unstructured.io have a feature to support multithreading

For eg: Large pdf and docx takes so much time,
There should be support for multithreading which can for eg split pages in pdf to threads and later combine it into one, For docx we don't have page numbers but paragraphs, we can do same for paragraphs

Just an idea
Thanks

[cragwolfe]

@abhishek-gh7 , your suggestion certainly makes sense and it's on our radar to parallelize the processing of pages, esp. in the PDF case. We've just been more focused on quality of outputs thus far. We'll be paying closer attention to performance (esp. latency) in the weeks and months ahead.

[muazhari]

are there any updates about that?

[muazhari]

A year already, any updates?

[kinthaiofficial]

Multithreading for document processing is useful but requires care because of how the underlying dependencies handle parallelism.

What works well with threads:

from concurrent.futures import ThreadPoolExecutor
from unstructured.partition.auto import partition
import threading

# Thread-safe: each call gets its own file handle + return value
def process_file(filepath: str) -> list:
    return partition(filepath)

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = [executor.submit(process_file, p) for p in file_paths]
    results = [f.result() for f in futures]

Where it gets tricky:

• PDF processing (pdfminer, tesseract) has global state issues in some versions
• ONNX models for table detection aren't always thread-safe
• partition_pdf(strategy="hi_res") with OCR can deadlock under some configurations

Multiprocessing is often safer for CPU-bound work:

from multiprocessing import Pool

with Pool(processes=4) as pool:
    results = pool.map(process_file, file_paths)

Async for I/O-bound pipelines (e.g., reading from S3, then processing):

import asyncio

async def process_batch(paths: list[str]):
    # Download async, process sync in threadpool
    loop = asyncio.get_event_loop()
    tasks = [
        loop.run_in_executor(None, partition, path)
        for path in paths
    ]
    return await asyncio.gather(*tasks)

For our document ingestion pipeline (feeding agent knowledge bases), we use a process pool for PDF/image OCR and threads for plain text/HTML. The split keeps OCR parallelism safe while avoiding multiprocessing overhead for lighter files.