LEARNING HUB
PREDICTIVE TREND INSIGHT
How to self-host Llama-3-Vision models for automated invoice processing Illustration

How to self-host Llama-3-Vision models for automated invoice processing

Written by Nikhil Badjatya
Direct Summary:

To self-host Llama 3.2 Vision for automated invoice processing, pull the 11B or 90B model into Ollama and send each invoice image alongside a structured-extraction prompt asking for JSON output (vendor, date, line items, totals). The 11B model needs about 8GB of VRAM; document-heavy vision models like MiniCPM-V are worth comparing since dense-text accuracy, not general image Q&A, is what invoice extraction actually depends on.

"The best way to predict the future is to invent it."

— Alan Kay

Key Insights

  • Model sizing: Llama 3.2 Vision 11B needs at least 8GB of VRAM; the 90B variant needs at least 64GB — for a single workstation, the 11B model is the realistic default.
  • Document specialists beat generalists: Models trained specifically on scanned documents and dense text (e.g. MiniCPM-V) tend to out-perform general vision-chat models like plain LLaVA on tables and invoice line items, because that's what their training data emphasized.
  • Structured output still needs validation: Asking the model for JSON doesn't guarantee correct totals — always reconcile extracted line-item sums against the invoice's stated total before trusting the output downstream.

Llama 3.2 Vision is Meta's first Llama release with real image understanding built in — not a bolt-on OCR step, but a model that takes an image and text together and reasons about both. Since November 2024 it's been available through Ollama in two sizes, and by mid-2026 it's a reasonably mature option for a very specific, high-value use case: pulling structured fields out of scanned invoices without sending vendor and pricing data to a third-party API.

Why self-hosting matters here

Invoices carry vendor names, account numbers, and pricing terms that many finance teams don't want leaving their network, even to a reputable cloud API. A local vision model removes that exposure entirely — nothing about the document ever crosses the network boundary — at the cost of running your own inference hardware and accepting a lower ceiling on raw accuracy than the largest hosted models.

Setup with Ollama

1. Install Ollama and confirm you're on version 0.4 or later, which is the minimum required for vision model support.

2. Pull the model: ollama run llama3.2-vision for the 11B model (needs at least 8GB VRAM), or ollama run llama3.2-vision:90b if you have a workstation with 64GB+ VRAM.

3. Send an invoice image with an extraction prompt that explicitly asks for structured output — the model will follow a JSON schema reasonably well if you show it the shape you want in the prompt.

extract_invoice.py 
# Extract structured fields from a scanned invoice using
# Llama 3.2 Vision served locally by Ollama
import ollama

response = ollama.chat(
    model='llama3.2-vision',
    messages=[{
        'role': 'user',
        'content': (
            'Extract vendor name, invoice date, line items '
            '(description, quantity, unit price), and the '
            'total amount due. Return valid JSON only.'
        ),
        'images': ['invoice_scan.png']
    }]
)

print(response['message']['content'])
Model Size Min. VRAM Notes for document work
Llama 3.2 Vision 11B 11B ~8GB General-purpose: OCR, charts/tables, handwriting, image Q&A
Llama 3.2 Vision 90B 90B ~64GB Same capability set, higher accuracy ceiling, much heavier hardware
MiniCPM-V (document-tuned) 8B ~6GB Trained on document scans specifically; often stronger on dense tables at a smaller size

Whichever model you pick, treat the extraction as a draft, not a ledger entry. Have the pipeline recompute the line-item sum and compare it against the invoice's printed total; a mismatch is your signal to flag the document for a human to check rather than post it automatically. That single guardrail catches the majority of extraction errors before they touch accounting data.

Practical Challenge

Pull llama3.2-vision in Ollama, run it against three sample invoices with different layouts, and write a small script that sums the extracted line items and flags any invoice where that sum doesn't match the extracted total.

Concept Check

Why might a document-specialized model like MiniCPM-V outperform Llama 3.2 Vision on invoice line items despite being a smaller model?
Correct! Model accuracy on a specific task tracks how closely the training data matches that task, not just parameter count — a smaller model tuned on document scans can beat a larger general-purpose vision model on dense tables and invoices.
Incorrect. Try again! Hint: it comes down to what the model was actually trained on, not its size — document-scan training data transfers better to invoice extraction than general photo/image-chat data does.

Sources & Further Reading

Course Home Dashboard Next Guide