Case study · Professional work · IDA Automation

Document Intelligence

Turning scanned property records and hand-marked parcel maps into structured, verifiable data.

Described at a high level. No client documents, internal systems or proprietary detail are shown.

Explore pipeline workflow ↓ Interactive crop demo ↓

Role: AI/ML Engineer
Documents: Mortgage, title, ownership, parcel maps
Models: Fine-tuned LLMs, vision-language models
Platform: FastAPI on AWS Bedrock, SageMaker, EC2, S3

100+

Documents processed per day in production

~80%

Less manual processing effort

15–20%

Higher extraction accuracy than a general-purpose LLM baseline

~90%

Fewer extraction errors after fine-tuning and pipeline improvements

The Problem

Documents built for people, not parsers.

U.S. real estate records arrive scanned and inconsistent: rotated pages, stamps over text, typed and handwritten marks side by side, tables that only look like tables. A legal land description can run for paragraphs of boundary calls.

Generic OCR reads the characters and loses the meaning. A language model asked to read the whole page fills the gaps it can't see. Either way the extracted value has to be right, because something downstream depends on it, and a reviewer has to be able to confirm it quickly.

The Approach

Narrow the page before asking a model to read it.

Specialized CV preprocessing paired with domain-fine-tuned LLMs & vision-language models.

A Document Ingestion & Computer Vision Preprocessing Flow

Computer Vision Pipeline · Ingress Normalization

📄 Scanned Records

Step 1 · Ingress Prep

Deskew & Scale

FastAPI + OpenCV deskews orientation, denoises, and normalizes to 300 DPI.

Step 2 · CV Detection

Region Segmentation

Detects bounding boxes around legal text blocks, parcel tables, and notary stamps.

Step 3 · Dispatcher

Modal Router

Dispatches text crops to Fine-Tuned LLMs, and diagram maps to CV + VLM.

🔀 Isolated Crops

B Specialized Neural Reading & Guardrails Flow

Fine-Tuned LLMs + Multimodal VLMs

🔀 Isolated Crops

Step 4 · Domain LLM

Text Extraction

Fine-tuned models extract grantors, PINs, and metes & bounds into schema JSON.

Step 5 · Spatial VLM

Parcel Geometry

CV isolates polygon boundaries; VLM reads lot numbers & compass bearings.

Step 6 · Pydantic Guard

Schema Validation

Rigid type validation & legal regex checks reject malformed extractions.

Step 7 · Audit Loop

Human Verification

Low-confidence fields surface alongside the exact crop for 2-second audit.

💾 Structured JSON

A Document Ingestion & Computer Vision Preprocessing Specifications

3 Pipeline Stages · Ingress Normalization

01 Ingestion & Deskew

OpenCV Normalizer

FastAPI on AWS receives scans; OpenCV deskews orientation, denoises speckle, and scales DPI.

Engineering Rationale Deskewing and normalizing contrast once at ingress eliminates orientation hallucinations inside downstream model calls.

Input / Output Boundary Multi-page TIFF, PDF, JPEG scans → Normalized 300 DPI oriented image arrays.

Safeguard / Metric 100+ documents/day processed with zero orientation failure retries.

02 Region Segmentation

CV Detection

Computer vision locates legal text blocks, boundary tables, parcel polygons, and notary stamps.

Engineering Rationale Passing whole complex pages into LLMs leads to hallucinated fields. Isolating text blocks produces high-precision crops.

Input / Output Boundary Full page image → Bounding box coordinate arrays with region classification tags.

Safeguard / Metric Crops page into targeted sub-sections; eliminates 90% background noise.

03 Modal Dispatcher

Routing Gate

Categorizes crops: legal deeds go to Fine-Tuned LLMs; maps go to Classical CV + VLM.

Engineering Rationale Legal deeds require linguistic reasoning; maps require spatial visual reasoning. Routing each region optimizes accuracy and cost.

Input / Output Boundary Region crops → Dispatched to Fine-Tuned LLMs OR Multimodal VLM reader.

Safeguard / Metric Zero wasted LLM calls on visual diagrammatic areas.

B Specialized Neural Reading & Guardrails Specifications

4 Pipeline Stages · LLM + VLM Inference

04 Text Field Extraction

Fine-Tuned LLM

Fine-tuned models extract metes & bounds, PINs, and grantors with ~15–20% higher accuracy than baseline.

Engineering Rationale Generalist LLMs struggle with archaic real estate legalese. Models fine-tuned on county records extract clean schema fields.

Input / Output Boundary Legal text crops → Structured JSON (Grantor, Grantee, PIN, Legal Land Description).

Safeguard / Metric 15–20% higher extraction accuracy than general-purpose LLM baselines.

05 Parcel Map Geometry

Classical CV + VLM

Classical CV isolates parcel boundaries; vision-language models read lot numbers and compass rose orientation.

Engineering Rationale Classical CV accurately detects polygon vertices and frontages; a VLM reads lot numbers and compass rose orientations.

Input / Output Boundary Assessor parcel map crops → Vector geometry coordinates + orientation angle.

Safeguard / Metric Automates hand-marked map indexing that previously required manual surveying.

06 Schema Validation

Pydantic Guard

Rigid type validation & domain rule checks reject malformed extractions, cutting errors by ~90%.

Engineering Rationale Rigid validation ensures bad extractions fail at the pipeline boundary rather than corrupting downstream title registries.

Input / Output Boundary Structured JSON candidate → Strict Pydantic validated models.

Safeguard / Metric ~90% fewer extraction errors escaping into downstream systems.

07 Human-in-the-Loop

Review Loop

Low-confidence predictions surface beside the exact document crop for 2-second reviewer verification.

Engineering Rationale Predictions below strict confidence thresholds surface side-by-side with original crop, enabling instant human verification.

Input / Output Boundary Flagged fields + document crop → 1-click human verification console.

Safeguard / Metric Cuts manual processing effort by ~80% while maintaining 100% legal auditability.

Interactive Demo

Verifiable extraction with bounding crops.

Click any structured field to highlight its localized bounding region on the record.

● Real Estate Deed Extraction Simulator AWS Bedrock + Classical CV

RECORDED DEED

WARRANTY DEED: COUNTY OF TRAVIS

THIS INDENTURE, made this 12th day of September, 2024, between Apex Realty Holdings, a Delaware LLC, ("Grantor"), and Highland Ridge Investment Group, ("Grantee").

WITNESSETH, that Grantor, for consideration paid, does hereby grant and convey with WARRANTY COVENANTS unto Grantee, all that parcel: Lot 14, Block 7, Oak Ridge Subdivision, Section 3, Recorded in Plat Book 42, Page 118, with property tax PIN: 48-120-994-00.

Subject to easements and restrictions of record. Bearings referenced to: North-Up (Grid N 14° 22' E).

Grantor Entity

Apex Realty Holdings LLC

99.8% Conf

Legal Land Description

Lot 14, Block 7, Oak Ridge Subdiv.

99.4% Conf

Parcel Identification Number (PIN)

48-120-994-00

99.9% Conf

Grantee Entity

Highland Ridge Investment Group

99.1% Conf

Parcel Map Orientation

Grid N 14° 22' E (North-Up)

98.5% Conf

Specialized Pipelines

Text records and maps need different readers.

Property records

Field extraction with fine-tuned LLMs

Extracts, validates and structures fields such as legal land descriptions from mortgage, title and ownership documents. LLMs fine-tuned on real estate data outperform general-purpose models on these fields, and RAG with semantic chunking supplies context across large document sets.

Assessor parcel maps

Computer vision + a vision-language model

Classical computer vision isolates parcel geometry and markings; a vision-language model reads them back, including the map's compass orientation. Each tool does the part it's good at.

Engineering Decisions

What makes it hold up.

Vision first, language second

Isolating the region first turns an open-ended reading task into a narrow one: more accurate and cheaper per page.

Fine-tune for the domain

Real estate language is specialised. Fine-tuned models extract ~15–20% more accurately than a general-purpose LLM baseline.

Schema at the boundary

Typed, validated outputs mean a bad extraction fails at the pipeline edge, not three systems later.

Context over guesswork

Model and pipeline optimisation cut unsupported or hallucinated outputs, with retrieval supplying the surrounding record.

Have a document-heavy workflow to automate?

Extraction that holds up, with a review path a person can trust.

Email me Read: Knowra →