AI contract analysis (OCR & PII)

Starting point

This was the biggest project of my internship. A colleague handed it to me and asked me to upgrade it. The starting point was his repo: a mostly-Python pipeline in which a local LLM read the Tesseract output of the PDF. Unfortunately, it analysed each document in isolation and could not handle a full contract file (contract, amendments, invoices), so I decided to restart from a fresh architecture.

Rather than follow the same direction as my colleague, I first mapped the full skeleton on paper. I did not want to write a single line of code before having the whole architecture written down. I also reviewed his work and noted every part that needed an upgrade. The biggest problem was the local LLM, which was only there because of the sensitive data and GDPR restrictions: a local LLM, with or without a big harness, hallucinates a lot. The OCR was also too fragile: Tesseract is old, and we needed a reliable extraction to reduce the risk of LLM hallucinations.

Choosing the OCR

My first task was therefore to evaluate different OCR engines. I also wanted Markdown output, since LLMs tend to perform better with Markdown inputs. I chose PaddleOCR after comparing it with Tesseract on the same documents (EasyOCR was unavailable outside Python, and the single VLM-based OCR I tested was set aside mainly because of hardware). It extracted better text, and its confidence scores stayed usable, where Tesseract produced zero-confidence fragments.

Pseudonymisation in three layers

My second task was pseudonymisation. I wanted to pseudonymise the OCR output before sending it to the LLM, replacing "Mr Alan Toro" with "Mr [[PERSONNE_1]]" and keeping the mapping in an ephemeral in-memory vault that is never sent to the LLM nor persisted. To do so, I first listed all the data to pseudonymise (SIRET, SIREN, IBAN, names, companies, …), which resulted in 15 categories.

The work was split into three layers: Regex, Anchor, and NER. I chose three layers mainly for robustness: each one complements the others, with little overlap.

The Regex layer focuses on structured PII (Personally Identifiable Information) and detects 15 types. Nine of them are validated by a checksum: social security numbers (control key), company identifiers (Luhn algorithm), VAT numbers, IBANs (modulo 97), bank account numbers, bank cards, healthcare professional IDs and customs IDs. The validation is essential, because it separates a genuine identifier from a random string of digits. The six others (BIC, SEPA creditor ID, personal tax number, email, phone, address) rely on the format and, for some, on a keyword in the context. This layer is fully deterministic, so it is reproducible and can run in CI.

The Anchor layer captures party names through the stereotyped phrases that introduce them in a contract ("hereinafter referred to as", "the company X", "X, a limited company with a share capital of"), and through their position, such as signature blocks. It fills a gap of the third layer: a name in capitals or an acronym is systematically ignored by an NER.

The NER layer handles names in free text. I used CamemBERT, a French model, quantised to 8 bits in ONNX format and running locally. Three mechanisms complete the whole: a normalisation pass for identifiers split by a line break, a precedence rule for overlaps (Regex, then Anchor, then NER), and a propagation step that masks the other occurrences of a name once it has been captured reliably.

All of this follows one rule that I set at the beginning and never changed: a false negative, meaning a piece of PII left in clear, is a catastrophic failure, while a masking applied with the wrong label is acceptable. For the same reason, a failed checksum does not block the masking when the context strongly signals an identifier: an identifier damaged by the OCR must not leak. It also means that recall is measured independently of the type attributed.

Extraction and quarantine

Once the text is masked, the LLM extracts a structure from it: parties, dates, pricing clauses. The expected output follows a strict schema, and any non-conforming answer is rejected and asked again. Doubtful cases go to a quarantine queue where a lawyer can step in, instead of being approximated. A wrong amount shown with confidence costs more than ten unprocessed cases.

Deterministic computation

The main principle of the architecture is: the model understands, the code computes. No computed amount is ever produced by the LLM. Its output is a set of parameters (nature of the clause, index, deadline), and a deterministic function computes the amount. Every step is written in a hash-chained audit log, so that anyone can redo the calculation by hand.

Results

Results, measured on 320 annotated fictional cases: the deterministic layers reach an F1 of 93.8% (recall 94.6%, precision 93.0%), while the NER alone reaches 70.0% recall and 63.6% precision (on only 15 cases). The main identifiers (company, bank, social security, tax) are all detected, but recall on names of people in free text drops to 50%. The gap is not accidental: it separates what follows a verifiable rule from what depends on context. It also shows where the next effort should go: not on the regular expressions, already saturated, but on the NER layer. On the first four real contracts, the number of occurrences left in clear went from 473 to 246 after a round of fixes.

Limits

The limits are clear. I had no real data until two weeks before the end of the internship, so most of the evaluation relies on a corpus I wrote myself, and the semantic extraction was not measured. Everything also ran on my personal laptop (8 GB of RAM, no GPU), which constrained some choices, the OCR first. This is a prototype meant to lift uncertainties before any industrialisation, not a finished product.