Projects / agentic-file-ingestion
Coding agent for loading files whose layout keeps changing
Partner files arrive by email and their layout drifts. A known layout runs an approved loading script with no model call; a new one goes to a sandboxed coding agent, then to automated checks and a human approval.
- Period
- 2026
- Role
- Tech Lead Data & AI, working with two developers
- Context
- Large services company with legacy data systems
- Cloud
- AWS
[ Architecture ]
How it works
Run a scenario to follow it through the system, step by step. Select a component to read its role and see what it connects to.
a partner
Monthly export with a renamed sheet
The same export as last month, but a sheet and two column headers were renamed.
export.xlsx
- Press Send a file with a new layout to watch each step run through the system.
- 01PartnerSends the monthly file
- 02Mailbox & queueFile stored unchanged · work item queued
- 03Layout recogniserSignature differs from every approved loader
- 04Coding agentCandidate loader and tests written from a sample
- 05Checks & approvalChecks pass · operator approves · file replayed
- 06PublisherPublished atomically · lineage written
- ✓The file is published and the new layout becomes an approved loader, so next time it needs no model.
- 01PartnerSends the monthly file
- 02Mailbox & queueFile stored unchanged · work item queued
- 03Layout recogniserSignature matches an approved loader · no model call
- 04PublisherPublished atomically · lineage written
- ✓The file is loaded and published with no model call.
- 01PartnerSends the monthly file
- 02Mailbox & queueFile stored unchanged · work item queued
- 03Layout recogniserRequired column missing · stopped before any model call
- ✓Nothing is loaded and no column is invented. The file is stopped, and the operator is told which column is missing.
Problem
Partner files arrive by email every day, and their layout drifts: a renamed sheet, translated headers, a column added in the middle. The legacy tool reads columns by position, so a change either breaks the load or shifts values without warning.
Data
Sample partner files, the expected table contracts (the structure each table must have) and the list of frequent load failures. Nothing leaves the client’s environment: the agent has no network access and its model calls go through the client’s own gateway.
Approach
- Each file is identified, its structure is read (sheet, headers, columns) and compared with the approved loaders. A known layout runs its loader with no model call.
- Simple rules run before the model. An extra optional column is loaded with a warning, missing required columns stop the file with a clear reason, and only a layout that can be adapted safely goes to the agent.
- The coding agent runs in an isolated container with limited time. It receives the file, the expected contract and the current loader, and returns a proposed loader, its tests and a summary of its choices.
- The platform runs the checks, not the agent: code rules, declared structure, tests, an isolated trial load and a replay on comparable files already published. A person approves, and only the exact version that passed is promoted.
Evaluation
Test files with renamed sheets, extra or missing columns and formulas cover each of these cases. Every run keeps the source file, the contract, the loader and the checks it used, and a fingerprint of the source file prevents publishing the same file twice.
Result
A known layout loads with no model call, and a layout change becomes a reviewed loader instead of a broken load. The operator sees every file, its status and the next action in one place.
My role
Tech lead with two developers: design of the ingestion flow and of the rules that run before the model, the operator app and the deployment on the client’s Kubernetes platform.