Projects / agentic-file-ingestion
Agentic file ingestion and legacy ETL migration
Retiring a legacy ETL estate in favour of PySpark while keeping lineage, and an ingestion flow where a sandboxed agent writes loaders for unseen file layouts.
- Period
- 2026
- Role
- Tech Lead Data & AI, working with two developers
- Context
- Large services company with a legacy data estate
- Cloud
- AWS
[ Architecture ]
How it works
Run a scenario to follow a request through the system. Select a component to read its role and its flows.
a partner
Monthly export with a renamed sheet
The same export as last month, but a sheet and two column headers were renamed.
export.xlsx
- Press Send a file with a new layout to watch the request go through the system.
- 01PartnerSends the monthly file
- 02Mailbox & queueFile stored unchanged · work item queued
- 03Layout recogniserSignature differs from every approved loader
- 04Coding agentCandidate loader and tests written from a sample
- 05Checks & approvalChecks pass · operator approves · file replayed
- 06PublisherPublished atomically · lineage written
- ✓The file is published and the new layout becomes an approved loader, so next time it needs no model.
- 01PartnerSends the monthly file
- 02Mailbox & queueFile stored unchanged · work item queued
- 03Layout recogniserSignature matches an approved loader · no model call
- 04PublisherPublished atomically · lineage written
- ✓The file is loaded and published with no model call.
Problem
A legacy ETL tool has to be retired in favour of PySpark on the cloud without losing track of which jobs feed which tables. Separately, customer files arrive by email with layouts that change, and each change breaks a load.
Data
Legacy job definitions, the table catalogue, frozen output baselines and sample customer files. Nothing leaves the client’s environment.
Approach
- Jobs are parsed into a lineage between jobs and tables, then PySpark code is generated from it and completed by an agent.
- A migrated job is accepted when its outputs match the frozen baseline.
- A known file layout runs an approved loader with no model call; an unknown one goes to a sandboxed coding agent, then to platform checks and human approval.
Evaluation
The inventory, a transformation proof of concept and the first ingestion flow were demonstrated to the client.
Result
More than a thousand legacy jobs were inventoried and their lineage rebuilt, giving the client a map of the estate that did not exist before.
My role
Tech lead with two developers: roadmap, lineage and acceptance methods, ingestion flow design, a first operator app and the cloud architecture proposal.