AI agent for schema mapping and data onboarding

I own a real-time data platform that integrates 250+ systems at a peak of 500 transactions a second and serves four internal domains: Fraud, Risk, Servicing and ML. Every new system has to be onboarded, which means mapping its schema onto the platform's data model.

That mapping was done by hand. Analysts read each system's schema documentation, matched fields one by one and checked their work with system owners. Onboarding a single system took weeks, and the queue of systems waiting held back the teams that needed the data.

Schema mapping was the bottleneck in onboarding: slow, repetitive and error-prone, because schema documents varied widely in format and quality. The goal was to cut manual effort without compromising accuracy. On a platform feeding fraud and risk decisions, a wrong mapping isn't a cosmetic bug: it means bad data reaching the teams who rely on it.

• Most analyst time went to a few repeatable patterns: matching common fields such as customer IDs, timestamps and channel codes, and translating between naming conventions.

• The platform's data dictionary was huge, and checking every new field against it was slow on its own.

• The genuinely hard cases (ambiguous fields, undocumented legacy systems) were a minority, but they were where mistakes happened.

• Analysts were open to automation only if they could see why a mapping was suggested and stay accountable for approving it.

• Keep a human in the loop. The agent proposes mappings with a confidence score, and analysts review and approve them. High-confidence mappings get a quick check; low-confidence ones get a full review. Even at 99% accuracy, the remaining errors had to be caught before data reached Fraud and Risk.

• Give the model the right context. For each new system, the agent prompts the LLM with the system's schema document together with the platform's mapping rules and examples of approved mappings, and the model proposes mappings directly. We chose prompting over fine-tuning because it was faster to ship and easy to update as conventions changed.

• Validate before scaling. We measured accuracy against mappings analysts had already completed by hand, and only expanded to new systems once accuracy held across different system types.

• Build guardrails in with Compliance and Risk. Rate limiting, audit logging and fail-safes were part of the design from the start. Every suggestion and approval is logged, and if the agent fails or confidence drops, onboarding falls back to the manual process.

Time saved was measured as analyst hours per system onboarded, before and after the agent. Accuracy was the share of agent mappings that matched analyst-verified mappings across diverse systems.

• 60% less manual analysis time.

• 99%+ mapping accuracy across diverse systems.

• Patterns extracted from 250+ system schema documents.

• Analysts now spend their time on the hard edge cases instead of routine field matching.

• Design the review screen with analysts from day one. The first version showed suggestions without the reasoning, and trust only grew once we showed the reasoning behind each suggestion.

• Agree on the accuracy definition with Risk at the start. We spent time debating whether accuracy meant an exact match or an acceptable one.