Data integration, quality and governance

Two numbers for one measure. Three records for one customer. No name on either dataset.

This is the work of making several systems agree on one set of records, and keeping them agreeing. Schema contracts at the boundaries, checks that fail a run rather than warn in a log nobody reads, a named owner per dataset, reconciliation between systems that are each certain they are right, and lineage that answers where a figure came from. Pipelines and storage layout are the neighbouring discipline.

Companies delivered for
20+
Median to first production-grade artefact
6 weeks
Week 1 audit
Fixed fee, two calls, one-pager kept either way
Team on the work
Pod of three
Published case study for this pillar
None yet

Where agreement is won, and where it is lost

Two systems do not start disagreeing in the report. They start disagreeing at a boundary, months earlier, where one of them handed over a field whose meaning nobody had written down.

  1. 01

    The definition

    What counts as a customer, when an order becomes revenue, which day a month ends on. An organisation can hold several answers at once, with no record of who chose any of them.

  2. 02

    The contract at the boundary

    The shape, the types, the nullability and the meaning of what one system hands to another, written down and checked on arrival. Without it, an upstream change is discovered downstream, by a person, in a meeting.

  3. 03

    The check that can stop a run

    Quality rules that fail the load rather than warn in a log nobody reads. A warning is a decision to carry on with data you have just been told not to trust.

  4. 04

    The named owner

    One person per dataset who decides what its fields mean and approves changes to them. A team name is not an owner. A distribution list is not an owner.

  5. 05

    The reconciliation

    Systems that are each certain they are right, compared on a schedule, with the differences listed individually rather than netted off into a figure that looks close enough.

  6. 06

    The change process

    What happens when a definition genuinely has to move: who is told, what gets versioned, and what last year's reported number becomes now that it was produced a different way.

The arguments that bring people to this page

None of these arrive described as a governance problem. They arrive as an argument nobody can win, usually in front of somebody senior.

  • Two teams quote different figures for the same measure

    The meeting settles it by seniority rather than by evidence, because neither figure can be traced back to what produced it. The next meeting settles it again, differently.

  • A nightly job has been quietly dropping rows

    Nothing failed. The load reported success every night. It surfaced when somebody happened to compare a total against a source system by hand, for an unrelated reason.

  • A field changed meaning upstream and nobody was told

    A status code was reused, a currency column started arriving in a different unit, an optional field became mandatory. The first people to find out were in a board meeting.

  • Nobody can name the owner of a dataset everyone depends on

    It has been in production for years. Three teams read it, two write to it, and the person who built it has left. Changing anything about it is now nobody's decision to make.

  • The same customer exists three times, with three spellings

    So does the same supplier, and the same SKU under two part numbers. Every count is slightly wrong and every join loses rows that nobody can account for afterwards.

  • Nobody can answer where a number came from

    Not because the answer is complicated, but because getting it means opening a chain of jobs and spreadsheets, then asking a person who is on leave.

If three or more of these are true, what is missing is agreement between your systems, not another report built on top of them.

Each argument traces back to a control nobody put in

The complaint is always about a number. The fix is always a control at a boundary, one layer away from where the argument is happening.

  1. Two teams, two figures for one measureAgreed definitions, held where queries reach them
  2. A job quietly dropping rowsChecks that fail the run
  3. A field that changed meaning upstreamSchema contracts at the boundary
  4. No named owner for a shared datasetOwnership per dataset
  5. The same customer three timesMaster and reference data
  6. Nobody can say where the number came fromLineage and evidence
Left, the argument as it is described to us. Right, the control that was never put in. These controls tend to be missing together rather than one at a time, which is why putting one of them in on its own rarely settles the argument.

Six controls, and the failures they answer

These are not six products. They are the six ways systems stop agreeing, and they turn up tangled together in the same engagement.

ContractsAt the boundary

Written where one system hands over to another

Shape, types, nullability and meaning for every field crossing a boundary, checked on arrival rather than assumed. When an upstream team changes something, the contract breaks in their pipeline, which is the correct place for it to break.

QualityFail, not warn

Checks that stop a run instead of writing a line in a log

Counts against expectation, referential integrity, ranges, freshness, and the business rules that actually matter. A failed check halts the load and reaches a named person. Warnings accumulate somewhere nobody has opened since the week they were built.

OwnershipOne name

A person per dataset, not a team mailbox

Every dataset more than one team depends on gets a named owner who decides what its fields mean, approves changes and answers questions about it. We write the register and get the names agreed while everybody is still in the room.

Master dataOne record

The same customer, supplier or SKU, once

Matching and merging rules for the entities that exist several times over, a survivorship rule deciding which value wins when two records disagree, and a review queue for the pairs the rules are not confident about. Nothing is merged on a hunch.

ReconciliationRecord by record

Between systems that are each certain they are right

Scheduled comparisons that list differences individually rather than summarising them into a percentage that reads as close enough. The point is to find the differences while there are still a handful of them, not after most have been invoiced.

LineageWhere it came from

An answer to the only question Finance ever asks

A trace from the figure on the screen back through every job, join and definition that produced it, plus the change process for when a definition moves. This is where this work meets our pipeline and storage practice.

See the data engineering page

How a disputed measure actually gets settled

Both figures can be correct at once. They answer slightly different questions, and nobody has written the two questions down.

  1. 01

    Have both numbers produced in front of us

    Not described, produced. Whoever owns each figure runs it while we watch, from wherever they really run it, including the spreadsheet step they were not going to mention.

  2. 02

    Trace each one back to raw records

    Job by job, join by join, filter by filter. A dispute can end here, at a date boundary or at a filter somebody added for a sound reason and never removed.

  3. 03

    Write down where the two definitions diverge

    In plain sentences. Cancellations included or not. Returns netted or not. The month closed on the calendar or on the ledger. The currency converted at which rate, on which day.

  4. 04

    Decide which one is the reported figure

    By the people whose decisions depend on it, not by us and not by whoever is most senior in the room. The other definition can survive under a different name, because it was answering a real question.

  5. 05

    Put the decision somewhere a query can reach

    The agreed definition goes into the layer that produces the number. If it lives only in a document, the next round of the same argument is already scheduled, it just has not been booked yet.

  6. 06

    Make the superseded version hard to produce by accident

    The old logic is removed or renamed, the extract that used it is retired, and a check goes in that would have caught the divergence. Otherwise it returns the first time somebody is in a hurry.

Three grades of check, and only one of them stops anything

Your platform probably has quality checks already. If they are grade one, the problem walked straight past them.

  1. Grade 1

    A warning in a log

    It runs, it writes a line, the pipeline carries on. Useful for a short while after somebody builds it, then invisible. A check nobody is subscribed to is not a control, it is material for the post mortem.

  2. Grade 2

    An alert to a named person

    Somebody is told and can act. The right grade wherever stopping the load would cost more than the bad data does. It only works if the person is named in advance and knows they are the one on the hook.

  3. Grade 3

    A gate that fails the run

    The load stops, and downstream consumers keep yesterday's data rather than being handed today's wrong data. Reserved for the checks where wrong is worse than late, which is how anything that gets reported outside the building tends to be weighed.

Deciding the grade for each check is the argument worth having, and it belongs to the business rather than to us. Grade three on everything stops the platform so often that somebody switches the gates off, which leaves you with nothing.

One customer, several records, and the rule that decides

Deduplication is the part everyone underestimates, because the hard bit is not the matching. It is agreeing in advance what happens to the fields that disagree.

  1. ACME Industries Pvt LtdOne customer record, tax ID as the key
  2. Acme Industries P. Ltd.One customer record, tax ID as the key
  3. ACME INDUSTRIESOne customer record, tax ID as the key
  4. Acme Ind. (Mumbai)Held for review, a person decides
  5. Acme Holdings LtdDifferent legal entity, deliberately not merged
An illustration of the shape, not a real client's customer list. The names change every time, the three outcomes do not: some records merge on a hard key, some go to a person because the rules are not confident enough, and one turns out to be a different company that happens to share a name. The greyed line is the merge we refuse to make.

What we have actually delivered

There is no published case study for this pillar yet, so these are firm-wide delivery figures. On the call we will say which of them came from work shaped like yours.

20+Companies delivered forFrom Pune, across India, APAC and the US.
50+Projects deliveredSince 2023. This work tends to run as a strand through one of them rather than as a project with its own name.
6 weeksMedian to first production artefactSomething running that your team can break and we fix, not a document about future intentions.
ThreePeople in a pod, no benchThe people on the first call are the people who do the work.

How this runs, week by week

The same three phases as every Woodfrog engagement, with the definitions argument started early because it has the longest lead time in the whole programme.

  1. Week 1

    The audit, fixed fee

    Two calls. We take one disputed measure and one shared dataset and follow both back to their sources. You get a one-pager naming what is broken, what it is costing you and the order to fix it in, and you keep it whether or not we go further.

  2. Weeks 2 to 6

    Contracts, gates and owners go in

    A pod of three. Contracts on the boundaries that break most often, gates on the checks where wrong is worse than late, and an ownership register with real names against it. Across our engagements the median time to a first production-grade artefact is six weeks.

  3. Weeks 2 to 6

    The disputed measures get settled in parallel

    Not after the engineering, alongside it. Getting a room to agree what a month end is takes longer than writing the check that enforces the answer, so it starts first.

  4. Week 6

    Reconciliation runs without anyone starting it

    The comparison between systems is on a schedule, and the differences arrive as a list with owners against them rather than as a percentage on a dashboard that nobody can act on.

  5. Week 7 onward

    Operate

    Quarterly reviews and on-call governance. Definitions drift because businesses change, and the review exists to catch that drift while it is still a conversation rather than a discrepancy in front of a board.

Why this is faster than the last time you tried it

Two things shorten the work: harnesses that have already run in production, and Claude doing the reading that used to consume the opening weeks.

  1. Already built

    Harnesses we bring with us

    Contract checkers, quality gate runners, duplicate-matching rules and reconciliation comparators, proven in production elsewhere and fitted to your systems rather than written again from an empty repository.

  2. AI first pass

    Reading what nobody documented

    Claude reads schemas, job code, stored procedures and the report definitions that quietly contradict each other, then drafts candidate definitions and flags the contradictions it cannot resolve. Those flags are where the useful conversations start.

  3. AI first pass

    Proposing the checks from the data itself

    Candidate quality rules are drafted from what the data actually does rather than from what a template suggests, then argued down to the few worth failing a run for. Proposing is cheap. Deciding the grade is the work.

  4. Human call

    Every decision that carries weight stays with a person

    Which definition is the reported one, which duplicates merge and which are separate companies, what fails a run and what only warns. AI does not make those calls here, and we will not pretend otherwise on a call.

  5. What you keep

    Yours to run without us

    Contracts, checks, the ownership register, the reconciliation jobs and the harnesses, documented and versioned in your repositories. If your team cannot change it without us in the room, we have failed at the part that matters.

We have not published a case study for this pillar

There is no named client story on this page. That is worth explaining rather than papering over with somebody else's number.

What we can say01

Firm-level figures, not a governance result

Woodfrog has delivered for 20+ companies across 50+ projects since 2023, from Pune, working across India, APAC and the US. The median time to a first production-grade artefact is six weeks.

What we will not do02

Borrow a result from another pillar

The recovery and margin figures published elsewhere on this site belong to one APAC retail grocery engagement and stay on the pages for that work. Reprinting them here would tell you nothing about whether we can get your two teams to agree on one figure.

What is claimed here03

Method, not a measured outcome

Everything above describes how the work is done: what gets contracted, what fails a run, who owns what. None of it is presented as a result we measured on a governance engagement, because we have not published one.

When there is one04

The bar a case study has to clear

A named case will appear here once a customer agrees to be named and the numbers have been through their own Finance team. That is the bar every other figure on this site cleared, and we are not lowering it for a page that needs one.

Questions data and finance teams ask us

The ones that come up on almost every first call.

Is this not just what a data catalogue does?

A catalogue is a place to write things down, and it helps once the decisions exist. It does not decide which of two figures is the reported one, it does not stop a load, and it does not give a dataset an owner. A catalogue can be accurate about every table and still silent about the argument.

How is this different from your data engineering work?

Data engineering is about jobs finishing inside their window, migrations reconciling and storage laid out for how the data is read. This page is about several systems agreeing on one set of records and staying agreed. The two meet at lineage, which is why both pages mention it. If your problem is a batch that no longer finishes before the working day starts, begin there instead.

We already run quality tests. Why do our numbers still disagree?

Because the tests warn rather than fail, and because they test the data rather than the definitions. A column can be complete, in range and fresh, and still be the wrong measure. The disagreement lives one level above where the tests are looking.

Nobody here will agree to own a dataset.

Ownership has probably meant being blamed for a dataset with no power to stop the upstream change that broke it. It works when it comes with the right to reject a change and a contract that makes upstream changes visible before they land. We put those in first, then ask for names.

Will the new checks stop our pipelines every night?

Some of them will early on, and that is the point. A gate that fails soon after it goes in is the platform telling you something true that it had been keeping to itself. Thresholds get tuned with your team watching, and any check that fails for a reason nobody can act on gets demoted or removed.

Do we have to freeze reporting while definitions get settled?

No. The existing numbers keep being produced. A settled definition is added alongside the old one, the two are compared for a period, and it becomes the reported figure once the difference is understood and explainable. Freezing reporting in order to fix reporting is how these programmes lose their sponsor.

Our data is a mess. Should we clean it up first?

Nobody has ever finished that clean-up before starting. The Week 1 audit exists for exactly this state: two calls, a fixed fee, and a one-pager naming what is broken, what it is costing you and the order to fix it in. You keep it either way, including when the honest answer is that this is not your first problem.

Is AI writing our rules, then?

No. Claude does first passes: reading schemas and job code, drafting candidate definitions, proposing checks. Every decision that carries weight, which definition is reported, which duplicates merge, what fails a run, is made by a person whose name is on it. We are an Anthropic Build Partner and joined the Claude Partner Network at launch, which is a statement about how we build, not about who decides.

What happens after week six?

Quarterly reviews and on-call governance from week seven, and you can stop at any point. The reviews exist because definitions drift as the business changes. A set of contracts put in place and never revisited becomes a formality, and then a fiction.

You have no case study for this. Why should we believe you?

Because we told you, rather than reprinting a figure from a different pillar and hoping you did not check where it came from. What we offer instead is week one: two calls, one disputed measure traced back to its sources, and a one-pager you keep whether or not we do anything else.

Start with the Week 1 audit

Two calls, a fixed fee, and one disputed measure followed back to the records that produced it. You get a one-pager naming what is broken, what it is costing you and the order to fix it in, and you keep it either way. NDA-friendly, fixed scope. Write to hello@woodfrog.tech.