Data Lineage: Chasing Coverage or Achieving Control?
In a previous article, I argued that lineage should be earned rather than built. That raises a natural follow-up question. If lineage is an outcome of good governance and engineering practices, how much should organisations invest in pursuing lineage directly?
Lineage has become one of the most sought-after capabilities in data management. Mention lineage in a meeting and there is usually little resistance. The benefits appear obvious. Better impact analysis. Better incident investigations. Better regulatory traceability. Better transparency.
On the surface, it sounds like a sensible investment.Yet the more I think about it, the more I find myself asking a different question.Not whether lineage is valuable. But whether we have become so focused on the answer that we have forgotten the original question.Because the objective of an organisation is not lineage.
The objective is control. Those two things are related, but they are not the same.
Most executives approving large lineage investments will never use lineage themselves.They won’t spend their day tracing data elements through twenty systems.They won’t navigate lineage diagrams.They won’t investigate transformation chains.
What they actually care about is much simpler.
Are we in control? Do we know what is happening? Can we understand the impact of change? Can we explain our numbers? Can we respond when something goes wrong? Can we satisfy ourselves that the organisation understands its own operations? These are control questions. Lineage is merely one possible way of answering them.
The challenge begins when lineage becomes the objective rather than the mechanism. The justification is usually familiar. “We need lineage for impact analysis.” “We need lineage for incident investigations.” “We need lineage for regulatory requests.” All reasonable arguments.
But before investing millions, shouldn’t we ask whether lineage is the most effective and proportionate way to achieve those outcomes? Or have we accidentally confused a capability with the objective itself?
If the objective is impact analysis, perhaps architecture has a role to play.If the objective is regulatory traceability, perhaps ownership, documented transformations, and controlled reporting processes matter just as much.If the objective is incident investigation, perhaps observability, logging, change records, and operational discipline deserve equal attention.
The goal should be control. Not lineage.
One aspect of lineage that rarely gets discussed is cost. Not the initial cost. The long-term cost. The first part of the journey is often relatively straightforward. The major systems are known. The critical reports are understood. The primary transformations can be identified. Coverage increases quickly and progress is easy to demonstrate.
Everyone feels good about the investment. Then something changes.The organisation starts chasing the remaining gaps.Legacy systems appear.Manual interventions surface.Spreadsheets emerge from unexpected places.Undocumented integrations suddenly become important.The closer an organisation gets to complete lineage, the harder every additional percentage point becomes.
The cost curve is not linear.
Moving from 20% to 70% lineage coverage may be relatively achievable. Moving from 70% to 90% can become significantly harder. Moving from 90% to 100% may consume a disproportionate amount of effort.The final stretch is often the most expensive. And that is where the executive conversation should begin.
Many organisations celebrate lineage coverage. 60%. 70%. 80%. 90%. But I often wonder whether these numbers create a false sense of confidence. If your lineage is 70% complete, do you know which 30% is missing? Because not all lineage has equal value. The missing 30% might be irrelevant. Or it might contain the one dependency that explains the incident you’re investigating.The one manual adjustment affecting a regulatory figure. The one transformation that changes a business outcome.
Coverage creates comfort.Control requires understanding.
The two should not be confused.
This becomes particularly interesting when regulatory expectations are used to justify the investment. Regulators typically ask for traceability. Explainability. Accountability. Evidence. These are reasonable expectations.
What is often less clear is where the organisation decides to stop. At what point is there sufficient lineage? At what point have we achieved enough traceability?At what point does additional investment stop producing meaningful control?
I rarely hear these questions being asked. Instead, the pursuit of lineage often becomes self-sustaining. Every uncovered dependency becomes another justification for investment.Every gap becomes another reason to expand scope.
The organisation gradually starts optimising for lineage coverage rather than control. And those are very different objectives.
The question becomes even more important when viewed through the lens of opportunity cost. Every euro spent on lineage is a euro that cannot be spent elsewhere. Every team assigned to lineage is a team not working on something else.The real question is not whether lineage has value.The real question is whether it has more value than the alternatives.
Could the same investment improve:
- Data ownership?
- Data quality controls?
- Data contracts?
- Metadata quality?
- Architectural simplification?
- Observability?
- Control automation?
- Change management?
All of these contribute to control.All of these reduce uncertainty.All of these help organisations understand what is happening. The difference is that they often address the underlying conditions rather than improving visibility into the outcome.
This is why I believe executives should ask a different question.
Not:
“How much technical lineage do we have?”
But:
“How do we know we are in control?”
That question forces a broader conversation. It moves the discussion away from tools and towards outcomes. Away from coverage percentages and towards confidence. Away from capabilities and towards decisions.
Because ultimately, lineage is not the goal.The goal is understanding. Understanding how the organisation works. Understanding the impact of change. Understanding where risks exist. Understanding whether decisions can be explained and defended.
Lineage can absolutely contribute to that understanding. In some organisations it may contribute significantly. But before investing millions in the next phase of a lineage programme, perhaps the most important question is this:
If lineage is the only answer, what exactly was the question?
