Data Products: The mythical Magic Pill

Data products have become one of those ideas that almost everyone agrees with.If you say you’re building data products, people nod.If you say data products are the future, few disagree.If something around data isn’t working, the answer is often simple: we need data products.

That’s where my discomfort starts.Not because data products are a bad idea.But because they are increasingly treated as if they can fix almost everything.They can’t. And they were never meant to.

The idea

The original idea behind data products is sensible. Some data is reused heavily. Some datasets need to be stable, curated, well understood, and trustworthy across teams. Packaging that data deliberately, with clear semantics and expectations, makes sense.

That is where data products shine.The problem starts when this idea quietly expands. When data products stop being a pattern and start being an answer.When governance struggles, we talk about data products.When ownership is unclear, we talk about data product owners.When architecture is fragmented, we talk about data products.

The clear limitation

At that point, the concept is no longer being used carefully. It’s being used as a substitute for decisions we haven’t made. One uncomfortable truth is that most data inside an organisation is not product-like.A lot of data exists because systems exist. Because processes run. Because regulations require it. Because events happen.

That data is often operational, transient, local, or tightly coupled to a specific workflow. It changes frequently. It has limited reuse. Its value is contextual and time-bound.

Trying to force all of that into a “product” shape isn’t maturity.It’s overengineering.A data product implies intent, stability, consumers, and a promise of fitness for use. Most data does not meet those criteria and never needs to.

The magic pill thinking

This is where the magic-pill thinking becomes dangerous.When data products are positioned as the way to organise all data, selectivity disappears. Everything starts looking like a candidate for productisation. The result is predictable: too many products, unclear boundaries, confused ownership, and rising coordination cost.What was meant to simplify consumption quietly becomes another layer to manage.

If everything is a data product, then nothing really is.

Another assumption I struggle with is the idea that data products should become the default or preferred ingestion path across the organisation.

Organisations don’t operate at a single level of risk, reuse, or stability. Some data benefits from strong curation and controlled change. A lot of data does not. Declaring a preferred path flattens those differences and quietly imposes a maturity model where it doesn’t fit.

Data products should not be the default.They should be conditional.Used deliberately, where the data itself justifies the overhead.

The real time conundrum

There is also a boundary that often gets ignored entirely: real-time.Data products are fundamentally not designed for real-time needs. They optimise for understanding, not immediacy. For consistency, not latency. For stability, not reaction speed.

Real-time use cases live close to source systems, event streams, and operational APIs. They care about freshness and availability far more than curation. Introducing a data product layer there doesn’t add control, it adds delay, coupling, and failure modes.

When organisations try anyway, one of two things happens. Either the data product becomes a thin wrapper over a real-time feed, or it gets bypassed entirely. In both cases, the model quietly breaks.That’s not a failure of data products. It’s a reminder that they were never meant to serve every type of data flow.

The Data Mesh Angle

What makes this even more interesting is the relationship with data mesh.

Most organisations are comfortable saying that data mesh is not for everyone. It assumes strong domain ownership, decentralised teams, platform maturity, and a willingness to accept local autonomy and duplication. Without those conditions, data mesh becomes fragile very quickly.That honesty is healthy.

What’s inconsistent is rejecting data mesh as context-dependent, while treating data products as universal.

Data products don’t exist in isolation. They make sense within the same organisational assumptions that data mesh relies on. If you don’t have or don’t want that structure, data products lose the conditions that make them work.At that point, they stop being products and start being curated datasets with a better name.

The Conclusion

None of this means data products are a bad idea.It means they are being asked to do too much.They are very good at improving consumption where reuse and stability matter.They are very bad at fixing structure, authority, architecture, or delivery models.

Those are not product problems. They are organisational ones.Strong ideas don’t need to be defended as cures. They work quietly, within clear boundaries. Data products should be treated the same way: as one tool among many, applied selectively, not ideologically.

The real question isn’t “how do we build more data products?”It’s “which data actually deserves to be treated as one?” Until that question is answered honestly, data products will keep being treated as a magic pill and will keep disappointing people for reasons that have nothing to do with execution.