All writingAntvia · 6 min read

Open table formats matter on the day you want to leave

The storage format is invisible until a renewal, a migration or an acquisition makes it the whole conversation. Test your exit while you are still happy.

Apache IcebergDatabricksDuckDBSnowflakePostgresData engineeringGovernanceFintechSaaSPublic sector

A renewal comes round and somebody in procurement, doing their job properly, asks the data team a question that sounds simple. If we decided not to sign this, how long would it take to move? The room goes quiet, and not because the answer is bad. Nobody has ever measured it. The tables work, the dashboards load, and whether the bytes on disk can be read by anything other than the tool that wrote them has never come up, because until that meeting it never had to.

That is the honest position most teams are in. Storage format is an implementation detail while everything is fine. It becomes the entire conversation on three specific days: the renewal, the day the group decides everything moves to a different cloud, and the day you are acquired by somebody who already runs a different stack and has no intention of running two. On all three days the question is the same. Can another engine read these tables, as they stand, without a rewrite.

Three cards naming the days the storage format becomes the whole conversation: the renewal, where procurement asks how long a move would take; the cloud move, where the group decides everything goes elsewhere; and the acquisition, where the buyer already runs a different stack. Arrows from all three lead into one question: can another engine read these tables, as they stand, without a rewrite. — Three different meetings, one question underneath all of them, and no reason to wait for any of the three to ask it.

So the useful thing to know is not whether your platform is described as open. Almost all of them are, on the slide. The useful thing to know is what happens when you point a second engine at the same files this afternoon, while you are still a happy customer with nothing at stake. That test takes a day and it tells you more than any architecture review.

Three layers, and only two of them travel

It helps to separate what is actually stacked up under the word table. At the bottom sit the data files, almost always Parquet, a columnar format that any serious engine has read for years. Above that sits the table format, Iceberg or Delta or Hudi, which is a metadata layer describing which files make up the table right now, what the schema is, and what it looked like at previous points in time. Above that sits the catalog, which answers one question: for the name analytics.orders, where is the current metadata pointer.

The first two layers are genuinely portable and the specifications are public. Iceberg keeps a metadata file, manifest lists and manifests describing snapshots. Delta keeps an ordered log of JSON commits with periodic checkpoints. Neither is a secret and both can be parsed by anybody willing to read the spec. If your data sits in your own object storage in one of these, a large part of the exit problem is already solved and you should know that, because it is worth real money in a negotiation.

Three stacked layers labelled data files, table format and catalog, with the bottom two marked portable and the top marked as the place lock-in now sits, plus a side column listing SQL dialect, UDFs, access policies and semantic definitions as living outside all three — Open formats solve the bottom two layers. The exit cost moved upward and sideways.

Lock-in moved up into the catalog

The layer people forget is the catalog, and it is now where the commercial gravity lives. A vendor can write immaculate Iceberg into your bucket and still be the only thing that knows which snapshot is current, which tables exist, and who is allowed to see them. If the catalog is proprietary, a second engine either cannot find your tables at all or has to be pointed at metadata paths by hand, which works for a demo and not for two thousand tables.

This is why the Iceberg REST catalog specification matters more than it sounds. It is the interface that lets a different engine ask a catalog the current-pointer question and get a standard answer. Ask your vendor, in writing, whether their catalog implements it, whether an external engine can commit through it or only read, and whether that access is included in your contract or is a separately licensed feature. The three answers are often different and only the first one appears in marketing.

A vendor can write perfectly open files into your own bucket and still be the only thing on earth that knows which snapshot is current.
The pattern, stated plainly

The features that quietly cost you

Table formats are versioned specifications and engines implement them unevenly. The gap between what a format allows and what a given engine supports today is where exits actually go wrong, and it is very rarely the fault of the format.

Merge-on-read deletes Rows deleted logically and reconciled at read time. An engine that ignores the delete files will happily return rows you deleted, which is worse than failing.
Deletion vectors A newer and more compact way to express the same thing. Support arrived at different times in different engines, so it is a common source of a table that reads in one place and not another.
Column mapping and renames Physical file columns tracked by identifier rather than name. Very useful, and a place where an older reader silently sees the wrong shape.
Hidden partitioning and clustering The transforms that make queries fast are metadata. Another engine may read the table correctly and scan far more of it, so correctness is fine and cost is not.
Write access, not read access Many engines can read a table format. Far fewer can commit to it, because committing means an atomic swap of the metadata pointer and that has to go through the catalog.

A second engine with two paths into the same tables. The read path goes straight to the Parquet files and table format metadata sitting in your own bucket, and works because the specifications are public. The write path has to pass through the catalog, the only thing that knows which snapshot is current, because a commit is an atomic swap of the metadata pointer. — The two paths are not the same problem. One is answered by a public specification, the other by your vendor's contract.

The part no format solves

Even with everything above answered well, the files are the smaller half of the migration. The bigger half is everything that was never in the table to begin with. The SQL dialect your transformations are written in, with its date functions and its window syntax and its particular idea of what a lateral join is. The user-defined functions. The stored procedures somebody wrote in 2022 that reconcile the ledger. The row and column access policies, the masking rules, the semantic definitions that decide what active customer means, and the orchestration that decides what runs when.

None of that is described by Iceberg or Delta and none of it moves when the files do. This is the uncomfortable part, and it is why we build Antvia so governance is enforced on the query rather than baked into copies of data. A rule that lives next to the query is a rule you can carry. A rule that lives inside a vendor's policy engine is a rule you will rewrite from a screenshot.

The test, run while you are still happy

01
Pick a table that hurtsNot the tidy dimension table. Take something wide, partitioned, with schema history and a few deletes, ideally one that supports a report a director looks at.
02
Point a second engine at it, read-onlyDuckDB is the cheap choice because it runs on a laptop and reads Parquet and Iceberg without a cluster. The point is not to adopt it. The point is that it is not the vendor.
03
Reconcile, do not eyeballRun the same aggregate in both engines and compare row counts, sums and a few null rates. Deleted rows reappearing is the failure mode to hunt for, because it looks like success.
04
Try the catalog path, then the file pathFirst resolve the table by name through the catalog. If that fails, resolve it by metadata location. The difference between those two outcomes is the size of your exit problem.
05
Write the result downOne page, dated, listing what worked, what needed hand-holding and what could not be read at all. Repeat it every quarter or after any major platform upgrade.

A day of work, run when nothing is at stake, so the answer exists before anyone needs it.

What comes out of that page is usually not a crisis. It is a short list of specific, fixable things. A catalog that only speaks its own protocol. Three tables using a feature the second engine has not implemented. A transformation layer written in a dialect that only one engine parses. Each of those has a decision attached, and you can take those decisions calmly in a quiet quarter or badly in the last fortnight of a contract.

When none of this is worth your attention

There is a version of this argument that is wrong, and it is worth saying so. If you are a team of four with a few hundred gigabytes and one warehouse that everybody likes, running your own catalog and maintaining table maintenance jobs is work you will feel every week in exchange for an option you may never exercise. A managed proprietary warehouse is a perfectly good answer at that size. Portability is insurance, and insurance is priced against the thing you are insuring.

The picture changes when the data is regulated, when the parent company has a stated cloud, when an acquisition is plausible, or when the annual bill has grown to the point where the vendor knows you cannot leave and prices accordingly. At that point the cheapest thing you can buy is a reliable answer to how long would it take to move, and the only way to have one is to have tried it.

Book the day. Pick the ugly table. Find out now, when the answer is interesting rather than urgent.