Unity Catalog Migration: Leave Hive Metastore, Keep Jobs Up

Paweł Szczepanik
Paweł Szczepanik
November 17, 2026
•
9 min read
Loading the Elevenlabs Text to Speech AudioNative Player...

A Unity Catalog migration moves every table, view, grant and job that still points at the legacy Hive metastore into Unity Catalog, and then turns legacy access off. The Hive metastore documentation is plain about the status: "Although Databricks workspaces continue to include the built-in Hive metastore, data governance using Hive metastore is deprecated." The page on UC-only workspaces adds the date: "Starting October 8, 2026, all new workspaces are provisioned with Unity Catalog and without access to certain legacy features. Existing workspaces and their workflows are not impacted." The estate as a whole needs no maintenance window; each table you convert in place gets a short, planned write pause. Downtime comes from doing the steps in the wrong order, not from the size of the data, and that order is what a Unity Catalog migration plan has to get right.

Is the Hive Metastore Deprecated? What Changed in 2026

The legacy features page lists what workspaces created since October 8, 2026 no longer get: DBFS root and mounts, the Hive metastore, no-isolation shared clusters, and Databricks Runtime versions prior to 13.3 LTS. Accounts created after December 18, 2025 never had these features at all. For older workspaces with nothing running against the Hive metastore, the guide to disabling Hive metastore access warns that "legacy access might be disabled automatically on your behalf starting in July 2026."

What the documentation does not give is a shutdown date for existing workspaces that still use it. As of November 2026 it says "deprecated," and that "you should fully transition your workspaces as soon as you can." For an existing workspace, a Unity Catalog migration is a question of when, not whether, and the answer gets harder each time a new job lands on hive_metastore.

If you need the case for Unity Catalog itself, start with what Unity Catalog and volumes change for data engineers; this article assumes the decision is made and covers the move. Moving a company onto Databricks from another platform is a different project, covered in our guide to Databricks implementation work; here the platform stays the same and only the metastore under your tables changes.

Start With a UCX Assessment, Not a Copy Job

Every Unity Catalog migration we run starts with an inventory, because the estate is rarely what the team remembers. UCX, the Databricks Labs toolkit, installs with databricks labs install ucx and runs an assessment of group identities, storage locations, storage credentials, access controls and tables. The UCX documentation notes that "You can run the assessment workflow multiple times," and we rerun it after every wave. It needs Databricks CLI v0.213 or above, Python 3.10 or above, and account and workspace admin rights.

Know what you are installing. The table upgrade guide says "Databricks recommends UCX for most workspace upgrade scenarios," while the UCX page says the toolkit "is not formally supported by Databricks with service-level agreements (SLAs)." Both hold: issues go to GitHub, not to Databricks support. Its code migration workflow "is not yet available," so UCX moves groups and tables, while notebooks, jobs and BI queries that use two-level names are yours to rewrite.

Cut scope here too. A customer story on the Databricks blog reports that 7-Eleven de-scoped about 40 percent of inactive or low-value legacy jobs. Our rule: a job whose output nobody reads gets switched off first, and migrated only if someone asks for it back.

Pick an Upgrade Path per Table Type

A Unity Catalog migration does not have one upgrade path; it has one per table type. The workspace upgrade guide names the default: "The recommended approach is to first federate your Hive metastore or AWS Glue catalog as a foreign catalog, then upgrade the foreign tables in place." That, it says, lets you "migrate tables without data movement while preserving table history, configuration, permissions, and views." The in-place step is ALTER TABLE ... SET MANAGED MOVE, or SET EXTERNAL for tables that should stay external. Hive metastore federation here means federating your own metastore, not an outside database.

The direct routes still exist. SYNC registers external tables, and managed tables outside DBFS root, as external tables, but "You cannot use it to upgrade Hive managed tables stored in workspace storage." Those take DEEP CLONE, and CREATE TABLE AS SELECT covers the rest. Cloning has a price: "Table history is not migrated when you run CREATE TABLE CLONE," so time travel to earlier versions stops working. And since "Unity Catalog manages partitions differently than Hive," code that manipulates partitions by hand needs changes.

Upgrade paths by Hive table type:

Hive tablePathResult in Unity CatalogTable historyWriter pause
Managed Delta, in DBFS rootFederate, then SET MANAGED MOVE; or DEEP CLONEManagedKept with SET MANAGED; lost with DEEP CLONEShort, at switch (SET MANAGED); none on source while cloning, but writes must be redirected
Managed, outside DBFS rootSYNC ... AS EXTERNAL, then optional SET MANAGEDExternal, then managedNot stated for SYNC; kept by SET MANAGEDShort, at switch
External (Delta)Federate or SYNC, then optional SET MANAGEDExternal or managedKept with SET MANAGEDShort, at switch
Non-Delta or unsupported formatCREATE TABLE AS SELECTManagedLost (new table)Plan a write freeze per table
ViewsRecreate after their source tables are upgradedViewn/aNone

Pick the path per table, not per workspace: most estates need at least two of these.

Rebuild Permissions on Account Groups

Grants are where people lose access. The workspace upgrade guide says to "Convert any workspace-local groups to account-level groups," because workspace-local groups, as the page on managing them puts it, "cannot be assigned to additional workspaces or granted access to data in a Unity Catalog metastore." UCX has a group migration workflow that carries their permissions over to account groups.

The grant model also behaves differently. The page on running the Hive metastore alongside Unity Catalog lists the differences, and these are the ones that bite:

  • a principal needs USE CATALOG on the parent catalog and USE SCHEMA on the schema, whatever it holds on the table;
  • there is no DENY: "Privileges that are not granted are implicitly denied";
  • the owner of a view does not need to own the tables behind it.

Every DENY in Hive has to become an absent grant, which usually means splitting groups. In our projects this step most often leaves an analyst locked out on Monday morning, so we test grants per group before any table switches.

How to Cut Over Without a Maintenance Window

Jobs keep running when both catalogs answer at once. The Hive metastore federation documentation describes federation as "a step in the migration path to Unity Catalog, enabling incremental migration without code adaptation," and a federated internal metastore accepts both reads and writes. Queries can run marginally slower while both metastores sit on the path. The alternative is SYNC, which "is designed to be run on a schedule."

The switch happens table by table. According to the SET MANAGED documentation, conversion runs in two phases: a data copy while the table stays in use, then a switch during which "all writes to the external location are temporarily blocked, resulting in writer downtime." Readers on Databricks Runtime 16.4 LTS or above see no downtime; readers on 15.4 LTS and below might. Running SET MANAGED requires Databricks Runtime 17.3 LTS or above, or serverless compute.

The documentation estimates that a table of 100 GB or less copies in about 6 minutes or less, with about 1 to 2 minutes of downtime or less. At 1 TB it puts the copy at about 30 minutes and downtime at about 1 to 2 minutes; at 10 TB, about 1.5 hours and about 1 to 5 minutes. File and commit counts move those numbers.

Streams need their own plan: "After table conversion, existing read and write streams fail," so each one needs a scheduled restart with the same configuration. To find code that still reads old names, add deprecation comments to the old tables; notebooks flag them and offer a Genie Code quick fix. Then set spark.databricks.unityCatalogOnlyMode on individual clusters and watch which jobs still reach for Hive.

Disable Legacy Access, and Know What You Cannot Undo

The last step of a Unity Catalog migration is the only one with a deadline you set yourself and cannot take back. It is not optional, because clusters keep connecting to the Hive metastore by default until you disable access. Once you do, jobs against Hive metastore tables fail, jobs on runtimes below 13.3 fail, and running clusters need a restart. The setting takes about five minutes to apply.

The irreversible part is the clock: "After 30 days with the legacy metastore disabled, the workspace permanently becomes a Unity Catalog-only workspace." Treat those 30 days as a test with real users. Table rollback has its own window. After SET MANAGED, "Databricks retains the data in your Unity Catalog external location for 14 days to allow rollback." With DEEP CLONE, the old table stays until you drop it.

One trap is easy to miss. On a federated internal metastore, "DROP commands are reflected in the Hive metastore," where dropping a managed table deletes its data files immediately. Restrict who can drop objects in the foreign catalog until the cutover is done.

How We Sequence a Unity Catalog Migration

The order we use for a Unity Catalog migration follows the dependencies above:

  • run the UCX assessment and de-scope dead jobs;
  • move to account groups and rebuild the grant model;
  • federate the Hive metastore so every table is visible through Unity Catalog;
  • upgrade tables in waves, least dependent first, with a read and write test after each wave;
  • repoint jobs and BI to three-level names, and move DBFS mount paths to Unity Catalog storage;
  • turn on UC-only mode per cluster, then disable legacy access for the workspace.

Most of the work sits in step five, not in copying data, because UCX does not automate code yet. That is where our effort goes in Databricks platform and migration work. For a second pair of eyes on your plan, bring us your UCX assessment report, or the list of jobs that still read hive_metastore.

FAQ

Is Databricks shutting down the Hive metastore?

Not for existing workspaces, as of November 2026. Governance on it is deprecated and new workspaces since October 8, 2026 are created without it, but the documentation gives no shutdown date for workspaces that already use it.

Can you migrate to Unity Catalog without downtime?

For the estate as a whole, yes: a Unity Catalog migration can run while Hive metastore federation or scheduled SYNC keeps both catalogs answering. Each table you convert in place still has a short write pause; the documentation estimates about 1 to 2 minutes for tables up to 1 TB and about 1 to 5 minutes at 10 TB. Streams on a converted table have to be restarted.

Does DEEP CLONE keep table history?

No. Tables cloned from the Hive metastore start a new history in Unity Catalog, so time travel to versions before the move stops working. If history matters, federate the Hive metastore and convert the table in place with SET MANAGED, which retains table history.

Is UCX officially supported by Databricks?

UCX is a Databricks Labs project: Databricks recommends it for most workspace upgrades, but it comes without SLAs and issues go to GitHub. It handles assessment, group migration and table migration; its code migration workflow is not yet available, so plan the code side of a Unity Catalog migration yourself.

Share this post
Data Engineering
Paweł Szczepanik
MORE POSTS BY THIS AUTHOR
Paweł Szczepanik

Curious how we can support your business?

TALK TO US

More insights

More news

No items found.