Moving/Restoring/Recovering Streaming Tables From a Deleted Pipeline

Hi folks,

There was an accident which resulted in the deletion of our pipelines, and in turn, their tables (in dev, thankfully). I wanted to double check the steps I’ve tried and the final plan to remediate. The pipelines were all on legacy mode and are responsible for raw to bronze ingestion. We do have all of the files, but rebuilding from scratch is considered too time consuming.

I have UNDROP’d all of the STs, and I wanted to reconnect to a newly deployed version of each pipeline. Unfortunately, I found that I was unable to “move” them between pipelines since Databricks can’t verify ownership of the source pipeline, because it’s gone. I know there’s more secret sauce under the hood (e.g., backing tables), so I’m not going to attempt to modify the delta tables outside of Databricks.

My current best idea, and what I plan to do this evening, is to rename the schema with restored tables and create a new “recovery” pipeline that streams all of the data from the restored table into a new version of the table (with the correct name, since it’s available again after renaming the schema). Then, I can redeploy a pipeline and move the ST from the recovery pipeline to the actual pipeline. This would mean I lose the checkpoint information for the autoloader, which isn’t a massive issue for these pipelines, and it seems like the cleanest way to insert the restored data into the pipeline. I would also need to move any other assets, e.g., views/tables, to the new schema (with the original name).

I don’t know if moving the STs between legacy pipelines would rename them automatically, in which case I could rebuild elsewhere then drop the original and avoid moving other assets.

Any thoughts would be welcome and greatly appreciated.