Hi all! Does anyone have any experience doing a full backup and restore with Firestore, from one db into another? I use Datastore mode, but I expect native is similar. The db I’m planning to do this on is 2.3TB total, 667M entities (944GB), 7.6B built in indexes, 1.6B composite indexes. Any idea how long that should take to backup and restore?
I’m open to hearing about other options too. Export/import is the main one I know of, but it sounds like exports don’t include indexes, only objects themselves, so when you import, the indexes get rebuilt from scratch. Backups evidently include indexes, so restoring a backup is generally quicker?
I’m also familiar with PITR, but for a full db, PITR evidently just uses normal import.
If it helps, my use case here is moving a multi-region (nam5) db to single-region (us-central1).
Well, that’s a voluminous database. I’m not an expert on Firestore but I had a lot of fun with it and other NoSQL databases. With these, writing massive amounts of data always often is laborious but from my experience, backup and restore seems to be the way to go.
To estimate the time, we could calculate this with a pen and paper as it seems there are no tools for that as far as I know.
Or, may I propose using this workaround to Estimate the completion time on Firestore (Firebase) using gcloud firestore operations describe (gcloud doc).
Estimate the completion time
A request for the status of a long-running operation returns the metrics workEstimated and workCompleted. Each of these metrics is returned in both number of bytes and number of entities:
workEstimated shows the estimated total number of bytes and documents an operation will process. Cloud Firestore might omit this metric if it cannot make an estimate.
workCompleted shows the number of bytes and documents processed so far. After the operation completes, the value shows the total number of bytes and documents that were actually processed, which might be larger than the value of workEstimated.
Divide workCompleted by workEstimated for a rough progress estimate. This estimate might be inaccurate, because it depends on delayed statistics collection.
That sounds appropriate to do, what do you think?
You may start the backup on a calm day/evening, describe the operation, and see how long it is supposed to take, then cancel it.
Then, the day you have your backup, you can do the same thing again to check how long it might take to restore it.
Other than this, we may give you an approximate answer that will probably omit many external factors that may mislead you while you can have Firestore itself do the job for you.