I’m trying to import data into Firestore from a Bucket using Google Firestore’s Import/Export user-interface. The data to be imported was exported from another Firestore database (using Google’s Export user-interface), downloaded from that Bucket and uploaded into another Bucket. The import fails without reason (there’s no place in the UI to show a reason anyway). How do I figure this out?
And, I am billed everytime for the exports and imports. How rich is that?
To troubleshoot the import of data into Firestore from a Bucket, consider the following steps:
Check Bucket Permissions: Ensure that the service account used for the import has the appropriate permissions. It should have roles like Storage Object Viewer for the Cloud Storage bucket. You can verify and manage these permissions in the IAM & Admin > Service Accounts section of the Google Cloud Console.
Verify Exported File Integrity: Make sure that the exported data from Firestore is intact and unaltered. Firestore exports data in a specific format, including an overall_export_metadata file and other components. It’s crucial not to change these files or their structure.
Review Data Volume: While Firestore doesn’t explicitly limit the size of imports to 1GB, large data volumes can cause issues. If you’re dealing with a substantial amount of data, consider splitting it into smaller, manageable chunks for import.
Examine Import Logs: For more detailed information on what might have gone wrong during the import, check the Logs Explorer in the Google Cloud Console. These logs can provide insights into errors or issues encountered during the process.
Use the gcloud CLI: If the UI doesn’t provide enough information or if the issue persists, try using the gcloud CLI for the import. This tool often gives more detailed error messages and additional troubleshooting options. The command for importing data into Firestore is:
gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_FOLDER_PATH]
Replace [BUCKET_NAME] and [EXPORT_FOLDER_PATH] with the appropriate names for your bucket and export folder.
Understand Billing: Be aware that you are billed for both export and import operations in Firestore. These costs are based on the amount of data processed. To manage expenses effectively, monitor the size of the data being exported and imported. More details on Firestore pricing can be found on the Google Cloud website.
Why do you assume that I am so clueless? I’ve checked 1 and 2. 3 is not and issue because (like I said) I just exported it from another Firestore database. 4 shows nothing - NOTHING - related to the import process. I KNOW 6 is how it works and that’s what I’m complaining about. Gawd!
This is just terrible code. Using the gcloud CLI, it now says “ERROR: (gcloud.firestore.import) too many index entries for entity”. On WHICH entity? No idea. Maybe we should move to Amazon - we at least get decent support for the money we’re spending.
@ozayrMoorad I had an identical error message under two conditions:
1 When the name of the directory that has the files to import and the “prefix” of the overall_export_metadata were not identical. So the overall_export_metadata file in the directory (say) abc has to be named abc.overall_export_metadata.
2 When one of the included collections had a document that had an array that was longer than 40,000 bytes. I tracked that down based on a comment on StackOverflow.
Hope this helps.
