Deleted data not compacted?

Google Cloud Database
1

For one of our tables, we are levering unlimited versions to store historical values. However, when we delete rows (i.e. all versions of a row), we don’t see any drop in storage, even after more than 1 week.

The rows are deleted using a Java Hadoop HBase client executing towards a (single location) Bigtable instance. We first do a row prefix scan using https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/Scan.html#setStartStopRowForPrefixScan(byte%5B%5D) and then sends batches of delete operations (https://hbase.apache.org/1.4/apidocs/org/apache/hadoop/hbase/client/Delete.html#Delete(byte[])) for the returned rows.

We can verify that after this operation, the rows are no longer returned from any scans on this table.

Is there an edge case with the compactor not being able to compact deleted rows in tables with unlimited versions?

2

Hi @omw,

When you delete data in Bigtable, it doesn’t get wiped out right away. Instead, the cells are marked for deletion and become unreadable immediately, but the actual cleanup happens during compaction, which can take up to a week. During that time, the deletion metadata might temporarily add a bit of overhead (a few KB per row), so storage might even look slightly higher before it drops.

If storage still doesn’t decrease after a reasonable time, it might be worth reaching out to our Google Cloud Support team for a deeper look.