Firestore Memory limit issue

Google Cloud Database
1

MongoServerError[InvalidArgument]: The query failed since it attempted to use 128.01 MiB (134,225,920 B) when the limit is 128.00 MiB (134,217,728 B).

Getting this issue in most of the query we use

eg:


db.Customer.aggregate([ { $match: { merchantId: { $in: ["MER000023"] }, discriminator: "customer" } }, { $group: { _id: 1, n: { $sum: 1 } } } ])

Uncaught:

MongoServerError[InvalidArgument]: The query failed since it attempted to use 128.00 MiB (134,221,824 B) when the limit is 128.00 MiB (134,217,728 B).

 {

    v: 2,

    key: { discriminator: 1, merchantId: 1 },

    name: 'mid',

    sparse: false,

    unique: false,

    firestoreOptions: { multikey: true }

 {
    v: 2,
    key: { merchantId: 1 },
    name: 'merchantId_2',
    sparse: false,
    unique: false,
    firestoreOptions: { multikey: false }
  },

The above collection has overall data of 44Lakh, but even having the proper index .it is getting this issue.

2

This means the group aggregate is pulling more than 128MB of data in memory. The index is not the issue, there is just too much data pulled. Need more aggressive filtering here to fix the issue.

3 
db.Customer.aggregate([{$group: {_id: "$merchantId",count: { $sum: 1 }}},{$sort: { _id: 1 }} ])
[
{ _id: [ 'MER000023' ], count: 4975724 },
{ _id: [ 'MER000024' ], count: 1 },
{ _id: [ 'MER000025' ], count: 4 }
]

This group aggregation is working, but when filtered the data based on the field mentioned before not working. what would be the best solution and is there any way to increase the memory limit ,and also allow disk usage is also not applicable in firestore.

4

You cannot increase the memory limit but here you might need some tuning. Try below:

1. Optimization of the Index Structure (Crucial)

Your current compound index is:

{ discriminator: 1, merchantId: 1 }

However, in your aggregation $match stage, you filter on merchantId and discriminator:

{ $match: { merchantId: { $in: [“MER000023”] }, discriminator: “customer” } }

To optimize how the engine uses memory, try the following:

  • Match Index Order to Query Constraints: Ensure your compound index matches the exact equality fields of your filter first. If merchantId is heavily filtered, ensure it occupies a highly selective position in your index.

  • Avoid Multikey Indexes for Aggregation Filters: Your mid index has firestoreOptions: { multikey: true }.

  • Multikey indexes (indexes on arrays) require significantly more memory footprint and overhead to scan and process than single-value indexes. If discriminator or merchantId are not arrays, verify that the index is created with multikey: false to drastically reduce the index scan memory usage.

2. Redesign Aggregation with Pagination or Sub-Queries

Running a global grouping/counting aggregation over millions of documents at once will inevitably hit the 128 MiB query memory barrier.

  • Use Paginated Batches: Instead of querying all data in one aggregate call, split your aggregation by sub-segments of your dataset (e.g., partitioning by date ranges, or smaller IDs) and merge the counts on the application side.

  • Reduce the Match Scope: Add more granular filters (like a creation date range) inside the $match block so that the engine loads far fewer than 4.4 million documents into memory at once.

3. Leverage Sharding/Scale out with App Design

If your application depends on real-time counts across 44 Lakh documents:

  • Maintain Pre-computed Counters: Consider incrementing/decrementing a counter in a metadata document whenever a customer is added or deleted. Running real-time $group sums over millions of records on every query is highly inefficient for any Document DB and quickly exhausts execution limits.

Hope this helps.

5

previously i have tried this indexing order and mutikey false. i found 2 things

  1. The query still doesn’t work
  2. In the execution plan i still could see mutikey true, even when the created index is mutikey false
v: 2,

    key: { merchantId: 1, discriminator: 1 },

    name: 'mid_disc',

    sparse: false,

    unique: false,

    firestoreOptions: { multikey: false }

  },

db.Customer.find({

|   merchantId: { $in: ["MER000023"] },

|   discriminator: "customer"

| }).limit(8).hint("mid_disc").explain("executionStats")

{

  ok: 1,

  queryPlanner: { namespace: '' },

  executionStats: {

    nReturned: Long('8'),

    requestPeakMemoryUsage: '4.00 KiB (4,096 B)',

    dataBytesRead: '2.71 KiB (2,770 B)',

    totalDocsExamined: Long('8'),

    totalKeysExamined: Long('8'),

    readUnits: Long('9'),

    executionTimeMillis: Long('65'),

    executionStages: {

      stage: 'COMPUTE',

      '$out1': 'toBson(map_set($record_1, "__id__", $__id___1, "__key__", unset))',

      isQueryResult: true,

      nReturned: Long('8'),

      executionTimeMillisEstimate: Long('65'),

      inputStage: {

        stage: 'COMPUTE',

        '$id1': '_id($__key___2)',

        nReturned: Long('8'),

        executionTimeMillisEstimate: Long('65'),

        inputStage: {

          stage: 'FETCH',

          order: 'PRIMARY_KEY_ORDER',

          source: '**/Customer',

          inputRecordId: '$rid_2',

          outputBindings: {

            '$__key___2': 'row().__key__',

            '$record_1': 'row[* - { __create_time__, __update_time__ }]()'

          },

          output: [ '$__key___2', '$record_1' ],

          nReturned: Long('8'),

          executionTimeMillisEstimate: Long('65'),

          dataBytesRead: '2.02 KiB (2,066 B)',

          inputStage: {

            stage: 'LIMIT',

            limit: Long('8'),

            nReturned: Long('8'),

            executionTimeMillisEstimate: Long('63'),

            inputStage: {

              stage: 'IDXSCAN',

              fields: [ null, null, '$rid_2' ],

              index: '**/Customer (merchantId ASC, discriminator ASC)@[id = CICAgLiI_5oK, isMultikeyIndex = true, multikeyPaths = {merchantId=[merchantId], discriminator=[]}]',

              ranges: '/\n|----["MER000023"]\n     |----["customer"]\n',

              nReturned: Long('8'),

              executionTimeMillisEstimate: Long('63'),

              docsExamined: Long('8'),

              dataBytesRead: '704 B'

            }

          }

        }

      }

    }

  }

}



db.Customer.aggregate([ { $match: { merchantId: { $in: ["MER000023"] }, discriminator: "customer" } }, { $group: { _id: 1, n: { $sum: 1 } } } ])

Uncaught:

MongoServerError[InvalidArgument]: The query failed since it attempted to use 128.00 MiB (134,221,824 B) when the limit is 128.00 MiB (134,217,728 B).
6

Can you please raise a support ticket with Google Cloud for further analysis?