In-Database Merge Queries for Looker on BigQuery are Now in Preview
Merging datasets through Looker’s merged results feature has been a powerful way for Looker users to bring together data from different Explores without needing a developer to rewrite your LookML. But query row limits can reduce the scope of potential analysis. Today, we are introducing the option to merge BigQuery datasets with Looker’s new in-database merge query capability, massively expanding your opportunity to get grounded analysis from even more of your data.
From Memory to the Database
Historically, Looker has processed merged results in its internal memory - a quick way to join smaller datasets (up to 5,000 rows per query). But as many organizations’ data volumes have grown, so have your expectations.
With this launch in preview, if your queries reside on the same BigQuery connection, Looker can shift that heavy lifting directly to the database. By performing the join, specifically a left join, inside BigQuery, you get the ease of Looker’s UI, combined with the raw power of your data warehouse. This architectural shift allows you to merge datasets at the scale of millions, or billions, of rows.
Why This Matters
By leveraging the native compute power of BigQuery, you gain three immediate advantages when merging queries:
-
Massive Scale: Say goodbye to the 5,000-row limit. Because the join happens in the database, you can now merge datasets of virtually any size.
-
Boosted Performance: Leveraging BigQuery’s processing power means your complex merges run faster and more efficiently.
-
Full Dataset Integrity: In the past, row limits on source queries could truncate the data before the join happened. With in-database merges, Looker intelligently disregards those source row limits to ensure the join happens across your entire dataset.
A Simplified UI Experience
We’ve kept the workflow you are familiar with, but added some smart indicators to help you along:
-
Automatic Detection: If your merge is eligible to run in-database, Looker will show a banner in the Merged Results dialog letting you know it’s optimizing the query for you.
-
Smart Limits: You’ll notice a new Merged Results field in your results table. While Looker joins the full datasets in the background, this field simply controls how many rows are displayed to keep your browser snappy, without restricting the actual data processing.
Getting Started
Turn on the In-database merge queries preview toggle to try this feature.
We’re excited to see how this helps you tackle even larger, more complex data challenges with ease.