New to looker and would like to implement some tests for ETL validation (examples below from @rebecca21 ‘s post on this topic)
This document has some limited examples…test | Looker | Google Cloud
Does anyone have examples they can share of these? Especially interested in #5 and how to write a test with queries. I’d likely be counting, summing, taking the max of fields and comparing them to other time periods or current date.
Tests for ETL validation:
- Data exists: count measure is greater than 0
- Data is sufficient: count measure is greater than or equal to…
- Data is not redundant: count of primary key field is equal to 1 [the Docs page shows an example]
- All data is available: count or sum measure is equal to…for some filters [picking a fixed timeframe in the past works best]
- Data is recent: max of some timestamp is greater than or equal to… [You can do something like DATE_ADD(current_date(), interval -1 day) so you are always getting “yesterday” without hard-coding a date here]