Send BigQuery Results to Datadog?
The Problem
We found discrepencies between two datasets and go on a journey to fix those discrepencies. Also, we would like visualize our progress and alert us when production data is bad again.
Since we already have custom SQL to BigQuery, and we are already using Datadog, the idea is run the query every once a while and send the query results as custom metrics to Datadog.
I was looking for out of box solution, but it turns out there isn’t any.
Current BigQuery + Datadog integration documentation is all about collecting system metrics. Datadog has the support to run custom queries against MySQL, but there isn’t anything like that for BigQuery.
Temporary Solution
We ended up using Google Data Studio. Their documentation is a bit concise, and it took me a little time to configure the a simple report to pull data from BigQuery.
Manage data source:
Enter custom query (Note the CUSTOM QUERY tab on the left):
It took a little time to adjust to their UI, but essentially it is like building SQL queries:
Closing Thoughts
With Google Data Studio, we can at least visualize the count of records that we are interested in. Also, it allows us to set the refreshness for the data source, or manually refresh it. So it satisfies our need in the short term.
However, we can’t set up any alert from there like we do in Datadog. It would be nice to see more integration support from Datadog.