Power real-time uses cases from Google Cloud’s BigQuery with continuous queries

The data warehouse isn’t just an analytical foundation; it can be a source for action. We’ve helped customers activate their BigQuery data for years and are thrilled today to announce our integration with BigQuery’s latest innovation: Continuous Queries.

BigQuery continuous queries operate non-stop, processing SQL statements, allowing companies to analyze, transform, and replicate data in real time as new events arrive in BigQuery. The impact: companies can update their data immediately when it reaches BigQuery and then immediately sync that data via Hightouch to the other downstream applications and tools they need it in. This unlocks real-time use cases powered by data in BigQuery, such as triggered marketing campaigns, immediate personalization, anomaly detection, and real-time analytics.

“Continuous queries enable teams to power real-time use cases directly from Google Cloud’s BigQuery, with flexible and user-friendly SQL”, said Naveen Punjabi, Director of Analytics and Data Science Partnerships at Google Cloud. “These capabilities fit seamlessly in our robust data and partner ecosystem, so we’re thrilled to see companies like Hightouch leverage them to support real outcomes for their customers.”

How it Works

Users can create and execute continuous queries either through Google Cloud Console, bq-command-line-interface, or the BigQuery API after choosing Quote mode as “continuous query”. From that point forward, the continuous query will continue to run and generate as new data enters BigQuery. Below is the architecture of how this solution will work. BigQuery continuous queries are issued with Pub/Sub as the export option; then Hightouch consumes data from pub/sub via a source connector, processes data as needed, and pushes it to downstream applications to open up a lot of use cases and possibilities.

The outputs of this continuous query can also be written directly to tables in BigQuery and Bigtable. Real-time action based on this new data is supported via Pub/Sub data streams, which is what Hightouch integrates with to sync that data to any downstream tool or application destination in real time.

Within BigQuery, you can monitor your continuous queries with pre-existing toolsets such as BigQuery Jobs Information_Schema views, BigQuery execution graphs, the personal/project history of the BigQuery SQL editor, bq ls CLI commands, and the BigQuery jobs.listAPI. Downstream, you can use Hightouch to monitor the real-time data syncs to your end destinations and set up highly customizable real-time alerts to address anomalies.

For more details, check out BigQuery’s continuous query documentation, and our documentation of how we activate data in real time with a Pub/Sub integration.

Getting started

There’s a perception that data warehouses and composable Customer Data Platforms (CDPs) aren’t fast enough for real-time use cases. However, this is no longer the case. We are excited to announce our support for real-time use cases, facilitated by BigQuery's continuous queries, alongside our other solutions such as Streaming Reverse ETL and event streaming.

If you want to learn more about how you can power business value with your BigQuery data, grab some time with our solutions engineers.