Search documentation...

K
ChangelogBook a demoSign up

Google Cloud Pub/Sub

Connect to your internal microservices via a message queue

Overview

Hightouch integrates directly with Google Cloud Pub/Sub to support high-throughput, distributed, or asynchronous workloads, letting you build a custom connector to your internal systems.

Getting started

Connect to your Google Cloud account

When setting up the Google Cloud Pub/Sub destination for the first time, you need to enter your Google Cloud Credentials to give Hightouch access to your Google Cloud account. Hightouch needs permission to send messages to Google Cloud Pub/Sub on your behalf. Consult our GCP documentation for guidance on how to create a Google Cloud credential.

Entering Google Cloud Credentials in Hightouch

Project details

First, you need to enter your GCP Project ID and Regional API Endpoint.

Grant permissions

Hightouch believes in the principle of least privilege. We ask for no more permissions than necessary. For the Google Cloud Pub/Sub destination, Hightouch requires roles/pubsub.publisher and roles/pubsub.viewer. Ensure you run the command in your Google Cloud Console to grant Hightouch's service account the correct permissions.

Grant permissions to Google Cloud service account

Syncing data

Once you've authenticated your Google Cloud account in Hightouch, entered project details, and granted the correct permissions, you can configure a sync that sends messages whenever rows are added, changed, or removed in your model.

Configure your events trigger

Hightouch monitors your data model for added, changed, and removed rows. In this step, you specify which of these events should trigger message publishing.

Events to trigger message publishing

Choose your topic

In this step, you choose which topics to publish the messages to. Hightouch allows you to sync to existing topics that are already in your Google Cloud Pub/Sub.

Toggle use column to send message to multiple queues

Customize your message

Customize message data options

In this step, you tell Hightouch how to build the JSON message data object using data from your model.

This destination offers three methods of composing a JSON object:

Use JSON editor

Selecting the JSON editor method in the Hightouch UI

With the JSON editor, you can compose any JSON object using the Liquid template language. This is particularly useful for complex message data bodies containing nested objects and arrays, which can sometimes be difficult to model entirely in SQL.

Suppose your data model looks like this:

full_nameageemail_addressphone_number
John Doe30john@example.com+14158675309

And you want your message data like this:

{
  "name": "John Doe",
  "age": 30,
  "contact_info": [
    {
      "type": "email",
      "value": "john@example.com"
    },
    {
      "type": "phone",
      "value": "+14158675309"
    }
  ]
}

Your Liquid template should look like this:

{
  "name": "{{row.full_name}}",
  "age": {{row.age}},
  "contact_info": [
    {
      "type": "email",
      "value": "{{row.email_address}}"
    },
    {
      "type": "phone",
      "value": "{{row.phone_number}}"
    }
  ]
}

This makes it so you can reference any column using the syntax {{row.column_name}}. You can also use advanced Liquid features to incorporate control flow and loops into your dynamic message data.

When injecting strings into your JSON object, be sure to surround the Liquid tag in double quotes.

Use one column from model

Selecting using one column from the model as the JSON construction method in the Hightouch UI

If you're already storing JSON data in your source, or if you have the ability to construct a JSON object using SQL, you can select one column in your model that already contains the full message data.

This setting is commonly used when syncing web events that have already been collected and stored as JSON objects in your database.

Use multiple columns from model

Selecting using multiple columns from the model as the JSON construction method in the Hightouch UI

For the simplest use cases, Hightouch can construct a JSON object with key/value pairs based on multiple columns in your model.

Suppose your model looks like this:

emailfirst_namelast_name
alice.doe@example.comAliceDoe
bob.doe@example.comBobDoe
carol.doe@example.comCarolDoe

The field mapping in the screenshot above would generate the following message data for the first row:

{
  "customer_first_name": "Alice",
  "customer_last_name": "Doe",
  "customer_email": "alice.doe@example.com"
}

You can use the field mapper to rename fields. For example, first_name can be mapped to customer_first_name.

Configure optional message properties

Along with your row data in JSON format, you can also optionally include an ordering key to configure the order your topic receives your message and metadata fields as additional attributes.

orderingKey

This is a string field that ensure subscribers receive messages in a specific order. Hightouch automatically tries to cast the value to a string. If we can't cast the value to a string then it is sent as null.

attributes

This is an object containing key/value pairs of custom mapping fields. Values can be text or byte strings. Attributes can be text strings or byte strings. You can have at most 100 attributes per message. Attribute keys should not start with goog and should not exceed 256 bytes.

Configure initial sync behavior

In this step, you tell Hightouch how to handle rows present in your model results during the first sync run.

Certain workflows may require performing a backfill of all rows during the initial sync. For other use cases, you might only want to send messages in response to future data changes.

Declaring how to handle initial sync behavior in Hightouch

Tips and troubleshooting

Common errors

To date, our customers haven't experienced any errors while using this destination. If you run into any issues, please don't hesitate to . We're here to help.

Live debugger

Hightouch provides complete visibility into the API calls made during each of your sync runs. We recommend reading our article on debugging tips and tricks to learn more.

Sync alerts

Hightouch can alert you of sync issues via Slack, PagerDuty, SMS, or email. For details, please visit our article on alerting.

Ready to get started?

Jump right in or a book a demo. Your first destination is always free.

Book a demoSign upBook a demo

Need help?

Our team is relentlessly focused on your success. Don't hesitate to reach out!

Feature requests?

We'd love to hear your suggestions for integrations and other features.

Last updated: Nov 30, 2022

On this page

OverviewGetting startedConnect to your Google Cloud accountProject detailsGrant permissionsSyncing dataConfigure your events triggerChoose your topicCustomize your messageConfigure optional message propertiesConfigure initial sync behaviorTips and troubleshootingCommon errorsLive debuggerSync alerts

Was this page helpful?