Hightouch needs the following IAM actions to store items in your bucket:
Action
Details
s3:PutObject
Grants permission to add an object to a bucket
s3:AbortMultipartUpload
Grants permission to to abort or cancel an ongoing multipart upload in an S3 bucket
s3:ListMultipartUploadParts
Grants permission to list the parts of a multipart upload in an S3 bucket
s3:ListBucketMultipartUploads
Grants permission to list all the ongoing multipart uploads within an S3 bucket
For small file transfers, Hightouch only uses the s3:PutObject action.
For large files, multi-part uploads may be necessary, hence the other required actions.
You can use the following JSON sample to create your IAM policy:
Go to the Destinations overview page and click the Add destination button. Select Amazon S3 and click Continue. You can then authenticate Hightouch to Amazon S3 by entering your Bucket Name. The Bucket Name should just be the name of the bucket, not a URL.
Once you've set up your S3 destination and have a model to pull data from, you can set up your sync configuration to begin syncing data. Go to the Syncs overview page and click the Add sync button to begin. Then, select the relevant model and the S3 destination you want to sync to.
The filename or object key field lets you specify the parent directory and the name of the file you want to use for your results.
You can include timestamp variables in the filename, surrounding each with {}.
Hightouch supports these timestamp variables:
YYYY: Represents the full year in four digits.
YY: The last two digits of the year.
MM: Two-digit month format (01-12).
DD: Two-digit day format (01-31).
HH: Two-digit hour format in 24-hour clock (00-23).
mm: Two-digit minute format (00-59).
ss: Two-digit second format (00-59).
ms: Three-digit millisecond format.
X: Unix timestamp in seconds.
x: Unix timestamp in milliseconds.
All dates and times are UTC.
For example, you could enter upload/{YYYY}-{MM}-{DD}-{HH}-{mm}-result.json to dynamically include the year, month, date, hour, and minute in each uploaded file.
Hightouch would insert each file in the upload directory, which would need to already exist in your bucket.
You can also use other variable values to include sync metadata in the filename:
{model.id}
{model.name}
{sync.id}
{sync.run.id}
If a file already exists at the path you entered at the time of a sync,
Hightouch overwrites it. To keep different versions of the same results file,
you can enable
versioning
in your bucket, or your app can copy the data to another location.
If you are using an audience and would like to include the audience name, you
will still use {model.name}.
By default, Hightouch uses the timestamp of the sync run to fill in timestamp variables. You can optionally include an offset in seconds. For example, if you want the filename's date to be 24 hours before the sync takes place, enter -86400 (24 hours * 60 minutes * 60 seconds). If you want the filename's data to be one hour after the sync takes place, you would enter 3600 (60 minutes * 60 seconds).
You can export all columns exactly as your model returns them or choose to export specific ones.
If you need to rename or transform any column values you're syncing, you can use the advanced mapper to do so. If you choose this option, Hightouch only syncs the fields you explicitly map.
The preceding example shows how to selectively export the customer_id, email, first_name and last_name, columns. These columns are mapped to new fields in the destination file as id, email, and name—a templated concatenation of first_name and last_name—respectively. Hightouch exports these fields to new fields in the file and ignores all other columns from your results.
You can select how Hightouch should handle empty results files. Empty result files can occur if your model's query results haven't changed since the last sync.
You can select whether to skip empty files. If you skip empty files, it means Hightouch won't export any files if your model's query results haven't changed since your last sync.
You can enable gzip compression to reduce the size of your exported files. Note
that Hightouch does not automatically add the .gz extension to your files. You
can add this or any other extension to your filename.
The ACCESS_DENIED error is a permissions-related error. Ensure your Hightouch AWS credentials have read/write access to the S3 bucket specified in your destination. The minimal permissions required are s3:GetObject, s3:PutObject, and s3:ListBucket.