Inbound Customer Data Ingestion FAQ inbound-customer-data-ingestion-faq
Frequently asked questions about bringing offline data into Audience Manager.
Can you summarize the onboarding process?
The onboarding process consists of two steps described in Send Batch Data to Audience Manager Overview:
- Step 1: synchronize user IDs;
- Step 2: create and transfer your inbound data file, by adhering to the file format requirements.
Can you summarize the deployment process?
We recommend the following:
-
Work with your data provider to format the daily inbound data file according to the ۶Ƶ specifications. See the following documentation for file naming and syntax requirements:
-
Work with your ۶Ƶ consultant to transfer a test data file to ۶Ƶ for format verification.
-
Work with your ۶Ƶ consultant to produce a taxonomy suitable for interpreting the contents of the data file.
-
In the staging/development environment, confirm that the ID synchronization is configured to properly pick up the data provider’s visitor ID and transfer it to the Audience Manager servers in realtime.
-
Deploy DIL/ID synchronization to production. The ID synchronization will already be configured as a module within the DIL code by your ۶Ƶ consultant.
-
Transfer production data files to Audience Manager. Given the dependencies on ID synchronization mappings, it might make sense to begin transferring data up to one week after production-code deployment, although you can start transferring the data files as soon as code goes into production.
What FTP mode should I use to transfer compressed or encrypted files?
See File Compression for Inbound Data Transfer Files.
Can I upload an inbound data file (.sync or .overwrite file) before deploying Audience Manager code into production?
Yes. As long as you use a cross-device data source to store the CRM data that you upload, Audience Manager always stores the data. In fact, following the Profile Merge Rules enhancements that Audience Manager launched in October 2019 that allow for offline-only use cases, you can upload and action on data without deploying Audience Manager code into production at all. See:
- Overview of Profile Merge Rules Enhancements
- People-based Destinations - Personalization Based on Offline-Only Data
What time of day should I transfer my file?
Audience Manager checks for and processes files multiple times throughout the day. Upload your data whenever you’re ready.
How long does it take before data from an uploaded file is available for targeting?
Data is available for targeting after 48 hours. Also, do not interpret the “successful upload” email as a statement that the data is available. This only means that Audience Manager has picked up the file and completed the first step of processing.
How often should I send files and should these be full or incremental files?
As a best practice, send an incremental file once per day for new visitors and for visitors whose data has changed. Many Audience Manager customers send a full file once per month. However, these file intervals and increments are flexible. You should send data in increments and at times that make sense for you.
How long does Audience Manager keep my files on the server?
FTP files are removed after they’ve been processed. S3 files are removed after 30 days. Files that cannot be processed due to format, syntax, or other errors, are removed. See also, Privacy and Data Retention FAQ.
What’s the difference between full and incremental files?
-
Full: A full file overwrites all of your existing visitor profiles and replaces them with the data in your file. Full files are identified by the
.overwritetag appended to the file name. You can use a.overwritefile to reset visitor traits or remove stale, obsolete traits.note note NOTE The .overwrite files only overwrite Audience Manager profile data associated to this data provider. In other words, all Audience Manager data associated to the visitor remains intact after a .overwrite file has been processed. -
Incremental: An incremental file appends new data to your existing visitor profiles. Incremental files are identified by the
.synctag appended to the file name. Sending in an incremental file does not erase or overwrite existing profiles.
The following use cases demonstrate how these file types affect stored visitor profiles.
- Day 1
.syncfile contents:visitor123 = a,b,c - Day 2
.overwritefile contents:visitor123 = c,d,e - Day 3 visitor profile ID 123 contents:
c,d,e
- Day 1
.syncfile contents:visitor123 = a,b,c - Day 2
.syncfile contents:visitor123 = c,d,e - Day 3 visitor profile ID 123 contents:
a,b,c,d,e
For more information about full and incremental file types, see:
What happens if I send in a file with IDs for visitors that have never performed the on-page ID sync?
During processing, Audience Manager skips that record and moves on to the next. If a DPID (Data Provider ID) is set up as a cross-device DPID, data that is ingested before an ID synchronization is saved and is available for use shortly after the ID synchronization occurs.
What is the time stamp, what is it for, and can you provide an example?
Time stamps are used for logging and record keeping. They are required by the syntax used for a properly formatted inbound file name. See:
What is a Data Provider ID (DPID) and how do I get it?
Your ۶Ƶ consultant will assign a three-digit or four-digit DPID (Data Provider ID) to your particular data source. This ID is unique and does not change.
How large can the daily data files be?
See File Compression for Inbound Data Transfer Files.
Does Audience Manager support file compression?
Yes, see:
The primary key in my data source database is an email address. Is that considered personally identifiable information?
Yes. Audience Manager does not store email addresses in its database. Visitors should be assigned a randomly generated ID or a one-way-hashed version of the email address prior to initiating ID synchronizations.
Are the data file contents case-sensitive? How about the ID synchronization?
There are two basic components of a data file: A User ID (see User ID in File Variables Defined) and profile data, usually in the form of key-value pairs or codes. The User ID is case-sensitive. Generally, profile or key-value data is not case-sensitive.
Should I use FTP or Amazon S3 to transfer files?
As best practice, we recommend Amazon S3 because the process is simpler. Audience Manager transfers FTP files to S3 regardless, so the process is more streamlined if you drop the files on Amazon S3 yourself. What’s more, customers uploading simultaneously to FTP share the FTP’s bandwidth, so they should expect slower upload speeds. Amazon S3 is also replicated and distributed, so it is generally safer and more reliable than an FTP server. For more information, see About Amazon S3.
How does Audience Manger process inbound files?
Audience Manager uses Amazon Simple Queue Service (SQS) for inbound data processing. Here is how this works:
- Audience Manager customers upload their inbound data to an Amazon S3 bucket.
- The data enters the Amazon SQS queue, waiting to be processed by Audience Manager.
- Audience Manager reads up to 119000 entries from the Amazon SQS queue and splits them in up to 3 batches. Files in each batch get processed simultaneously.
I need to upload multiple files at the same time. Will the files be processed simultaneously?
It depends. Audience Manager reads up to 119000 entries from the Amazon SQS queue and splits them in up to 3 batches. Your files will be processed simultaneously only if they end up in the same batch. However, due to the high amount of data ingested by Audience Manager on a daily basis, we cannot guarantee any file processing order.