Amazon S3 name and file size requirements for inbound data files amazon-s-name-and-file-size-requirements-for-inbound-data-files
Describes the required fields, syntax, naming conventions and file sizes you need to follow when sending data to Audience Manager. Set the names and sizes of your files according to these specifications when you send data to an Audience Manager / Amazon S3 directory.
monospaced text
, italics, brackets [ ]
( )
, etc.) in this document indicate code elements and options. See Style Conventions for Code and Text Elements for more information.File name syntax file-name-syntax
S3 file names contain the following required and optional elements:
-
S3 prefix:
s3n://AWS_directory/partner_name/date=yyyy-mm-dd/
-
File name elements:
ftp_dpm_DPID[_DPID_TARGET_DATA_OWNER]_TIMESTAMP(.sync|.overwrite)[.SPLIT_NUMBER][.gz]
For other accepted file name formats, see Custom Partner Integrations.
Name elements
The table defines the elements in an S3 file name.
AWS_directory
date=yyyy-mm-dd
DPID
An lD that tells Audience Manager if a data file contains your own user IDs, Android IDs, iOS IDs, or other IDs belonging to global data sources. Accepts the following options:
- Data Source ID (also known as Data Provider ID): This is a unique ID that Audience Manager assigns to a data source (refer to the Audience Manager index of IDs). Use this assigned ID in a file name when sending in data that contains your own user IDs. For example,
...ftp_dpm_21_123456789.sync
tells Audience Manager to onboard data to IDs belonging to data source 21. - Android IDs (GAID): Use ID 20914 in a data file name if it contains Android IDs. You need to use the field
_DPID_TARGET_DATA_OWNER
when you use Android IDs. For example,...ftp_dpm_20914_DPID_TARGET_DATA_OWNER_123456789.sync
tells Audience Manager that the data file contains Android IDs only and the IDs should qualify for the traits belonging to the_DPID_TARGET_DATA_OWNER
data source. - iOS IDs (IDFA): Use ID 20915 in a data file name if it contains iOS IDs. You need to use the field
_DPID_TARGET_DATA_OWNER
when you use iOS IDs. For example,...ftp_dpm_20915_DPID_TARGET_DATA_OWNER_123456789.sync
tells Audience Manager that the data file contains iOS IDs only and the IDs should qualify for the traits belonging to the_DPID_TARGET_DATA_OWNER
data source. - IDs belonging to other global data sources: You can onboard Roku IDs for Advertising (RIDA), Microsoft Advertising IDs (MAID), and other IDs. Use the ID corresponding to each data source, as described in the global data sources article.
Note: Do not mix ID types in your data files. For example, if your file name includes the Android identifier, don't put iOS IDs or your own IDs in the data file.
_DPID_TARGET_DATA_OWNER
This field tells Audience Manager which data source to onboard data to. This field is mandatory if you set the DPID to an Android ID or iOS ID or another ID belonging to global data sources. This lets Audience Manager link the file data back to your organization.
This target data source needs to be owned by your company. For second-party data sharing purposes, in order to ingest data into a target data source belonging to another company, you must have an access mapping between your company and the target data source. Contact your ÃÛ¶¹ÊÓƵ consultant or Customer Support in order to set up the mapping.
Important note: You do not need to request a mapping for existing data sharing relationships (for target data sources belonging to other companies into which you onboarded data prior to March 14, 2022). The mapping is also not required when onboarding data into target data sources that belong to your PID.
For example:
...ftp_dpm_33_21_1234567890.sync
tells Audience Manager that you're qualifying customer IDs belonging to data source 33 for traits or signals belonging to data source 21.- Android IDs (GAID):
...ftp_dpm_20914_21_1234567890.sync
tells Audience Manager that the data file contains Android IDs only and the IDs should qualify for the traits belonging to data source 21. - iOS IDs (IDFA):
...ftp_dpm_20915_21_1234567890.sync
tells Audience Manager that the data file contains iOS IDs only and the IDs should qualify for the traits belonging to data source 21. - IDs belonging to other global data sources:
...ftp_dpm_121963_21_1234567890.sync
tells Audience Manager that the data file contains Roku IDs only and the IDs should qualify for the traits belonging to data source 21. Use the ID corresponding to each data source, as described in the global data sources article.
partner_name
TIMESTAMP
(.sync|.overwrite)
Synchronization options that include:
sync
: Normal scenario when third-party data providers send traits on a per-user basis to be added or removed in the Audience Manager system.overwrite
: Lets data providers send a list of traits on a per-user basis that should overwrite all of this user's existing third-party traits for this data provider in the Audience Manager. You do not need to include all of your users in an overwrite file. Include only those users that you want to change.
[SPLIT_NUMBER]
An integer. Used when you split large files into multiple parts to improve processing times. The number indicates which part of the original file you're sending in.
For efficient file processing, split your data files as indicated:
- Uncompressed: 1 GB
- Compressed: 200-300 MB
See the first 2 file name examples below.
[.gz]
When sending files to Amazon S3, use gzip compression only. When compressed, these files get the .gz
extension. Do not use .zip compression.
Compressed files must be 3 GB or smaller. If your files are larger, please talk to Customer Care. Although Audience Manager can handle large files, we may be able to help you reduce the size of your files and make data transfers more efficient. See File Compression for Inbound Data Transfer Files.
File name examples file-name-examples
The following examples show properly formatted file names. Your file names could look similar.
s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.sync.1.gz
s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.sync.2.gz
s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.sync
s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_567_1366545717.sync.gz
s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.overwrite
You can download the sample file if you want additional examples. This file has been saved with the .overwrite
file extension. Open it with a simple text editor.
Accepted file sizes accepted-file-sizes
Consider the figures below for fastest/earliest processing of your files as well as for file size limitations when you send data to an Audience Manager / Amazon S3 directory.
Line length limits line-limits
Inbound data files have a line length limit of 102400 bytes. Lines exceeding this limit are excluded from the transfer.