Create an ÃÛ¶¹ÊÓƵ Analytics source connection in the UI
This tutorial provides steps for creating an ÃÛ¶¹ÊÓƵ Analytics source connection in the UI to bring ÃÛ¶¹ÊÓƵ Analytics report suite data into ÃÛ¶¹ÊÓƵ Experience Platform.
Getting started
This tutorial requires a working understanding of the following components of Experience Platform:
- Experience Data Model (XDM) System: The standardized framework by which Experience Platform organizes customer experience data.
- Real-Time Customer Profile: Provides a unified, real-time consumer profile based on aggregated data from multiple sources.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Key terminology
It is important to understand the following key terms used throughout this document:
- Standard attribute: Standard attributes are any attribute that is pre-defined by ÃÛ¶¹ÊÓƵ. They contain the same meaning for all customers and are available in the Analytics source data and Analytics schema field groups.
- Custom attribute: Custom attributes are any attribute in the custom variable hierarchy in Analytics. Custom attributes are used within an ÃÛ¶¹ÊÓƵ Analytics implementation to capture specific information into a report suite, and they can differ in their use from report suite to report suite. Custom attributes include eVars, props, and lists. See the following Analytics documentation on conversion variables for more information on eVars.
- Any attribute in Custom field groups: Attributes that originate from field groups created by customers are all user-defined and are considered to be neither standard nor custom attributes.
- Friendly names: Friendly names are human-provided labels for custom variables in an Analytics implementation. See the following Analytics documentation on conversion variables for more information on friendly names.
Create a source connection with ÃÛ¶¹ÊÓƵ Analytics
- A dataflow that does a 13-month backfill of historical report suite data into data lake. This dataflow ends when the backfill is complete.
- A dataflow flow which sends live data to data lake and to Real-Time Customer Profile. This dataflow runs continuously.
In the Platform UI, select Sources from the left navigation to access the Sources workspace. The Catalog screen displays a variety of sources that you can create an account with.
You can select the appropriate category from the catalog on the left-hand side of your screen. You can also use the search bar to narrow down the displayed sources.
Under the ÃÛ¶¹ÊÓƵ applications category, select ÃÛ¶¹ÊÓƵ Analytics and then select Add data.
Select data
The Analytics source add data step provides you with a list of Analytics report suite data to create a source connection with.
A report suite is a container of data that forms the basis of Analytics reporting. An organization can have many report suites, each containing different datasets.
You can ingest report suites from any region (United States, United Kingdom, or Singapore) as long as they are mapped to the same organization as the Experience Platform sandbox instance in which the source connection is being created in. A report suite can be ingested using only a single active dataflow. A report suite that is not selectable has already been ingested, either in the sandbox that you are using or in a different sandbox.
Multiple in-bound connections can be made to bring multiple report suites into the same sandbox. If the report suites have differing schemas for variables (such as eVars or events), they should be mapped to specific fields in the custom field groups and avoid data conflicts using Data Prep. Report suites can only be added to a single sandbox.
To create an Analytics source connection, select a report suite and then select Next to proceed.
Mapping
Before you can map your Analytics data to target XDM schema, you must first select whether you are using a default schema or a custom schema.
A default schema creates a new schema on your behalf, containing the ÃÛ¶¹ÊÓƵ Analytics ExperienceEvent Template field group. To use a default schema, select Default schema.
With a custom schema, you can choose any available schema for your Analytics data, as long as that schema has the ÃÛ¶¹ÊÓƵ Analytics ExperienceEvent Template field group. To use a custom schema, select Custom schema.
The Mapping page provides an interface to map source fields to their appropriate target schema fields. From here, you can map custom variables to new schema field groups and apply calculations as supported by Data Prep. Select a target schema to start the mapping process.
The Map standard fields section displays panels for Standard mappings applied, Non matching standard mappings and Custom mappings. See the following table for specific information regarding each category:
To preview the Analytics ExperienceEvent template schema field group, select View in the Standard mappings applied panel.
The ÃÛ¶¹ÊÓƵ Analytics ExperienceEvent Template Schema Field Group page provides you with an interface to use for inspecting the structure of your schema. When finished, select Close.
Platform automatically detects your mapping sets for any friendly name conflicts. If there are no conflicts with your mapping sets, select Next to proceed.
Custom mappings
You can use Data Prep functions to add new custom mapping or calculated fields for custom attributes. To add custom mappings, select Custom.
Depending on your needs, you can select either Add new mapping or Add calculated field and proceed to create custom mappings for your custom attributes. For comprehensive steps on how to use Data Prep functions, please read the Data Prep UI guide.
The following documentation provides further resources on understanding Data Prep, calculated fields, and mapping functions:
Filtering for Real-Time Customer Profile filtering-for-profile
Once you have completed mappings for your Analytics report suite data, you can apply filtering rules and conditions to selectively include or exclude data from ingestion to the Real-Time Customer Profile. Support for filtering is only available for Analytics data and data is only filtered prior to entering Profile. All data are ingested into the data lake.
Additional information on Data Prep and filtering Analytics data for Real-Time Customer Profile
- You can use the filtering functionality for data that is going to Profile, but not for data going to data lake.
- You can use filtering for live data, but you cannot filter backfill data.
- The Analytics source does not backfill data into Profile.
- If you utilize Data Prep configurations during the initial setup of an Analytics flow, those changes are applied to the automatic 13-month backfill as well.
- However, this is not the case for filtering because filtering is reserved only for live data.
- Data Prep is applied to both streaming and batch ingestion paths. If you modify an existing Data Prep configuration, those changes are then applied to new incoming data across both streaming and batch ingestion pathways.
- However, any Data Prep configurations do not apply to data that has already been ingested into Experience Platform, regardless of whether it is streaming or batch data.
- Standard attributes from Analytics are always mapped automatically. Therefore, you cannot apply transformations to standard attributes.
- However, you can filter out standard attributes as long as they are not required in Identity Service or Profile.
- You cannot use column-level filtering to filter required fields and identity fields.
- While you can filter out secondary identities, specifically AAID and AACustomID, you cannot filter out ECID.
- When a transformation error occurs, the corresponding column results in NULL.
Row-level filtering
You can filter data for Profile ingestion at the row-level and the column-level. Row-level filtering allows you to define criteria such as string contains, equals to, begins, or ends with. You can also use row-level filtering to join conditions using AND
as well as OR
, and negate conditions using NOT
.
To filter your Analytics data at the row-level, select Row filter.
Use the left rail to navigate through the schema hierarchy and select the schema attribute of your choice to further drill down a particular schema.
Once you have identified the attribute that you want to configure, select and drag the attribute from the left rail to the filtering panel.
To configure different conditions, select equals and then select a condition from the dropdown window that appears.
The list of configurable conditions include:
- equals
- does not equal
- starts with
- ends with
- does not end with
- contains
- does not contain
- exists
- does not exist
Next, enter the values that you want to include based on the attribute that you selected. In the example below, Apple and Google are selected for ingestion as part of the Manufacturer attribute.
To further specify your filtering conditions, add another attribute from the schema and then add values based on that attribute. In the example below, the Model attribute is added and models such as the iPhone 13 and Google Pixel 6 are filtered for ingestion.
To add a new container, select the ellipses (...
) on the top right of the filtering interface and then select Add container.
Once a new container is added, select Include and then select Exclude from the dropdown window that appears.
Next, complete the same process by dragging schema attributes and adding their corresponding values that you want to exclude from filtering. In the example below, the iPhone 12, iPhone 12 mini, and Google Pixel 5 are all filtered from exclusion from the Model attribute, landscape is excluded from the Screen orientation, and model number A1633 is excluded from Model number.
When finished, select Next.
Column-level filtering
Select Column filter from the header to apply column-level filtering.
The page updates into an interactive schema tree, displaying your schema attributes at the column-level. From here, you can select the columns of data that you would like to exclude from Profile ingestion. Alternatively, you can expand a column and select specific attributes for exclusion.
By default, all Analytics go to Profile and this process allows for branches of XDM data to be excluded from Profile ingestion.
When finished, select Next.
Filter secondary identities
Use a column filter to exclude secondary identities from Profile ingestion. To filter secondary identities, select Column filter and then select _identities.
The filter only applies when an identity is marked as secondary. If identities are selected, but an event arrives with one of the identities marked as primary, then those do not get filtered out.
Provide dataflow details
The Dataflow detail step appears, where you must provide a name and an optional description for the dataflow. Select Next when finished.
Review
The Review step appears, allowing you to review your new Analytics dataflow before it is created. Details of the connection are grouped by categories, including:
- Connection: Displays the source platform of the connection.
- Data type: Displays the selected Report Suite and its corresponding Report Suite ID.
Monitor your dataflow monitor-your-dataflow
Once your dataflow is complete, select Dataflows in the sources catalog to monitor the activity and status of your data.
A list of existing Analytics dataflows in your organization appears. From here, select a target dataset to view its respective ingestion activity.
The Dataset activity page provides information on the progress of data that is being sent from Analytics to Experience Platform. The interface displays metrics such as the total of records in the previous month, the total of ingested records in the last seven days, and the size of data in the previous month.
The source instantiates two dataset flows. One flow represents backfill data and the other is for live data. Backfill data is not configured for ingestion into Real-Time Customer Profile but is sent to the data lake for analytical and data-science use-cases.
For more information on backfill, live data, and their respective latencies, read the Analytics source overview.
Delete your dataflow delete-dataflow
To delete your Analytics dataflow, select Dataflows from the top header of the sources workspace. Use the dataflows page to locate the Analytics dataflow that you want to delete and then select the ellipses (...
) beside it. Next, use the dropdown menu and select Delete.
- Deleting the live Analytics dataflow will also delete its underlying dataset.
- Deleting the backfill Analytics dataflow does not delete the underlying dataset, but will stop the backfill process for its corresponding report suite. If you delete the backfill dataflow, ingested data may still be viewed through the dataset.
Next steps and additional resources
Once the connection is created, the dataflow is automatically created to contain the incoming data and populate a dataset with your selected schema. Furthermore, data back-filling occurs and ingests up to 13 months of historical data. When the initial ingestion completes, Analytics data and be used by downstream Platform services such as Real-Time Customer Profile and Segmentation Service. See the following documents for more details:
The following video is intended to support your understanding of ingesting data using the ÃÛ¶¹ÊÓƵ Analytics Source connector: