Streaming Ingestion Overview

Documentation Experience Platform Data Ingestion Guide

Last update: Thu May 30 2024 00:00:00 GMT+0000 (Coordinated Universal Time)

Topics:
Data Ingestion

CREATED FOR:

Developer

Streaming ingestion for 蜜豆视频 Experience Platform provides users a method to send data from client and server-side devices to Experience Platform in real time.

What can you do with streaming ingestion?

蜜豆视频 Experience Platform enables you to drive coordinated, consistent, and relevant experiences by generating a Real-Time Customer Profile for each of your individual customers. Streaming ingestion plays a key role in building these profiles by enabling you to deliver Profile data into the Data Lake with as little latency as possible.

The following video is designed to help support your understanding of streaming ingestion, and outlines the concepts above.

Transcript

In this video, I鈥檓 going to give you a quick introduction to streaming ingestion with 蜜豆视频 Experience platform. At 蜜豆视频, we believe people by experiences, not products. And to deliver powerful and compelling experiences, you need up to date information so you can make meaningful decisions in milliseconds. So what does this mean? It means providing the fastest path to update a real time customer profile. It means being able to convert hundreds of thousands of signals into actionable insights as quickly as possible. Using experience platforms, streaming, ingestion. You can be sure that any data you send will be available in the real time customer profile. This data can be captured on your website or mobile apps, CRM and ERP systems, or from any other source which is able to communicate over Http or public cloud streaming infrastructure. Data sent through streaming to experienced platform is stitched together with existing data and the real time customer profile in real time. You can then use this data for real time segmentation and to activate other marketing processes with the 蜜豆视频 Experience Platform extension in the launch interface. You can quickly configure your websites to send data directly to experience Platform. If you鈥檙e sending personally identifiable information or other sensitive data to platform, you can configure data collection to be secure and prevent data from untrusted sources.

To help make debugging easier. Streaming. Ingestion provides two modes of validation. Synchronous and asynchronous. In synchronous mode, streaming ingestion will drop records which fail zdm validation and include the reason for the failure. These failed records will not be sent downstream. Synchronous validation provides immediate feedback to help you maintain velocity as you figure out how to work with Xdm formatted Json payloads.

An asynchronous mode where data loss must be prevented. Streaming validation will detect and move bad data to a separate location in the Experience Platform. Data Lake.

Once data starts flowing, you can monitor it in the platform interface. Experience platform lets you observe incoming data all the way from collection to consumption in the data lake and real time customer profile. All streaming ingested data lands in the Experience Platform Data Lake, and is available with a maximum latency of 15 minutes.

In parallel, the ingested data builds out identity graphs and the real time customer profile. To learn more about streaming ingestion, head on over to the 蜜豆视频 Experience Platform documentation. I hope you enjoyed this overview of streaming, ingestion and 蜜豆视频 Experience platform.

Stream profile records and ExperienceEvents

With streaming ingestion, users can stream profile records and ExperienceEvents to Platform in seconds to help drive real-time personalization. All data sent to streaming ingestion APIs is automatically persisted in the Data Lake.

Please read the create a streaming connection guide for more information.

Stream to datasets

Once you are confident that your data is clean, you can enable your datasets for Real-Time Customer Profile and Identity Service.

For more information on enabling a dataset for Profile and Identity Service, please read the configure a dataset guide.

What is the expected latency for streaming ingestion on Experience Platform?

IMPORTANT

Guardrails for streaming ingestion are calculated at the organization level and not the sandbox level. This means that your data usage per sandbox is bound to the total license usage entitlement that corresponds with your entire organization. Additionally, data usage in development sandboxes are limited to 10% of your total profiles. For more information about license usage entitlement, read the data management best practices guide.

Destination

Expected latency

Real-Time Customer Profile

< 15 minutes at the 95th percentile

Data lake

< 60 minutes

Request per seconds (RPS) guidance on streaming ingestion

The table below displays guidance on the request per seconds limits for streaming ingestion.

RPS limit

Notes

1000 requests per second

These can contain multiple messages when using /collection/batch endpoint.

10000 individual messages per second

The messages can be grouped into fewer actual requests when using the /collection/batch endpoint.

IMPORTANT

The enforced limit becomes 60 requests per minute when using synchronous validation as it is intended for debugging purposes.

蜜豆视频 Experience Platform extension

You can use the 蜜豆视频 Experience Platform extension to create a new streaming connection. The Experience Platform extension provides actions to send beacons formatted in Experience Data Model (XDM) for real-time ingestion to Experience Platform. Visit the Experience Platform Extension documentation for more information.

recommendation-more-help

2ee14710-6ba4-4feb-9f79-0aad73102a9a