Data Prep mapping functions
Data Prep functions can be used to compute and calculate values based on what is entered in source fields.
Fields
A field name can be any legal identifier - an unlimited-length sequence of Unicode letters and digits, beginning with a letter, the dollar sign ($
), or the underscore character (_
). Variable names are also case sensitive.
If a field name does not follow this convention, the field name must be wrapped with ${}
. So, for example, if the field name is 鈥淔irst Name鈥 or 鈥淔irst.Name鈥, then the name must be wrapped like ${First Name}
or ${First\.Name}
respectively.
.
), you must use a backslash (\
) to escape special characters. For more information, read the guide on escaping special characters.If a field name is any of the following reserved keywords, it must be wrapped with ${}{}
:
new, mod, or, break, var, lt, for, false, while, eq, gt, div, not, null, continue, else, and, ne, true, le, if, ge, return, _errors, do, function, empty, size
Additionally, reserved keywords also include any of the mapper functions listed on this page.
Data within sub-fields can be accessed by using the dot notation. For example, if there was a name
object, to access the firstName
field, use name.firstName
.
List of functions
The following tables list all supported mapping functions, including sample expressions and their resulting outputs.
String functions string
- STRING: The strings that will be concatenated.
"Hi, there!"
\
: +, ?, ^, |, ., [, (, {, ), *, $, \
If you include multiple characters as the delimiter, the delimiter will be treated as a multi-character delimiter.- STRING: Required The string that needs to be split.
- REGEX: Optional The regular expression that can be used to split the string.
["Hi,", "there"]
- INPUT: Required The string that is being searched.
- SUBSTRING: Required The substring that is being searched for within the string.
- START_POSITION: Optional The location of where to start looking in the string.
- OCCURRENCE: Optional The nth occurrence to look for from the start position. By default, it is 1.
- INPUT: Required The input string.
- TO_FIND: Required The string to look up within the input.
- TO_REPLACE: Required The string that will replace the value within 鈥淭O_FIND鈥.
- INPUT: Required The input string.
- START_INDEX: Required The index of the input string where the substring starts.
- LENGTH: Required The length of the substring.
lcase
- INPUT: Required The string that will be converted to lowercase.
濒肠补蝉别(鈥淗别尝尝辞鈥)
ucase
- INPUT: Required The string that will be converted to uppercase.
耻肠补蝉别(鈥淗别尝尝辞鈥)
\
: \
. If you include multiple delimiters, the string will split on any of the delimiters present in the string. Note: This function only returns non-null indexes from the string, regardless of the presence of the separator. If all indexes, including nulls, are required in the resulting array, use the 鈥渆xplode鈥 function instead.- INPUT: Required The input string that is going to be split.
- SEPARATOR: Required The string that is used to split the input.
["Hello", "world"]
- SEPARATOR: Required The string that will be used to join the objects.
- OBJECTS: Required An array of strings that will be joined.
join(SEPARATOR, [OBJECTS])
join(" ", to_array(true, "Hello", "world"))
- INPUT: Required The string that is going to be padded out. This string can be null.
- COUNT: Required The size of the string to be padded out.
- PADDING: Required The string to pad the input with. If null or empty, it will be treated as a single space.
- INPUT: Required The string that is going to be padded out. This string can be null.
- COUNT: Required The size of the string to be padded out.
- PADDING: Required The string to pad the input with. If null or empty, it will be treated as a single space.
- STRING: Required The string you are getting the first 鈥渘鈥 characters for.
- COUNT: Required The 鈥渘鈥 characters you want to get from the string.
- STRING: Required The string you are getting the last 鈥渘鈥 characters for.
- COUNT: Required The 鈥渘鈥 characters you want to get from the string.
- STRING: Required The string you want to remove the whitespace from.
- STRING: Required The string you want to remove the whitespace from.
- STRING: Required The string you want to remove the whitespace from.
- STRING1: Required The first string you want to compare.
- STRING2: Required The second string you want to compare.
- STRING1: Required The first string you want to compare.
- STRING2: Required The second string you want to compare.
Regular expression functions
- STRING: Required The string that you are extracting the groups from.
- REGEX: Required The regular expression that you want the group to match.
- STRING: Required The string that you are checking matches the regular expression.
- REGEX: Required The regular expression that you are comparing against.
Hashing functions hashing
- INPUT: Required The plain text to be hashed.
- CHARSET: Optional The name of the character set. Possible values include UTF-8, UTF-16, ISO-8859-1, and US-ASCII.
- INPUT: Required The plain text to be hashed.
- CHARSET: Optional The name of the character set. Possible values include UTF-8, UTF-16, ISO-8859-1, and US-ASCII.
- INPUT: Required The plain text to be hashed.
- CHARSET: Optional The name of the character set. Possible values include UTF-8, UTF-16, ISO-8859-1, and US-ASCII.
- INPUT: Required The plain text to be hashed.
- CHARSET: Optional The name of the character set. Possible values include UTF-8, UTF-16, ISO-8859-1, and US-ASCII.
- INPUT: Required The plain text to be hashed.
- CHARSET: Optional The name of the character set. Possible values include UTF-8, UTF-16, ISO-8859-1, and US-ASCII.
URL functions url
- URL: Required The URL from which the protocol needs to be extracted.
- URL: Required The URL from which the host needs to be extracted.
- URL: Required The URL from which the port needs to be extracted.
- URL: Required The URL from which the path needs to be extracted.
- FULL_PATH: Optional A boolean value that determines if the full path is returned. If set to false, only the end of the path is returned.
- URL: Required The URL that you are trying to get the query string from.
- ANCHOR: Required Determines what will be done with the anchor in the query string. Can be one of three values: 鈥渞etain鈥, 鈥渞emove鈥, or 鈥渁ppend鈥.
If the value is 鈥渞etain鈥, the anchor will be attached to the returned value.
If the value is 鈥渞emove鈥, the anchor will be removed from the returned value.
If the value is 鈥渁ppend鈥, the anchor will be returned as a separate value.
get_url_query_str鈥(鈥渇oo://example.com:8042鈥/over/there?name=鈥媐erret#nose鈥, 鈥渞emove鈥)
get_url_query_str鈥(鈥渇oo://example.com鈥:8042/over/there鈥?name=ferret#nose鈥, 鈥渁ppend鈥)
{"name": "ferret#nose"}
{"name": "ferret"}
{"name": "ferret", "_anchor_": "nose"}
- URL: Required The input URL with special characters that you want to replace or encode with ASCII characters.
- URL: Required The input URL with ASCII characters that you want to decode into special characters.
Date and time functions date-and-time
date
function can be found in the dates section of the data format handling guide.2021-10-26T10:10:24Z
- DATE: Required The input date, as a ZonedDateTime object, that you want to format.
- FORMAT: Required The format that you want the date to be changed to.
yyyy-MM-dd HH:mm:ss
鈥)2019-10-23 11:24:35
- TIMESTAMP: Required The timestamp you want to format. This is written in milliseconds.
- FORMAT: Required The format that you want the timestamp to become.
yyyy-MM-dd'T'HH:mm:ss.SSSX
鈥)2019-10-23T11:24:35.000Z
- DATE: Required The string that represents the date.
- FORMAT: Required The string representing the format of the source date.Note: This does not represent the format you want to convert the date string into.
- DEFAULT_DATE: Required The default date returned, if the date provided is null.
2019-10-23T11:24:00Z
- DATE: Required The string that represents the date.
- FORMAT: Required The string representing the format of the source date.Note: This does not represent the format you want to convert the date string into.
2019-10-23T11:24:00Z
- DATE: Required The string that represents the date.
鈥测别补谤鈥
鈥测测测测鈥
鈥测测鈥
鈥渜耻补谤迟别谤鈥
鈥渜辩鈥
鈥渜鈥
鈥渕辞苍迟丑鈥
鈥渕尘鈥
鈥渕鈥
鈥渄补测辞蹿测别补谤鈥
鈥渄测鈥
鈥测鈥
鈥渄补测鈥
鈥渄诲鈥
鈥渄鈥
鈥渨别别办鈥
鈥渨飞鈥
鈥渨鈥
鈥渨别别办诲补测鈥
鈥渄飞鈥
鈥渨鈥
鈥渉辞耻谤鈥
鈥渉丑鈥
鈥渉丑24鈥
鈥渉丑12鈥
鈥渕颈苍耻迟别鈥
鈥渕颈鈥
鈥渘鈥
鈥渟别肠辞苍诲鈥
鈥渟蝉鈥
鈥渟鈥
鈥渕颈濒濒颈蝉别肠辞苍诲鈥
鈥沦厂厂鈥
- COMPONENT: Required A string representing the part of the date.
- DATE: Required The date, in a standard format.
鈥测别补谤鈥
鈥测测测测鈥
鈥测测鈥
鈥渕辞苍迟丑鈥
鈥渕尘鈥
鈥渕鈥
鈥渄补测鈥
鈥渄诲鈥
鈥渄鈥
鈥渉辞耻谤鈥
鈥渉丑鈥
鈥渕颈苍耻迟别鈥
鈥渕颈鈥
鈥渘鈥
鈥渟别肠辞苍诲鈥
鈥渟蝉鈥
鈥渟鈥
- COMPONENT: Required A string representing the part of the date.
- VALUE: Required The value to set for the component for a given date.
- DATE: Required The date, in a standard format.
- YEAR: Required The year, written in four digits.
- MONTH: Required The month. The allowed values are 1 to 12.
- DAY: Required The day. The allowed values are 1 to 31.
- HOUR: Required The hour. The allowed values are 0 to 23.
- MINUTE: Required The minute. The allowed values are 0 to 59.
- NANOSECOND: Required The nanosecond values. The allowed values are 0 to 999999999.
- TIMEZONE: Required The timezone for the date time.
2019-10-17T11:55:12Z
- DATE: Required The date that you are trying to convert.
zone_date_to_utc鈥(2019-10-17T11:55:鈥12 PST
2019-10-17T19:55:12Z
- DATE: Required The date that you are trying to convert.
- ZONE: Required The timezone that you are trying to convert the date to.
zone_date_to_zone(now(), "Europe/Paris")
2021-10-26T15:43:59Z
Hierarchies - Objects objects
- INPUT: Required The object that you鈥檙e trying to check is empty.
is_empty([1, null, 2, 3])
- INPUT: Required A grouping of key and array pairs.
arrays_to_objects('sku', explode("id1|id2", '\\|'), 'price', [22.5,14.35])
[{ "sku": "id1", "price": 22.5 }, { "sku": "id2", "price": 14.35 }]
- INPUT: Required A flat list of key/value pairs.
{"firstName": "John", "lastName": "Doe"}
- STRING: Required The string that is being parsed to create an object.
- VALUE_DELIMITER: Optional The delimiter that separates a field from the value. The default delimiter is
:
. - FIELD_DELIMITER: Optional The delimiter that separates field value pairs. The default delimiter is
,
.
get()
function along with str_to_object()
to retrieve values for the keys in the string.- Example #1: str_to_object(鈥渇irstName - John ; lastName - ; - 123 345 7890鈥, 鈥-鈥, 鈥;鈥)
- Example #2: str_to_object(鈥渇irstName - John ; lastName - ; phone - 123 456 7890鈥, 鈥-鈥, 鈥;鈥).get(鈥渇irstName鈥)
- Example #1:
{"firstName": "John", "lastName": "Doe", "phone": "123 456 7890"}
- Example #2: 鈥淛ohn鈥
is_set()
function.- INPUT: Required The path to be checked if it exists within the source data.
null
. This should be used when you do not want to copy the field to the target schema.null
- OBJECT: Required The object where the keys will be extracted from.
["book1", "book2"]
- STRING: Required The string that you want to parse.
- KEY: Required The key for which the value has to be extracted.
- VALUE_DELIMITER: Required The delimiter that separates the field and the value. If either a
null
or an empty string are provided, this value is:
. - FIELD_DELIMITER: Optional The delimiter that separates field and value pairs. If either a
null
or an empty string are provided, this value is,
.
- MAP: Required The input map data.
- KEY: Required The key can be a single string or a string array. If any other primitive type (data / number) is provided, then it is treated as a string.
- MAP: Required The input map data
- KEY: Required The key can be a single string or a string array. If any other primitive type (data / number) is provided, then it is treated as a string.
- KEY: Required Keys must be a string. If any other primitive values such as integers or dates are provided, then they are auto-converted to strings and are treated as strings.
- ANY_TYPE: Required Refers to any supported XDM data type except Maps.
- OBJECT: Required You can provide an incoming object or object array and point to an attribute inside the object as key.
- OBJECT: Required You can provide an incoming object or object array and point to an attribute inside the object as key.
For information on the object copy feature, see the section below.
Hierarchies - Arrays arrays
- INPUT: Required The array you want to find the first non-null object of.
- INPUT: Required The array you want to find the first element of.
- INPUT: Required The array you want to find the last element of.
- ARRAY: Required The array that you are adding elements to.
- VALUES: The elements that you want to append to the array.
- ARRAY: Required The array that you are adding elements to.
- VALUES: The array(s) you want to append to the parent array.
- INCLUDE_NULLS: Required A boolean value to indicate whether or not to include nulls in the response array.
- VALUES: Required The elements that are to be converted to an array.
[1, 2, 3]
- INPUT: Required The object that you鈥檙e trying to find the size of.
size_of([1, 2, 3, 4])
- ARRAY: Required The array to append the array in the Profile.
upsert_array_append([123, 456])
- ARRAY: Required The array to replace the array in the Profile.
upsert_array_replace([123, 456], 1)
- SEPARATOR: Required The separator used to join the elements in the array.
- ARRAY: Required The array that be joined (after flattening).
array_to_string(";", ["Hello", "world"])
- ARRAY: Required The array to be filtered
- PREDICATE: Required The predicate to be applied on each element of the given array.
filterArray([5, -6, 0, 7], x -> x > 0)
- ARRAY: Required The array to be transformed.
- PREDICATE: Required The predicate to be applied on each element of the given array.
transformArray([5, 6, 7], x -> x + 1)
- ARRAY: Required The array to be flattened.
Hierarchies - Map map
- INPUT: Required The object array you want to find the first non-null object of.
- KEY: Required The key must be a field name in the object array and the object as value.
- INPUT: Required The object array you want to find the first non-null object of.
"{line1 : \"345 park ave\",line2: \"bldg 2\",City : \"san jose\",State : \"CA\",type: \"office\"}"
"{\"firstName\" : \"John\", \"lastName\": \"Doe\"}"
Logical operators logical-operators
- KEY: Required The key to be matched.
- OPTIONS: Required A flattened array of key/value pairs. Optionally, a default value can be put at the end.
If the stateCode given is 鈥減a鈥, 鈥淧ennsylvania鈥.
If the stateCode doesn鈥檛 match the following, 鈥淣/A鈥.
- EXPRESSION: Required The boolean expression that is being evaluated.
- TRUE_VALUE: Required The value that is returned if the expression evaluates to true.
- FALSE_VALUE: Required The value that is returned if the expression evaluates to false.
Aggregation aggregation
- OPTIONS: Required One or more objects that can be compared to each other.
- OPTIONS: Required One or more objects that can be compared to each other.
Type conversions type-conversions
- STRING: Required The string that is to be converted to a BigInteger.
- STRING: Required The string that is to be converted to a Double.
- STRING: Required The string that is to be converted to a Float.
- STRING: Required The string that is to be converted to an Integer.
JSON functions json
- STRING: Required The JSON string to be deserialized.
Special operations special-operations
guid
guid()
guid()
c7016dc7-3163-43f7-afc7-2e1c9c206333
fpid_to_ecid
- STRING: Required The FPID string to be converted into ECID.
fpid_to_ecid(STRING)
fpid_to_ecid("4ed70bee-b654-420a-a3fd-b58b6b65e991")
"28880788470263023831040523038280731744"
User agent functions user-agent
Any of the user agent functions contained in the table below can return either of the following values:
- Phone - A mobile device with a small screen (commonly < 7")
- Mobile - A mobile device that is yet to be identified. This mobile device can be an eReader, a tablet, a phone, a watch, etc.
For more information on device field values, please read the list of device field values in the appendix of this document.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
- USER_AGENT: Required The user agent string.
Analytics functions analytics
- EVENT_STRING: Required The comma-separated Analytics event string.
- EVENT_NAME: Required The event name to extract and ID from.
- EVENT_STRING: Required The comma-separated Analytics event string.
- EVENT_NAME: Required The event name to extract a value from.
- PRODUCTS_STRING: Required The Analytics products string.
- PRODUCTS_STRING: Required The Analytics products string.
- PRODUCTS_STRING: Required The Analytics products string.
- PRODUCTS_STRING: Required The Analytics products string.
- PRODUCTS_STRING: Required The Analytics products string.
- EVENT_NAME: Required The event name to extract values from.
- PRODUCTS_STRING: Required The Analytics products string.
- EVAR_NAME: Required The eVar name to extract.
Object copy object-copy
You can use the object copy feature to automatically copy attributes of an object without making changes to the mapping. For example, if your source data has a structure of:
address{
line1: 4191 Ridgebrook Way,
city: San Jose,
state: California
}
and an XDM structure of:
addr{
addrLine1: 4191 Ridgebrook Way,
city: San Jose,
state: California
}
Then the mapping becomes:
address -> addr
address.line1 -> addr.addrLine1
In the example above, the city
and state
attributes are also ingested automatically at runtime because the address
object is mapped to addr
. If you were to create a line2
attribute in the XDM structure and your input data also contains a line2
in the address
object, then it will also be automatically ingested without any need to manually alter the mapping.
To ensure that the automatic mapping works, the following prerequisites must be met:
- Parent-level objects should be mapped;
- New attributes must have been created in the XDM schema;
- New attributes should have matching names in the source schema and the XDM schema.
If any of the prerequisites are not met, then you must manually map the source schema to the XDM schema using data prep.
Appendix
The following provides additional information on using Data Prep mapping functions
Special characters special-characters
The table below outlines a list of reserved characters and their corresponding encoded characters.
Device field values device-field-values
The table below outlines a list of device field values and their corresponding descriptions.
useragents
that have been altered by an anonymization software.useragents
that contain no information about the device.useragent
string.Code samples code-samples
map_get_values map-get-values
code language-json |
---|
|
map_has_keys map_has_keys
code language-json |
---|
|
add_to_map add_to_map
code language-json |
---|
|
object_to_map object_to_map
Syntax 1
code language-json |
---|
|
Syntax 2
code language-json |
---|
|
Syntax 3
code language-json |
---|
|
array_to_map array_to_map
code language-json |
---|
|