Interpret Auto-Allocate reports
Interpret the results of an Auto-Allocate A/B activity in 蜜豆视频 Target by examining important indicators, including lift and confidence.
Many marketers make the mistake of prematurely declaring a winning experience before the results indicate the clear winner. Target makes it easier for you to determine the winner.
For general information about declaring a winner, see Ten common A/B testing pitfalls and how to avoid them.
Identify the winning experience section_24007470CF5B4D30A06610CE8DD23CE3
When using the Auto-Allocate feature, Target displays a badge at the top of the activity鈥檚 page indicating 鈥淣o Winner Yet鈥 until the activity reaches the minimum number of conversions with sufficient confidence.
When a clear winner is declared, Target displays 鈥淲inner: Experience X.鈥
Statistical guarantees of Auto-Allocate section_7AF3B93E90BA4B80BC9FC4783B6A389C
At the end of an A/B activity, Auto-Allocate guarantees that the determined winner has an effective false-positive rate of 5%. This means that only 5% of the time, the determined winner is not actually the best experience among all the experiences in the activity. For an A/A test (with identical experiences), Target concludes a test less than 5% of the time. The expected behavior for an A/A test (with identical experiences) is for it to run indefinitely and so the winner badge should never appear.
Target does not use p-value based confidence for Auto-Allocate.
The Confidence column in an Auto-Allocate activity (illustrated below) displays the probability of an experience being the winner within 1% margin of error. The algorithm uses a minimum detectable effect of 1% between the best and the second-best conversion rates. The algorithm uses to compute this probability.
Normal A/B tests compute confidence based on p-values. Auto-Allocate does not use p-values. P-values 鈥渓oosely鈥 compute the probability that a given experience is different from the control. These p-values can be used only to determine whether an experience might be different from the control. These values cannot be used to determine if an experience is different from another experience (not control).
Understand Lift and Confidence reporting in Auto-Allocate activities lift-confidence
In Auto-Allocate activities, the first experience (by default named Experience A) is always defined as a 鈥淐ontrol鈥 experience on the Reports tab. This experience is not treated as a true statistical control in the modeling used to determine the performance of experiences, but it is treated as a reference or baseline for some figures in the report.
The 鈥淟ift鈥 numeric value and 95% bounds for each experience are always calculated with reference to the defined 鈥淐ontrol鈥 experience. The defined 鈥淐ontrol鈥 experience cannot have lift relative to itself, so a blank 鈥溾斺 value is reported for this experience. Unlike in A/B tests, in Auto-Allocate tests, if an experience performs worse than the defined control, a negative Lift value is not reported; instead 鈥溾斺 is displayed.
The displayed Confidence Interval bars represent the 95% confidence interval around the mean estimate of an experience鈥檚 conversion rate. These bars are also color-coded with respect to the defined 鈥淐ontrol鈥 experience. The 鈥淐ontrol鈥 experience鈥檚 bar is always colored gray. The portions of confidence intervals below the 鈥淐ontrol鈥 experience鈥檚 confidence interval are colored red and the portions of confidence intervals above the 鈥淐ontrol鈥 experience are colored green.
A winner is found when the leading experience鈥檚 95% Confidence Interval is not overlapping with any other experiences. The winning experience is designated with a green star badge to the left of the experience name and in the 鈥淲inner鈥 banner. When no star is visible, the banner reads 鈥淣o Winner Yet鈥 and a winner has not yet been found.
A 鈥淐onfidence鈥 number is also reported next to the currently leading or winning experience. This figure is reported only until the leading experience鈥檚 Confidence reaches at least 60%. If two experiences are present in the Auto-Allocate activity, this number represents the confidence level that the experience is performing better than the other experience. If more than two experiences are present in the Auto-Allocate activity, this number represents the confidence level that the experience is performing better than the defined 鈥淐ontrol鈥 experience. If the 鈥淐ontrol鈥 experience is winning, no 鈥淐onfidence鈥 figure is reported.
Frequently Asked Questions section_C8E068512A93458D8C006760B1C0B6A2
Consider the following answers to frequently asked questions:
It has been a few days into the activity. Why are all confidence values still showing 0%?
Any of the following reasons describe why 0% displays in the report鈥檚 Confidence column for all activities:
-
Manual A/B tests and Auto-Allocate use different statistics to display Confidence values.
Manual A/B tests use p-values based on . A P-value is the probability of finding the observed (or a more extreme) difference between an experience and the control, given that in reality there is no such difference. These P-values can be used only to determine whether observed data is consistent with a given experience and the control being the same. These values cannot be used to determine if an experience is different from another experience (not control).
Auto-Allocate shows the probability of a given experience being a true winner across all experiences in the activity. Only a winning experience (which is most likely to be the winner), has a non-zero confidence value. All others are most likely to be losers and display 0%.
-
Auto-Allocate starts showing confidence only after the winning experience gathers 60% confidence. These confidence levels typically appear in about half the time that a normal A/B test would take to complete (although this time frame is not guaranteed). To determine how long a normal A/B test would run, use the 蜜豆视频 Target Sample Size Calculator: plug control鈥檚 conversion-rate in 鈥淏aseline conversion rate,鈥 鈥5%鈥 for 鈥淟ift,鈥 and 95% for 鈥淐onfidence.鈥 Typically, confidence starts showing after each experience has amassed at least 50% of the required samples per-experience. This gives you an idea of when confidence starts appearing.
-
If the report is showing 0% across the board, it is likely too early into the activity.
Are the 鈥淣o Winner,鈥 鈥淲inner,鈥 and 鈥渟tar鈥 badges available for Auto-Allocate activities that use Analytics as the reporting source (A4T)?
The 鈥淣o Winner Yet鈥 and 鈥淲inner鈥 badges are currently not available in the A4T panel in Analysis Workspace. These badges are also not available if the same report is viewed in Target. A winner 鈥渟tar鈥 badge shown in a Target report for an Auto-Allocate activity using A4T should be ignored.
For more information about this and other limitations and notes, see Auto-Allocate in A4T support for Auto-Allocate and Auto-Target activities.