Patent attributes
Techniques for performing tests by a speech processing system are described. A speech processing system may apply a hash function to a group of user identifiers, a content provider identifier, and time data to segment the group of user identifiers into non-holdout, holdout control, and holdout treatment groups. A non-holdout group represents users not subject to a test for a duration of time. A holdout control group represents users subject to a test for the duration of time, but which do not receive unresponsive content during the duration of time. A holdout treatment group represents users subject to a test and which receive unresponsive content over the duration of time. User activity of the holdout control and holdout treatment groups may be compared to measure an effectiveness of the unresponsive content. The measured effectiveness may be used to determine a ranking of the content provider at runtime.
