US Patent 10372572 Prediction model testing framework

A prediction model testing system includes a test environment that is used to test a prediction model under test (PMUT). A metrics collector in a production environment collects and stores production metrics data generated from computing resources in a production environment. A production predictor in the production environment generates production predictions for the metrics, using a production prediction model. A test manager may make the production metrics data available in a test environment. Test predictions are generated in the test environment from the metrics data using the PMUT. The test manager may then calculate respective prediction errors of the production prediction model and the PMUT, and generate a report indicating the differences between the two sets of prediction errors. The report may be used by the test management system to determine whether a test of the PMUT was successful.