In one embodiment, a device identifies a potential change in user experience of an online application. The device selects, based on the potential change in user experience, a set of one or more users of the online application. The device obtains, from the set of one or more users of the online application, feedback regarding their experience with the online application. The device uses the feedback obtained from the set of one or more users of the online application to make a routing decision in a network regarding traffic of the online application.