US Patent 11648959 In-vehicle operation of simulation scenarios during autonomous vehicle runs

This document describes methods by which an autonomous vehicle deploys a simulation scenario while operating in a real-world environment. The vehicle's sensors collect perception data. During a run of the vehicle in a real-world environment, the vehicle's on-board computing system will: (i) receive the perception data; (ii) publish the perception data to a muxing tool of the on-board computing system; (iii) generate simulation data that identifies and labels one or more simulated objects in the environment; (iv) publish the simulation data to the muxing tool; (v) use the muxing tool to add at least a portion of the simulation data to the perception data to yield augmented perception data; and (v) use at least a portion of the augmented perception data to make one or more navigation decisions based for the autonomous vehicle.