Patent attributes
Embodiments herein describe an audio forwarding regularizer and an information bottleneck that are used when training a machine learning (ML) system. The audio forwarding regularizer receives audio training data and identifies visually irrelevant and relevant sounds in the training data. By controlling the information bottleneck, the audio forwarding regularizer forwards data to a generator that is primarily related to the visually irrelevant sounds, while filtering out the visually relevant sounds. The generator also receives data regarding visual objects from a visual encoder derived from visual training data. Thus, when being trained, the generator receives data regarding the visual objects and data regarding the visually irrelevant sounds (but little to no data regarding the visually relevant sounds). Thus, during an execution stage, the generator can generate sounds that are relevant to the visual objects while not adding visually irrelevant sounds to the videos.
