US Patent 11514948 Model-based dubbing to translate spoken audio in a video

Model-based dubbing techniques are implemented to generate a translated version of a source video. Spoken audio portions of a source video may be extracted and semantic graphs generated that represent the spoken audio portions. The semantic graphs may be used to produce translations of the spoken portions. A machine learning model may be implemented to generate replacement audio for the spoken portions using the translation of the spoken portion. A machine learning model may be implemented to generate modifications to facial image data for a speaker of the replacement audio.