US Patent 11334147 Visual question and answer based training and runtime methods

In some implementations, a runtime method is performed at a virtual effect system. The method includes: obtaining an image of an environment; obtaining a virtual action associated with one or more virtual effect instructions; generating, by an ML system, at least one estimated material property response to the one or more virtual effect instructions for at least a portion of the image of the environment; generating one or more virtual effect visualizations for at least the portion of the image of the environment based on the at least one estimated material property response to the one or more virtual effect instructions; and causing presentation of the one or more virtual effect visualizations for at least the portion of the image of the environment.