US Patent 11645476 Generating symbolic domain models from multimodal data

A computer generates a formal planning domain description. The computer receives a first text-based description of a domain in an AI environment. The domain includes an action and an associated attribute, and the description is written in natural language. The computer receives the first text-based description of the domain and extracts a first set of domain actions and associated action attributes. The computer receives audio-visual elements depicting the domain, generates a second text-based description, and extracts a second set of domain actions and associated action attributes. The computer constructs finite state machines corresponding to the extracted actions and attributes. The computer converts the FSMs into a symbolic model, written in a formal planning language, that describes the domain.