Patent attributes
A method of training a system to interact with an application is disclosed. The system includes an interface model configured to translate one or more requests into a sequence of user interface (UI) actions on the application; the interface model comprising: a static model configured to understand UI components of an application; and a dynamic model configured to understand interactions among the UI components and operations that occur when a UI action is completed. The method includes: recording a demonstration of interacting with the application; converting, by the static model, the demonstration into an intermediate representation (IR); and examining the IR, by the dynamic model, to determine an entity, transition, and semantic of an UI that is part of the demonstration.