US Patent 11783815 Multimodality in digital assistant systems

Systems and processes for operating an intelligent automated assistant are provided. An example process for determining user intent includes receiving a natural language input and detecting an event. The process further includes, determining, at a first time, based on the natural language input, a first value for a first node of a parsing structure; and determining, at a second time, based on the detected data event, a second value for a second node of the parsing structure. The process further includes in accordance with a determination that the first time and the second time are within the predetermined time: determining, using the parsing structure, the first value, and the second value, a user intent associated with the natural language input; initiating a task based on the determined intent; and providing an output indicative of the task.