US Patent 9824079 Providing actions for mobile onscreen content

Systems and methods identify actionable content in onscreen content and provide at least a default action for the content. For example, a method may include receiving an image of a screen captured from a display of a mobile device, determining areas of actionable content in the image, and determining a respective action for each area of actionable content. The method may also include generating annotation data for the image, the annotation data including a visual cue that corresponds to a first area of actionable content, the visual cue being actionable to initiate the respective action when selected and providing the annotation data for display on the mobile device.