US Patent 11604663 Detection of user interface controls via invariance guided sub-control learning

Computerized detection of one or more user interface objects is performed by processing an image file containing one or more user interface objects of a user interface generated by an application program. Sub-control objects can be detected in the image file, where each sub-control object can form a portion of a user interface object that receives user input. Extraneous sub-control objects can be detected. Sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects can be removed. Sub-control objects in the identified set of sub-control objects can be correlated to combine one or more of the sub-control objects in the identified set of sub-control objects to generate control objects that correspond to certain of the user interface objects of the user interface generated by the application program.