Methods and systems for aligning images of a specimen generated with different modes of an imaging subsystem are provided. One method includes separately aligning first and second images generated with first and second modes, respectively, to a design for the specimen. For a location of interest in the first image, the method includes generating a first difference image for the location of interest and the first mode and generating a second difference image for the location of interest and the second mode. The method also includes aligning the first and second difference images to each other and determining information for the location of interest from results of the aligning.