There is provided a system and method of inspecting a specimen. The method includes obtaining a first image of an inspection area of a die of a semiconductor specimen and a group of reference images corresponding to a group of candidate reference units, obtaining a second image informative of one or more partitions of the inspection area respectively associated with one or more inspection algorithms, for each given pixel of the first image, determining location of one or more reference pixels thereof based on information of the second image, selecting, from the group of candidate reference units, one or more specific reference units actually required for inspecting the inspection area based on the determined location, and using one or more reference images corresponding to the selected reference units to inspect the first image, thereby providing an inspection result of the inspection area.