A context-based inspection system is disclosed. The system may include an optical imaging sub-system. The system may further include one or more controllers communicatively coupled to the optical imaging system. The one or more controllers may be configured to: receive one or more reference images; receive one or more test images of a sample; generate one or more probabilistic context maps during inspection runtime using an unsupervised classifier; provide the generated one or more probabilistic context maps to a supervised classifier during the inspection runtime; and apply the supervised classifier to the received one or more test images to identify one or more DOIs on the sample.