Embodiments herein describe using visual passwords to control access to secure information. When a user attempts to access the secure information, she can provide her username to an authentication agent which identifies the visual password corresponding to the received username and selects a first set of images that contains the visual password and a second set of images that does not. The first and second sets of images are then transmitted to a user device. The user device can display the first and second sets of images to the user who selects which images have the visual password. An indication of which images the user selected is then transmitted to the authentication engine which determines whether the user selected all the images in the first set and none of the images in the second set. If so, the user is granted access to the secure information.