An imager may include depth sensing pixels that receive and convert incident light into image signals. The imager may have an associated imaging lens that focuses the incident light onto the imager. Each of the depth sensing pixels may include a microlens that focuses incident light received from the imaging lens through a color filter onto first and second photosensitive regions of a substrate. The first and second photosensitive regions may provide different and asymmetrical angular responses to incident light. Depth information for each depth sensing pixel may be determined based on the difference between output signals of the first and second photosensitive regions of that depth sensing pixel. Color information for each depth sensing pixel may be determined from a summation of output signals of the first and second photosensitive regions.