US Patent 10212408 Depth-map augmentation techniques

The techniques described herein segment objects represented in images using one or more depth maps, regardless of whether an initial depth map is of a threshold quality. To do so, the techniques contemporaneously generate, by a depth sensor, a depth map of an environment and capture, by an RGB camera, a color image of the environment. The techniques then determine whether the initial depth map is of the threshold quality. If so, the techniques use this depth map for performing segmentation on an image. If not, then the techniques generate a new depth in part by comparing an image captured by the depth sensor to the color image and performing block-matching between these images. That is, the techniques use the depth sensor and the RGB camera as stereo pair and fill in holes of the initial depth map based on the block-matching.