US Patent 8405680 Various methods and apparatuses for achieving augmented reality

Various apparatus and methods are described for a simulation and/or user interface environment. Image data and actual measured depth data of a real object from a real-world scene is acquired via one or more sensors. There is matching of the resolution of the image data and actual measured depth data as well as a linking formed between the image data and actual measured depth data for each pixel representing the real-world object in the scene. The actual measured depth data of the real-world object is processed to extract normal vector data to the surface of the real-world object. occlusion, lighting, and any physics-based interaction effects are based on at least the image data and actual measured depth data.