US Patent 10290155 3D virtual environment interaction system

There is provided a 3D virtual environment interaction system comprising: a processing unit for generating a 3D virtual environment comprising a planar reference frame for allowing a user to perform a user interaction with the 3D virtual environment; a display unit for displaying the generated 3D virtual environment; a portable input device for allowing the user to control the position and orientation of the planar reference frame within the generated 3D virtual environment, the portable input device comprising a planar input surface; a position and orientation sensor for monitoring the position and orientation of the planar input surface in a real-world space, the position and orientation sensor allowing the processing unit to modify at least one of the position and the orientation of the planar reference frame in response to a change in a corresponding one of the position and orientation of the planar input surface.