US Patent 10146414 Augmented physical and virtual manipulatives

Techniques are disclosed for detecting, tracking, and recording data associated with physical manipulatives in a physical environment, and a corresponding set of virtual manipulatives in a virtual environment. Physical manipulative sensor devices may be configured to detect the position and/or movements of physical manipulatives within a physical environment, including simple or multi-step user interactions with physical manipulatives. The sensor devices and/or associated user computing devices may use the sensor data to determine updated positions, orientations, and configurations for one or more physical manipulatives within the physical environment, as well as defining relationships between sets of the physical manipulatives. Data identifying updated positions, orientations, configurations, and relationship of the physical manipulatives may be transmitted to associated user devices, and a corresponding virtual environment having virtual manipulatives may be output and updated via a graphical display screen to reflect the changes to the physical manipulatives within the physical environment.