US Patent 11290686 Architecture for scalable video conference management

In some implementations, an endpoint device captures video data during a network-based communication session. The endpoint devices processes a stream of user state data indicating attributes of a user of the endpoint device at different times during the network-based communication session. The endpoint device transmits the stream of user state data over a communication network to a server system. The endpoint device receives, over the communication network, (i) content of the network-based communication session and (ii) additional content based on user state data generated by the respective endpoint devices each processing video data that the respective endpoint devices captured during the network-based communication session. The endpoint device presents a user interface providing the received content of the network-based communication session concurrent with the received additional content that is based on the user state data generated by the respective endpoint devices.