Restrictions on number of context coded bins are described. A method for video processing includes applying, for a conversion between a video and a bitstream representation of the video, control on maximum number of context coded bins per predetermined unit which is associated with the video; and performing the conversion based on the control.