A method for video processing includes determining, for a conversion between a current video block of a video picture of a video and a bitstream of the video, a number of reference samples in a virtual buffer comprising reference samples derived from the video picture based on a rule, wherein the rule specifies that a maximum number of available reference samples in the virtual buffer is less than a size of the virtual buffer; and performing the conversion based on the determining, wherein the current video block is coded in a prediction mode in which prediction samples are derived from blocks of the available reference samples in the virtual buffer based on a block vector.