The present invention relates to an image processing device and method, which realize improvement in encoding efficiency for color difference signals and reduction in address calculations for memory access. In a case where a block size of orthogonal transform is 4×4, and a macroblock of luminance signals is configured of four 4×4 pixel blocks appended with 0 through 1, the four luminance signal blocks are corresponded with one color difference signal 4×4 block appended with C. At this time, there exist four motion vector information of mv0, mv1, mv2, and mv3, as to the four luminance signal blocks. The motion vector information mvc of the one color difference signal 4×4 block is calculated by averaging processing using these four motion vector information. The present invention can be applied to an image encoding device which performed encoding based on the H.264/AVC format, for example.