US Patent 7944965 Transform domain based distortion cost estimation

Transform based distortion cost estimation, which may be used to calculate a rate distortion cost associated with a mode for encoding a macroblock, is described. In one embodiment of the invention, a distortion value for a particular mode is estimated within the transform domain, which allows for the elimination of both inverse transformation and inverse motion functions in this calculation. A spatial domain residual of an encoding mode is estimated by identifying a difference, within the transform domain, between a motion compensated prediction residual of a macroblock and its corresponding reconstructed signal. The estimated spatial domain residual may then be used in distortion matrix computations to estimate a distortion level, within the transform domain, for an encoding mode.