A method of content adaptive decoding involves receiving a bitstream encoded wherein video content is segmented into portions based on predefined classifications or models, such as action scenes, slow scenes, low or high detail scenes, and brightness of the scenes. Based on the segment classifications, each segment or portion is encoded with a different encoder chosen from a plurality of encoders. The chosen encoder is particularly suited to encoding the unique subject matter of the segment. The coded bit-stream for each segment includes information regarding which encoder was used to encode that segment. A matching decoder of a plurality of decoders is chosen using the information in the coded bitstream to decode each segment using a decoder suited for the classification or model of the segment on a generic decoder.