A method and apparatus for luma-based chroma intra prediction for a current chroma block are disclosed. The chroma intra predictor is derived from reconstructed luma pixels of a current luma block according to the chroma sampling format. Depending on the chroma sampling format, either sub-sampling, down-sampling or no processing is applied to the reconstructed luma pixels in horizontal or vertical direction. The information associated with the chroma sampling format can be incorporated in the sequence parameter set (SPS), the picture parameter set (PPS), the adaptation parameter set (APS) or the slice header of a video bitstream.