A novel macroblock layer rate control for H.264/AVC

A novel macroblock layer rate control algorithm for H.264/ AVC is proposed in this paper. To solve the issues of rate control model in H.264/AVC, we presented a new coding complexity measure based on integer transform coefficients, so-called mean absolute transform quantized distortion (MATQD). Then, we proposed four effective models respectively to improve our rate control performance, including MATQD prediction model, target bits allocation model, header bits prediction model for a marcoblock and the quadratic R-D model based on MATQD. Experimental results have shown that our method has more precise rate control performance than that of the standard rate control algorithm JVT-G01 2 in JM8.5 with the average absolute rate error reducing 0.396bps. Furthermore, the proposed method can produce much smoother bit fluctuation with the average rate deviation reducing 21.74% and obtain much better average image quality with average PSNR improving 0.336dB (the best gain is up to 0.77dB).