Rate-distortion-complexity optimization for x265

Rate-distortion optimization (RDO) is conventionally based on the analysis of rate-distortion (R-D) curve to minimize the coding distortion under the coding bits constraint. However, it is necessary to consider the computational complexity in the RDO process. In this paper, we obtain the Confidence LEvel - Computational complexity (CLEC) curves which indicate the characteristics of coding tree units (CTUs). Based on the CLEC curves, a rate-distortion-complexity optimization (RDCO) algorithm is proposed to optimize R-D under given computational complexity and achieve the optimal coding performance for x265. Experimental results demonstrate that the proposed algorithm can achieve a wide range of encoding speed under a given quantization parameter (QP) whereas the original x265 can only achieve a few fixed encoding speeds, and the proposed algorithm can reduce the BD-rate and increase the BD-PSNR by 6.59% and 0.13 dB on average under the same requirements of encoding speeds as the original x265.