V-LPDR: Towards a unified framework for license plate detection, tracking, and recognition in real-world traffic videos

License plate detection and recognition (LPDR) has attracted considerable attention in recent years, and many algorithms have presented the competitive performance on several datasets. However, there are still three significant issues to be addressed in this field. Firstly, most methods have poor detection performance in unconstrained scenarios with moving vehicles and highly distracting background objects. Secondly, existing systems generally focus on single image-based algorithms, yet traffic video sequences provide more effective information than individual frames for LPDR tasks. Thirdly, images and videos captured in complex environments may be adversely affected by distortions and low resolution, causing sensitive recognition performance and reduced robustness. To remedy these issues, we propose to automatically perform license plate detection, tracking, and recognition in real-world traffic videos and integrate them into a unified end-to-end framework via deep learning. The contributions of this paper are threefold: 1) A deep flow-guided spatiotemporal license plate detector is proposed to model the video contextual information by introducing optical flow and a novel spatiotemporal attention mechanism; 2) An online license plate tracker is developed to bridge video-based detection and recognition which utilizes both motion and deep appearance information, and innovatively, it can be end-to-end trained with the detector via multi-task learning; 3) The efficient quality-guided license plate recommender and recognizer are proposed to jointly perform stream recognition. The former recommends high-quality frames from video streams while the latter generates recognition results. We evaluate the proposed method on three traffic video-based license plate datasets, and ablation studies have been presented to verify the effectiveness of each component mentioned above. Moreover, extensive experiments are conducted for comparison with other approaches in different scenarios, and the results have demonstrated that our method achieves state-of-the-art performance on all datasets.