SFSN: smart frame selection network for multi-task human synthesis on mobile devices

Most human synthesis schemes use high-performance servers, so the user interaction experience of mobile devices is not satisfied. Viewing human synthesis results on smartphones directly increases user interaction and enhances user experience. This paper proposes a smart frame selection network (SFSN) on mobile devices to reduce the traffic between smartphones and cloud. We leverage the attention and relationship model to focus on the relationship between a single frame and the entire video, which can better select important frames, thus reducing the traffic and computing effectively. In addition, we build a multi-task human synthesis system based on SFSN to process the generation tasks such as background changing, pose transfer and virtual try-on in a unified framework. Evaluation results indicate proposed approach reduces the number of frames to be processed by more than 42.2%.