Tour route recommendation begins with multimodal classification

Location estimation of tourist photos by classification is challenging due to the unstable and lessdiscriminative features of photos taken in each city. In this paper, we originally deal with this issue in an alternative way, which begins with but is not limited to classification. We explore textual, temporal, geographic as well as visual information to make tour route recommendation. Given a query photo, we recommend a city by first classifying the photo to get scores that indicate its similarities with photos from each city, and then evaluating the attractiveness of each city by modeling its hotness potential. We do not aim at finding out where the photo was taken exactly; instead, we combine the similarity and hotness potentials according to the user's preference and recommend the city that has the highest combination value. Then, we suggest the next city by further modeling the correlation and interaction between cities pairwise with two potentials, i.e., proximity and covisitedness. Applying the greedy algorithm, we recommend one city at a time and eventually generate a tour route consisting of all the recommended cities in order. Furthermore, in order to show different visual and cultural characteristics across cities, we classify photos of each city into four categories, i.e., food, landscape, man-made and person. In experiments, we collected a database containing 41792 photos of 35 important cities along the silkroad and provided a query sample for tour route recommendation. Experimental results have shown the effectiveness and reliability of our recommendation model.