Extracting and Learning Visual Features to Improve Keyword based Web Search

Abstract

Retrieving and ranking the search result for a search engine is a machine learning problem
worth multibillion-dollar. Multimodal learning to rank model is implemented that uses the rich image features
along with textual features to rank the webpages. The visual features from images are extracted using pretrained
CNN model, VGG. These visual features extracted using transfer learning along with the textual
feature from LETOR are integrated to train a combined LTR model. The combined LTR model presented
provided a better loss and accuracy value. The mean average precision value of the ranking of webpages by
multimodal LTR shows an average improvement of 5-10%.