Exquisitor: Interactive Learning for Multimedia

Multimedia collections contain a wealth of information that can be used to gain insight into trends, performing investigations, finding media to represent concepts, and much more. Over the past decade, multimedia collections have seen tremendous growth, with technological advances allowing ever faster generation and sharing of multimedia data. Multimedia analytics is a research field that focuses on providing insight into large-scale multimedia collections. In this field, it has been stated that analysing such collections requires an interactive approach that combines the strengths of a human and a machine. The machine's objective is to present items relevant to the human's information needs, while the human may indicate their relevance, to improve the machine's future suggestions. To facilitate this process, an interactive learning approach capable of handling the scale of today's collections is required. While a preexisting approach can be scalable, it demands significant computational resources. In this thesis, a new interactive learning approach is proposed, called Exquisitor, which integrates high-dimensional indexing, incremental retrieval, and query optimisation policies into the interactive learning process, making it responsive, accurate, flexible, and scalable. Furthermore, the work emphasizes the need for better automated evaluation protocols, as existing protocols fail to capture different types of user interactions, making it reasonable to suspect whether or not interactive learning is suited for obtaining insight. New automated evaluation protocols are introduced in this work that analyse various user interaction strategies, to better evaluate the capabilities of interactive learning. While not eliminating the need for user testing, it allows for more detailed performance analysis of interactive learning approaches earlier in the developmental phase. Through extensive experiments, it shows that Exquisitor improves or maintains result quality, while drastically reducing response time and requirements for computational resources. In addition to the automated evaluation protocols, the approach has also been used in practice, through participation in live interactive search challenges. The research and development of Exquisitor has shown that interactive learning is efficient for gaining insight into large multimedia collections, establishing it as the new state of the art in large-scale interactive learning. By reducing requirements for computational resources, it opens up possibilities for future research on utilising these resources to introduce additional elements into the analytical process, such as concurrent classifiers, diversification of results, or dynamic combination of modalities.