Incremental Techniques for Large-Scale Dynamic Query Processing

Many applications from various disciplines are now required to
analyze fast evolving big data in real time. Various approaches
for incremental processing of queries have been proposed over the
years. Traditional approaches rely on updating the results of a query
when updates are streamed rather than re-computing these queries,
and therefore, higher execution performance is expected. However,
they do not perform well for large databases that are updated at
high frequencies. Therefore, new algorithms and approaches have
been proposed in the literature to address these challenges by, for
instance, reducing the complexity of processing updates. Moreover,
many of these algorithms are now leveraging distributed streaming
platforms such as Spark Streaming and Flink. In this tutorial, we
briefly discuss legacy approaches for incremental query processing,
and then give an overview of the new challenges introduced due to
processing big data streams. We then discuss in detail the recently
proposed algorithms that address some of these challenges. We
emphasize the characteristics and algorithmic analysis of various
proposed approaches and conclude by discussing future research
directions.