Resource-Aware Data Science

When considering large-scale hardware deployments from public cloud vendors and High Performance Computing (HPC) centers, data science applications powered by machine learning are not the only data-intensive applications run on these hardware resources. There are also large-scale big data analytics systems. Such big data analytics applications are fundamentally different from machine learning. Big data analytics helps us transform the sheer amount of complex data into discoveries, while machine learning enables forecasts based on learning from big data. An end-to-end data-driven pipeline in real-world use cases is typically composed of a combination of data-intensive systems that target different data-intensive application domains. This project extends the Resource-Aware Data Science (RAD) project by considering a combination of traditional data management, server-grade machine learning, and resource-constrained data science applications.