AI-assisted scheduling of stream processing applications

Potential supervisors: 
Research groups/keywords: 
Description: 

Motivation

Stream processing applications continuously process data to deliver streams of up-to-date results. When run by Stream Processing Engines (SPE) (e.g., Apache Flink), streaming applications are defined as graphs of streams and operators. Under the hood, SPEs instantiate operators by creating several copies of them and assigning them to dedicated threads that are then scheduled by the Operating Systems. As shown in the literature, custom scheduling of streaming applications is beneficial to achieve fine-grained control of performance metrics such as processing throughput and latency.

Challenge

The recent advances made in scheduling frameworks for stream processing applications allow separating the complexity of defining a certain scheduling policy from that of enforcing it. Once the latter challenge is addressed, this thesis focuses on the challenges behinds the lifting of the former. More concretely, the thesis poses the following question: can an AI/ML tool guide the scheduling decisions in order to achieve specific performance goals?
You can conduct this thesis individually or as a team of two students. The content can be adapted accordingly, depending also on the total number of students interested in this thesis. For details and further questions please contact us.

Attachments: 
Date range: 
September, 2021 to September, 2026