IntelliStream: Leveraging Machine Learning for Enhanced Throughput, Performance Of Brokers By Log Analysis

Optimizing the performance of real-time data streaming platforms is crucial in managing diverse and demanding workloads. This paper presents an innovative approach to auto-tune such platforms for enhanced throughput and overall performance using machine learning techniques. By analyzing garbage collection (GC) logs and broker logs, by employing regression models to identify performance bottlenecks and predict optimal configuration settings. The methodology involves extracting key metrics from the logs, training regression models to understand the relationship between these metrics and system performance, and then applying the models to fine-tune system parameters automatically. The proposed solution demonstrates potential to develop significant improvements in throughput and stability, offering a robust framework for dynamic performance optimization in complex data environ-ments. This approach not only enhances efficiency but also reduces the need for manual tuning, paving the way for more intelligent and autonomous data streaming infrastructures.