Hyperparameter Optimization Strategies for Tree-Based Machine Learning Models Prediction: A Comparative Study of AdaBoost, Decision Trees, and Random Forest

This study evaluates the performance of Decision Tree, AdaBoost, and Random Forest models on the case study dataset to address challenges in predictive modeling. The research highlights the limitations of single decision trees, including high variance and overfitting, and explores ensemble methods as solutions. The objective is to compare these algorithms through accuracy, precision, recall, and F1-score metrics. A rigorous methodology involving data preprocessing, hyperparameter tuning, and 5-fold cross-validation was employed to ensure robust evaluation. Results indicate that Random Forest outperformed with an accuracy of 80% and an F1-score of 71.9%, leveraging feature interdependencies effectively. AdaBoost achieved 75% accuracy, balanced precision-recall but was sensitive to noise, while Decision Trees provided 76% accuracy, interpretability but struggled with generalization. Future research should explore advanced ensemble techniques like XGBoost and feature engineering strategies to enhance model performance. These findings underscore the importance of selecting appropriate algorithms tailored to dataset characteristics for real-world applications.