Scalable AutoML Frameworks for End-to-End Optimization of Data Science Workflows

Abstract

The increasing complexity of machine learning workflows has created significant challenges in efficiently developing scalable and high-performing predictive models. Although Automated Machine Learning (AutoML) has emerged as a promising solution for reducing manual intervention in data science pipelines, existing approaches still suffer from limited scalability, high computational overhead, fragmented pipeline optimization, and insufficient resource-aware mechanisms. This study addresses these limitations by proposing a scalable end-to-end AutoML framework that integrates automated preprocessing, feature engineering, model selection, hyperparameter optimization, and adaptive resource allocation within a unified optimization architecture. The proposed methodology employs hierarchical search strategies, Bayesian and evolutionary optimization, and dynamic resource scheduling to improve optimization efficiency while maintaining predictive robustness. Experiments were conducted on medium- and large-scale datasets using repeated cross-validation and benchmark comparisons against conventional machine learning models and existing AutoML systems. The results demonstrate that the proposed framework achieved superior predictive performance, obtaining an accuracy of 0.93 ± 0.01 and an AUC of 0.96 ± 0.01, outperforming baseline approaches by approximately 8-12% across multiple evaluation metrics. In addition, the framework reduced optimization runtime by approximately 68% compared to existing AutoML systems while maintaining strong scalability across increasing dataset sizes. This research aims to advance scalable and efficient AutoML systems by providing a reproducible, resource-aware, and integrated framework for automating end-to-end data science workflows.