Uncertainty-Aware Machine Learning for Reliable Decision-Making in Data-Driven Systems
DOI:
https://doi.org/10.61453/jods.v20260207Keywords:
Uncertainty Estimation, Probabilistic Machine Learning, Reliable Al, Decision Support Systems, Predictive UncertaintyAbstract
Machine learning models are increasingly deployed in decision-critical environments such as healthcare, finance, and autonomous systems. However, most conventional models generate deterministic predictions without quantifying uncertainty, which can lead to overconfident mispredictions when data are noisy, incomplete, or outside the training distribution. This limitation exposes a critical gap between predictive accuracy and decision reliability in real-world Al systems. To address this challenge, this study proposes an uncertainty-aware machine learning framework that integrates probabilistic modeling techniques into conventional predictive architectures to jointly estimate epistemic and aleatoric uncertainty. The proposed framework enables models to produce predictive distributions rather than single point predictions, allowing systems to quantify confidence and identify high-risk predictions. Experiments were conducted on multiple benchmark datasets representing both classification and regression tasks under varying levels of noise and data incompleteness. The experimental results demonstrate that the proposed framework achieves predictive performance comparable to deterministic baselines while significantly improving reliability and uncertainty calibration. In classification tasks, the model maintained competitive accuracy and F1-scores while providing well-calibrated confidence estimates, whereas in regression experiments the approach reduced prediction risk by identifying high-error cases through increased uncertainty variance. Robustness tests further show that the framework effectively signals degraded prediction reliability when encountering noisy or incomplete inputs. These findings indicate that incorporating uncertainty estimation enhances trustworthiness and robustness without sacrificing predictive performance. The study highlights uncertainty modeling as a critical component for developing reliable and responsible Al systems capable of supporting risk-sensitive decision-making in real-world data-driven environments.
References
Albuquerque, J., Medeiros, A. M., Alves, A. C., Bourbon, M., & Antunes, M. (2022). Comparative study on the performance of different classification algorithms, combined with pre- and post-processing techniques to handle imbalanced data, in the diagnosis of adult patients with familial hypercholesterolemia. PLOS ONE, 17(6), e0269713. https://doi.org/10.1371/journal.pone.0269713
Chen, X., & Geyer, P. (2022). Machine assistance in energy-efficient building design: A predictive framework toward dynamic interaction with human decision-making under uncertainty. Applied Energy, 307, 118240. https://doi.org/10.1016/j.apenergy.2021.118240
Datta, S. D., Islam, M., Rahman Sobuz, M. H., Ahmed, S., & Kar, M. (2024). Artificial intelligence and machine learning applications in the project lifecycle of the construction industry: A comprehensive review. Heliyon, 10(5), e26888. https://doi.org/10.1016/j.heliyon.2024.e26888
He, W., Jiang, Z., Xiao, T., Xu, Z., & Li, Y. (2025). A Survey on Uncertainty Quantification Methods for Deep Learning. ACM Computing Surveys, 58(7), 179. https://doi.org/10.1145/3786319
Holzenkamp, M., Lyu, D., Kleinekathöfer, U., & Zaspel, P. (2025). Evaluation of uncertainty estimations for Gaussian process regression based machine learning interatomic potentials. Machine Learning: Science and Technology, 6(4), 045019. https://doi.org/10.1088/2632-2153/ae09ef
Jouini, O., Sethom, K., Namoun, A., Aljohani, N., Alanazi, M. H., & Alanazi, M. Ν. (2024). Α Survey of Machine Learning in Edge Computing: Techniques, Frameworks, Applications, Issues, and Research Directions. Technologies 2024, Vol. 12, 12(6). https://doi.org/10.3390/technologies12060081
Kasongo, S. M. (2023). A deep learning technique for intrusion detection system using a Recurrent Neural Networks based framework. Computer Communications, 199, 113-125. https://doi.org/10.1016/j.comcom.2022.12.010
Levi, D., Gispan, L., Giladi, N., & Fetaya, E. (2022). Evaluating and Calibrating Uncertainty Prediction in Regression Tasks. Sensors 2022, Vol. 22, 22(15). https://doi.org/10.3390/s22155540
Lin, Y. H., & Li, G. H. (2022). A Bayesian Deep Learning Framework for RUL Prediction Incorporating Uncertainty Quantification and Calibration. IEEE Transactions on Industrial Informatics, 18(10), 7274-7284. https://doi.org/10.1109/TII.2022.3156965
Linmans, J., Elfwing, S., van der Laak, J., & Litjens, G. (2023). Predictive uncertainty estimation for out-of-distribution detection in digital pathology. Medical Image Analysis, 83(12), 102655. https://doi.org/10.1016/j.media.2022.102655
Mehdiyev, N., Majlatow, M., & Fettke, P. (2024). Quantifying and explaining machine learning uncertainty in predictive process monitoring: an operations research perspective. Annals of Operations Research 2024 347:2, 347(2), 991-1030. https://doi.org/10.1007/s10479-024-05943-4
Nguyen, V. L., Shaker, M. H., & Hüllermeier, E. (2021). How to measure uncertainty in uncertainty sampling for active learning. Machine Learning 2021 111:1, 111(1), 89-122. https://doi.org/10.1007/s10994-021-06003-9
Nicora, G., Rios, M., Abu-Hanna, A., & Bellazzi, R. (2022). Evaluating pointwise reliability of machine learning prediction. Journal of Biomedical Informatics, 127, 103996. https://doi.org/10.1016/j.jbi.2022.103996
Olschewski, S., Luckman, A., Mason, A., Ludvig, E. A., & Konstantinidis, E. (2024). The Future of Decisions From Experience: Connecting Real-World Decision Problems to Cognitive Processes. Perspectives on Psychological Science, 19(1), 82-102. https://doi.org/10.1177/17456916231179138
Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., Peixoto, R. M., Guimarães, G. A. S., Cruz, G. O. R., Araujo, M. M., Santos, L. L., Cruz, M. A. S., Oliveira, E. L. S., Winkler, I., & Nascimento, E. G. S. (2023). Bias and Unfairness in Machine Learning Models: A Systematic Review on Datasets, Tools, Fairness Metrics, and Identification and Mitigation Methods. Big Data and Cognitive Computing 2023, Vol. 7, 7(1). https://doi.org/10.3390/bdcc7010015
Ren, J., Wen, J., Zhao, Z., Yan, R., Chen, X., & Nandi, A. K. (2024). Uncertainty-Aware Deep Learning: A Promising Tool for Trustworthy Fault Diagnosis. IEEE/CAA Journal of Automatica Sinica, 11(6), 1317-1330. https://doi.org/10.1109/JAS.2024.124290
Sarker, I. H. (2022). Machine Learning for Intelligent Data Analysis and Automation in Cybersecurity: Current and Future Prospects. Annals of Data Science 2022 10:6, 10(6), 1473-1498. https://doi.org/10.1007/s40745-022-00444-2
Sarvaiya, J., & Singh, D. (2023). Prediction of performance parameters in friction stir processing using ANN and multiple regression models. Materials Today: Proceedings. https://doi.org/10.1016/j.matpr.2023.04.422
Suk, H., & Kim, S. (2025). Uncertainty-Aware Multimodal Trajectory Prediction via a Single Inference from a Single Model. Sensors 2025, Vol. 25, 25(1). https://doi.org/10.3390/s25010217
Tabarisaadi, P., Khosravi, A., Nahavandi, S., Shafie-Khah, M., & Catalao, J. P. S. (2024). An Optimized Uncertainty-Aware Training Framework for Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 35(5), 6928–6935. https://doi.org/10.1109/TNNLS.2022.3213315
Teng, X., Zhang, J., Zwanenburg, A., Sun, J., Huang, Y., Lam, S., Zhang, Y., Li, B., Zhou, T., Xiao, H., Liu, C., Li, W., Han, X., Ma, Z., Li, T., & Cai, J. (2022). Building reliable radiomic models using image perturbation. Scientific Reports 2022 12:1, 12(1), 10035-. https://doi.org/10.1038/s41598-022-14178-x
Tootchi, A., & Du, X. (2026). Robust Design Under Machine Learning Model Uncertainty. Journal of Mechanical Design, 148(2). https://doi.org/10.1115/1.4068848
Tyralis, H., & Papacharalampous, G. (2024). A review of predictive uncertainty estimation with machine learning. Artificial Intelligence Review 2024 57:4, 57(4), 94-. https://doi.org/10.1007/s10462-023-10698-8
Valtonen, L., Mäkinen, S. J., & Kirjavainen, J. (2024). Advancing Reproducibility and Accountability of Unsupervised Machine Learning in Text Mining: Importance of Transparency in Reporting Preprocessing and Algorithm Selection. Organizational Research Methods, 27(1), 88–113. https://doi.org/10.1177/10944281221124947
Vouros, G. A. (2023). Explainable Deep Reinforcement Learning: State of the Art and Challenges. ACM Computing Surveys, 55(5). https://doi.org/10.1145/3527448
Wu, J., Huang, Z., & Lv, C. (2023). Uncertainty-Aware Model-Based Reinforcement Learning: Methodology and Application in Autonomous Driving. IEEE Transactions on Intelligent Vehicles, 8(1), 194-203. https://doi.org/10.1109/TIV.2022.3185159
Xie, Y., Li, C., Li, M., Liu, F., & Taukenova, M. (2023). An overview of deterministic and probabilistic forecasting methods of wind energy. IScience, 26(1), 105804. https://doi.org/10.1016/j.isci.2022.105804
Zhang, H., Chen, W. (Wayne), Iyer, A., Apley, D. W., & Chen, W. (2022). Uncertainty-aware mixed-variable machine learning for materials design. Scientific Reports 2022 12:1, 12(1), 19760-. https://doi.org/10.1038/s41598-022-23431-2
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Journal of Data Science
This work is licensed under a Creative Commons Attribution 4.0 International License.
