Interpreting a Multimodal Late-Fusion Model for Municipality-Level Community Bus and Demand-Responsive Bus Classification in Kyushu Area Using SHAP and Grad CAM

Raden Muhammad Azhar Ardlin Syah; Syuji  Yoshiki; Hera  Widyastuti

doi:10.37899/journallamultiapp.v7i4.5926

Raden Muhammad Azhar Ardlin Syah Master’s Student of Civil Engineering Department, Faculty of Civil, Planning, and Geo Engineering, at Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia
Syuji Yoshiki Associate Professor of Center for Water Cycle, Marine Environment and Disaster Management at Kumamoto University, Kumamoto, Japan
Hera Widyastuti Professor of Transportation and Pavement Material Laboratory at Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia

DOI: https://doi.org/10.37899/journallamultiapp.v7i4.5926

Keywords: Multimodal learning, SHAP, Grad-CAM, Community Bus, Demand-Responsive Bus

Abstract

Limited public transport accessibility remains a serious issue in aging and low-density areas such as Japan. Local governments have introduced Community Bus (CB) and Demand-Responsive Bus (DRB) services, but methods for explaining the regional patterns associated with these two service types remain limited. This paper examines the interpretability of a multimodal late-fusion model that integrates tabular regional indicators and Geographic Information System (GIS)-based spatial images for municipality-level CB and DRB classification. The research used municipalities in Kyushu, Japan. A multimodal late-fusion architecture was adopted as a commonly used integration strategy for image and tabular data. Model interpretation was conducted using SHAP for the tabular branch and Grad-CAM for the image branch. Performance was summarized using Macro F1-score as the main metric. The late-fusion model achieved a mean Macro F1-score of 0.5978 and a mean accuracy of 0.6797 under the non-TTA setting. TTA was evaluated as an inference-time robustness strategy and showed slightly smaller fold-wise variability, although the non-TTA setting produced higher mean predictive scores. Therefore, the subsequent interpretation analysis was based on the non-TTA results. SHAP showed that single elder household density, lowest elevation, elder rate, highest elevation, and total population were among the most influential features. Class-specific SHAP results also showed differences between correctly classified CB and DRB municipalities. Grad-CAM indicated that the model attended to meaningful internal spatial regions, although its interpretation was more qualitative than SHAP. These results show that multimodal late fusion can provide interpretable evidence for municipality-level CB and DRB classification and support more transparent regional transport analysisLimited public transport accessibility remains a serious issue in aging and low-density areas such as Japan. Local governments have introduced Community Bus (CB) and Demand-Responsive Bus (DRB) services, but methods for explaining the regional patterns associated with these two service types remain limited. This paper examines the interpretability of a multimodal late-fusion model that integrates tabular regional indicators and Geographic Information System (GIS)-based spatial images for municipality-level CB and DRB classification. The research used municipalities in Kyushu, Japan. A multimodal late-fusion architecture was adopted as a commonly used integration strategy for image and tabular data. Model interpretation was conducted using SHAP for the tabular branch and Grad-CAM for the image branch. Performance was summarized using Macro F1-score as the main metric. The late-fusion model achieved a mean Macro F1-score of 0.5978 and a mean accuracy of 0.6797 under the non-TTA setting. TTA was evaluated as an inference-time robustness strategy and showed slightly smaller fold-wise variability, although the non-TTA setting produced higher mean predictive scores. Therefore, the subsequent interpretation analysis was based on the non-TTA results. SHAP showed that single elder household density, lowest elevation, elder rate, highest elevation, and total population were among the most influential features. Class-specific SHAP results also showed differences between correctly classified CB and DRB municipalities. Grad-CAM indicated that the model attended to meaningful internal spatial regions, although its interpretation was more qualitative than SHAP. These results show that multimodal late fusion can provide interpretable evidence for municipality-level CB and DRB classification and support more transparent regional transport analysis.

References

Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010). Multimodal fusion for multimedia analysis: A survey. Multimedia Systems, 16(6), 345–379. https://doi.org/10.1007/s00530-010-0182-0

Baltrušaitis, T., Ahuja, C., & Morency, L.-P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. https://doi.org/10.1109/tpami.2018.2798607

Blomqvist, E. (2025). Fracture Risk Prediction Using Multimodal Neural Networks.

Coutinho, F. M., Van Oort, N., Christoforou, Z., Alonso-González, M. J., Cats, O., & Hoogendoorn, S. (2020). Impacts of replacing a fixed public transport line by a demand responsive transport system: Case study of a rural area in Amsterdam. Research in Transportation Economics, 83, 100910. https://doi.org/10.1016/j.retrec.2020.100910

Dahmen, V., Weikl, S., & Bogenberger, K. (2024). Interpretable Machine Learning for Mode Choice Modeling on Tracking-Based Revealed Preference Data. Transportation Research Record: Journal of the Transportation Research Board, 2678(11), 2075–2091. https://doi.org/10.1177/03611981241246973

Deressu, T. F., Bojer, A. K., Debelee, T. G., Negera, W. G., Nadarajah, S., & Gebissa, K. W. (2025). Enhancing land use and land cover classification with deep learning-based satellite imagery segmentation. International Journal of Applied Earth Observation and Geoinformation, 144, 104839. https://doi.org/10.1016/j.jag.2025.104839

Ding, J., Xue, N., Long, Y., Xia, G.-S., & Lu, Q. (2018). Learning RoI transformer for detecting oriented objects in aerial images. arXiv Preprint arXiv:1812.00155. https://doi.org/10.48550/arXiv.1812.00155

Du, S., Zheng, S., Wang, Y., Bai, W., O’Regan, D. P., & Qin, C. (2024). TIP: Tabular-image pre-training for multimodal classification with incomplete data. European Conference on Computer Vision, 478–496. https://doi.org/10.48550/arXiv.2407.07582

Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1). MIT press Cambridge.

Han, J., Ding, J., Xue, N., & Xia, G.-S. (2021). Redet: A rotation-equivariant detector for aerial object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2786–2795. https://doi.org/10.48550/arXiv.2103.07733

Hasegawa, D., & Suzuki, T. (2011). Urban size and density condition for the introduction of demand responsive transport. Journal of the City Planning Institute of Japan, 46(3), 817–822. https://doi.org/10.11361/journalcpij.46.817

Hasegawa D., & Suzuki T. (2019). Evaluation of Effectiveness by Combining of Bus and Demand Responsive Transit Service Focusing Bus Route Network Configuration. Theory and Applications of GIS, 27(1), 1–11. https://doi.org/10.5638/thagis.27.1

Hasegawa, M., Nakamura, T., Aou, Y., Suzumura, M., & Morikawa, T. (2021). Comparison Analysis Of User Service Need For Implementing Share Taxi. Journal of Japan Society of Civil Engineers, Ser. D3 (Infrastructure Planning and Management), 76(5), I_1417-I_1427. https://doi.org/10.2208/jscejipm.76.5_I_1417

He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition (arXiv:1512.03385). arXiv. https://doi.org/10.48550/arXiv.1512.03385

He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2019). Bag of Tricks for Image Classification with Convolutional Neural Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 558–567. https://doi.org/10.1109/CVPR.2019.00065

Higashimoto, Y., Takada, H., & Kishi, K. (2013). Evaluation of bus route efficiency by network DEA including social priority.

Ise, N., & Hino, Y. (2023). A Study on Community Bus Origin-Destination Demand focused on Characteristics of Bus Stop Catchment Area: A Case Study in Kishiwada City, Osaka Prefecture. JSTE Journal of Traffic Engineering, 9(2), A_335-A_341. https://doi.org/10.14954/jste.9.2_A_335

Joze, H. R. V., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020). MMTM: Multimodal transfer module for CNN fusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13289–13299. https://doi.org/10.48550/arXiv.1911.08670

Kato, H. (2025). Acceptance and usage of demand‑responsive transport among people in poor health: Evidence from Senboku New Town. Transportation Research Interdisciplinary Perspectives, 34, 101632. https://doi.org/https://doi.org/10.1016/j.trip.2025.101632

Kim, E.-J., Oh, D., & Kim, H. (2025). How demand-responsive transit enhances public transit equity: A spatial causal inference approach. SSRN. https://doi.org/10.2139/ssrn.5119770

Liyanage, S., Dia, H., Duncan, G., & Abduljabbar, R. (2024). Evaluation of the Impacts of On-Demand Bus Services Using Traffic Simulation. Sustainability, 16(19), 8477. https://doi.org/10.3390/su16198477

Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf

Nakanishi, T., & Matsumoto, Y. (2024). Interregional comparison of various effects by DRT focusing on individual attributes and regional characteristics. Proceedings of the City Planning Institute of Japan, Chubu Branch, 35, 27–32. https://doi.org/10.11361/cpijchubu.35.0_27

Nakanishi, T., & Matsumoto, Y. (2025). Understanding Transfer Intentions and Areas Suitable for Introducing Transfer-based DRT. Journal of the Eastern Asia Society for Transportation Studies, 16, PP3937. https://doi.org/10.11175/easts.16.PP3937

Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv Preprint arXiv:1712.04621. https://doi.org/10.48550/arXiv.1712.04621

Powers, D. (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation. Journal of Machine Learning Technologies, 2(1), 37–63.

Quiroga, F., Ronchetti, F., Lanzarini, L., & Bariviera, A. F. (2020). Revisiting Data Augmentation for Rotational Invariance in Convolutional Neural Networks. In J. C. Ferrer-Comalat, S. Linares-Mustarós, J. M. Merigó, & J. Kacprzyk (Eds.), Modelling and Simulation in Management Sciences (pp. 127–141). Springer International Publishing. https://doi.org/10.1007/978-3-030-15413-4_10

Rich, J., Seshadri, R., Jomeh, A. J., & Clausen, S. R. (2023). Fixed routing or demand-responsive? Agent-based modelling of autonomous first and last mile services in light-rail systems. Transportation Research Part A: Policy and Practice, 173, 103676. https://doi.org/10.1016/j.tra.2023.103676

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020). Grad-CAM: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128(2), 336–359. https://doi.org/10.1007/s11263-019-01228-7

Sibrel, S. C., Rathore, R., Lessard, L., & Schloss, K. B. (2020). The relation between color and spatial structure for interpreting colormap data visualizations. Journal of Vision, 20(12), 7–7.

Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427–437. https://doi.org/10.1016/j.ipm.2009.03.002

Takeuchi, R., Iwamoto, T., Takaku, M., Nakamura, T., & Ozaki, M. (2019). Research on the introduction situation of various local public transportation（Report No.153）. Policy Research Institute for Land, Infrastructure, Transport and Tourism. https://www.mlit.go.jp/pri/english/houkoku/gaiyou/english_kkk153.html?

Takeuchi, R., & Yoshida, I. (2022). A study on the difference of the introduction situation and planning process on community bus and demand responsive transport in local governments. Journal of Japan Society of Civil Engineers, Ser. D3 (Infrastructure Planning and Management), 78(6), II_613-II_622. https://doi.org/10.2208/jscejipm.78.6_II_613

Takeuchi, R., Yoshida, I., & Inoi, H. (2023). Perspective And Business Contiunity Influence Factors For Transportation Passenger Service With Private Car By Actual Situation Of Operation Body (Paratransit). Japanese Journal of JSCE, 79(20), 23–20026. https://doi.org/10.2208/jscejj.23-20026

Takeuchi, R., Yoshida, I., & Ozaki, M. (2023). Research On The Factors Influencing The Effectiveness Of The Introduction Of Demand Transport And Community Buses By Local Authorities. Journal of Japan Society of Civil Engineers, Ser. D3 (Infrastructure Planning and Management), 78(5), I_627-I_637. https://doi.org/10.2208/jscejipm.78.5_I_627

Tang, L., Tang, C., Fu, Q., & Ma, C. (2024). Predicting Travel Mode Choice With a Robust Neural Network and Shapley Additive Explanations Analysis. IET Intelligent Transport Systems, 18(7), 1339–1354. https://doi.org/10.1049/itr2.12514

Touvron, H., Vedaldi, A., Douze, M., & Jégou, H. (2022). Fixing the train-test resolution discrepancy (arXiv:1906.06423). arXiv. https://doi.org/10.48550/arXiv.1906.06423

Van Rijsbergen, C. (1979). Information retrieval: Theory and practice. Proceedings of the Joint IBM/University of Newcastle upon Tyne Seminar on Data Base Systems, 79, 1–14.

Vansteenwegen, P., Melis, L., Aktaş, D., Montenegro, B. D. G., Sartori Vieira, F., & Sörensen, K. (2022). A survey on demand-responsive public bus systems. Transportation Research Part C: Emerging Technologies, 137, 103573. https://doi.org/10.1016/j.trc.2022.103573

Velmurugan, P. (2023). Species Distribution Modelling: A Multimodal Learning Approach.

Wang, H., Li, J., Wang, P., Teng, J., & Loo, B. P. Y. (2023). Adaptability analysis methods of demand responsive transit: A review and future directions. Transport Reviews, 43(4), 676–697. https://doi.org/10.1080/01441647.2023.2165574

Wu, M., Sun, Y., & Xia, X. (2024). Computational assessment of the aesthetic quality of map colors. Transactions in GIS, 28(6), 1556–1576. https://doi.org/https://doi.org/10.1111/tgis.13206

Xing, J., & Sieber, R. (2023). The Challenges of Integrating Explainable Artificial Intelligence into GeoAI. Transactions in GIS, 27(3), 626–645. https://doi.org/10.1111/tgis.13045