Simulation-Based Explainable AI for Quantum Dynamics: Neural Proxies and SHAP for Entanglement Analysis

The challenge of quantum measurement, wherein observation collapses the wavefunction, hinders non-invasive study of phenomena such as entanglement and superposition. This is especially problematic when analyzing the temporal evolution of multi-qubit systems, where repeated measurement disrupts the very dynamics under investigation. Inspired by challenges in interpreting black-box machine learning models, we propose a simulation-to-explainable AI (XAI) framework for analyzing entanglement dynamics through neural network proxies and interpretability analysis. Using QuTiP-based simulations, we generate temporal correlation data from entangled two-, four-, and six-qubit systems evolving under structured Hamiltonians~\cite{johansson2013}. Classical neural networks trained on this data achieve high accuracy in reproducing entanglement metrics, with mean squared errors of 0.0196 (2 qubits), 0.025 (4 qubits), and 0.0361 (6 qubits). Applying SHAP (SHapley Additive exPlanations)~\cite{lundberg2017}, we **uniquely identify temporal features ($t \approx 2.5$ and $7.5$) that reveal entanglement oscillation phases, enhancing interpretability beyond traditional simulation endpoints.** We propose a framework integrating neural-network proxies with explainable AI techniques to analyze quantum dynamics. Importantly, our approach is strictly \emph{collapse-free within classical simulations}: it does not address the measurement problem in quantum foundations but avoids explicit projective measurement in the numerical pipeline. The method combines QuTiP-based simulations, neural approximations of unitary dynamics, and SHAP-based interpretability applied to temporal entanglement features. **As of September 2025, this work supports NISQ-era applications with freely accessible code and data at Zenodo (DOI: \href{https://doi.org/10.5281/zenodo.17216687}{10.5281/zenodo.17216687}).**