Causal AI for Data Scientists: A Framework to Automate Discovery of Causal Relationships in Noisy Datasets

In the pursuit of advancing causal discovery methodologies, this research introduces a robust framework designed to navigate environments laden with noise and latent confounding. By integrating mechanistic interpretability with causal inference, the framework facilitates the structural disentanglement of complex causal influences, enabling a more transparent understanding of underlying mechanisms.The approach is validated using both synthetic datasets (Noisy-CausalSim, Sim-NCE+) and real-world datasets (MIMIC-IV clinical data and NOAA climate records), demonstrating versatility across diverse contexts. Key metrics—Causal Effect Strength (CES), Intervention Specificity (IS), and Polysemanticity Score (PS)—are employed to quantify the magnitude, focus, and dispersion of causal effects. Findings reveal that variables with high CES and low IS values act as monosemantic agents, exerting targeted influence, while higher PS scores indicate polysemantic variables affecting multiple targets.The framework's encoder-decoder architecture aligns latent representations with causal graphs, enhancing interpretability. Mediation analysis uncovers indirect causal pathways, highlighting the role of latent variables in effect transmission. Compared to traditional methods like NOTEARS and DAG-GNN, the proposed approach achieves superior performance, evidenced by higher CES and IS scores and lower Structural Hamming Distance (SHD).This work advances causal inference by providing tools for uncovering modular and interpretable causal structures, even amidst noise and latent confounding. The combination of mechanistic interpretability methods like activation patching and attention saliency allows for a more profound comprehension of the mechanisms of machine learning models. By closing the black-box prediction and causal processes underlying, this method not only increases the transparency but also the trustworthiness of causal AI systems. In addition, its applicability to a wide range of datasets demonstrates its flexibility and potential generalizability to a variety of fields.