Counterfactual Latent Representations: A Neurosymbolic Approach

Leonhard Kestel1,2,3 Christoph Kern1,2
1Ludwig-Maximilians-Universität München 2Munich Center for Machine Learning 3Bavarian Research Institute for Digital Transformation

Motivation

Question: Counterfactual Fairness

  • ,,How would a person’s CV look like if its sensitive attribute(s) were different?’’
  • ,,Would this person be treated differently?’’
  • Requires causal reasoning to generate this person’s data in a counterfactual world
  • Non-trivial, particularly with unstructured high-dimensional data, such as text

Contribution

Methodology to generate a counterfactual latent representation of a text input

  • Modular: Pre-trained VAE to map text to a latent representation + neurosymbolic manipulator model to transform these representations
  • Causal-aware: Manipulator learns a latent transformation analogously to a causal intervention (abduction-action-prediction) on observable data w.r.t. a sensitive attribute.

Problem Formulation

Known:

  • \(X\) - text input (e.g., a CV)
  • \(A\) - sensitive attribute (e.g., gender)
  • \(Z\) - latent representation produced by a pre-trained encoder \(Z = f(X)\)

Goal:

Auxiliary information:

  • \(S\) - set of bias-relevant observable tabular features (descendants of \(A\) in the causal graph, e.g., number of programming languages, years of experience, etc.)
  • \(M\) - structural causal model over \(S\) that captures the causal effect of \(A\) on a task.

Procedure

0. Pre-trained encoder \(f: X \rightarrow Z\)

1. Semantic Decoder \(g: Z \rightarrow S\)

  • Partial mapping from latent space to structured representation \(S\)
  • Trained on synthetic data with matching \(X\) and \(S\)
  • Consistency signal for step 2

2. Neurosymbolic Manipulator \(h_{S/Z}\)

  • Symbolic operation: \(h_S: S \rightarrow S'\) calculates the counterfactual \(S'\) to \(S\) w.r.t. \(A\) via abduction-action-prediction on \(M\)
  • Neural manipulator: \(h_Z: Z \rightarrow Z\) learns the transformed latent representation \(Z' := h_Z(Z)\) analogously to \(h_S(S)\) \[\begin{equation} h_S(g(Z)) = g(h_Z(Z)) \end{equation}\]
  • Training objective (\(d_S\) is a distance function in \(S\)-space): \[\begin{equation} \mathcal{L} = \underbrace{d_S(h_S(g(Z)), g(h_Z(Z)))}_{\text{Consistency Constraint}} + \underbrace{ \alpha \cdot ||h_Z(Z) - Z||_1}_{L_1\text{: Sparsity Penalty}} + \underbrace{\beta \cdot ||h_Z(Z) - Z||_2^2}_{L_2 \text{: Deviation Penalty}} \end{equation}\]

3. Downstream Task

  • E.g., a predictor \(p\) with constraint \(p(Z) \neq p(Z')\)

Data & Hypotheses

Synthetic Data

  • Approach simliar to Toker et al. (2026)
  • Dataset with matching \(X\) and \(S\) to train the semantic decoder \(g\)

1. SCM $M$ models some causal paths from $A$ and involving $S$ as descendants. 2. Tabular data $S$ is sampled from $M$. 3. LLM generates $X$ from each instance of $S$.

  • Challenges:
    • LLM may introduce additional hidden confounders descending from \(A\)
    • External validty

Real World Data

  • CV dataset, e.g., Bias in Bios (De-Arteaga et al. 2019)
  • Challenges:
    • SCM requires strong causal assumptions
    • Tabular representation \(S\) is unknown

1. An SCM is constructed fo an existing CV dataset. 2. The dataset is annotated with a tabular entry in $S$ for each CV.

Hypotheses

  1. Analogy: \(h_Z\) can be trained to generate the counterfactual of \(Z\) analoguously to \(h_S\)
  2. Sparsity: \(h_Z\) only adjusts observed confounders
  3. Effectiveness: \(Z'\) and \(Z\) can be used to debias a prediction model
  4. Meaningfulness: There exists a decoder that can retrieve the counterfactual \(X'\) to \(X\) from \(Z'\)

Discussion

Advantage: Modularity

  • Flexibile, auditable and composable
  • Independence from encoder training

Limitations: SCM Assumptions

  • No hidden confounders \(\rightarrow\) \(M\) is accurate
  • Full observability of \(S\) \(\rightarrow\) \(g\) is well-fit

Scope

  • Value beyond fairness, e.g., privacy or safety
    \(\rightarrow\) Latent Symbolic Interventions

leo.kestel@lmu.de · leokestel.de