Why I Spent a Year Reading XAI Papers

When I started the literature survey that became our ACM Computing Surveys paper, I thought I’d find consensus on what makes a “good” explanation.

I was wrong.

After 202 papers, 17 technique benchmarks, and countless debates with co-authors, here’s what I actually learned—the insights that changed how I build explainable systems.

Surprise #1: We’re Explaining the Wrong Thing

What papers focus on: How models make predictions

What users actually need: Why they should trust this prediction

The Example That Opened My Eyes

Paper shows: “SHAP says feature X contributed 0.82 to the prediction”

User asks: “But is feature X even relevant to this decision? Should it be?”

The gap: Fidelity (accurate model representation) ≠ Trustworthiness (should I believe this?)

What Changed in My Work

Before survey:

# Generate explanation
shap_values = explainer(model, input)
display(shap_values)  # Show feature importance

After survey:

# Generate explanation WITH context
shap_values = explainer(model, input)

# Add domain validation
relevant_features = validate_against_domain_knowledge(shap_values)

# Add uncertainty
confidence = estimate_explanation_stability(input, shap_values)

# Present meaningfully
display_with_context(
    features=relevant_features,
    confidence=confidence,
    what_to_trust="These features align with medical guidelines",
    what_to_question="Feature Y unexpected, worth verifying"
)

Lesson: Explanations need context, not just attribution scores.

Surprise #2: Only 34% Include Human Evaluation

Of 202 papers:

  • 66%: Pure computational metrics (fidelity, consistency, etc.)
  • 34%: Actually tested with humans
  • 12%: Tested with domain experts (doctors, lawyers, etc.)

Why This Matters

Paper claimed: “Our method achieves 94% fidelity”

Reality: Users found it completely incomprehensible.

Fidelity measures: Does explanation match model? Comprehensibility asks: Can humans understand it?

The Humbling Experience

For one of my projects, I was proud of my SHAP implementation:

  • ✅ High fidelity scores
  • ✅ Computationally stable
  • ✅ Fast inference

Then I showed it to actual users:

  • ❌ “What do these numbers mean?”
  • ❌ “Why should I care about feature X?”
  • ❌ “This doesn’t help me decide”

Fix: Added three layers:

  1. Translate: Feature names → plain English
  2. Contextualize: Importance scores → comparative statements (“Higher than 80% of cases”)
  3. Actionable: “What would change the outcome?”

Lesson: If you’re not testing with real users, you’re not building XAI—you’re building model introspection tools.

Surprise #3: SHAP vs. LIME Isn’t the Right Question

Papers endlessly compare:

  • “SHAP is more consistent”
  • “LIME is faster”
  • “SHAP has theoretical guarantees”

What the survey revealed: Pick based on your stakeholder, not the method’s theoretical properties.

The Stakeholder Framework I Now Use

Stakeholder Needs Best Method
Data Scientist Debug model, find bugs SHAP (global + local insights)
Regulator Verify fairness, audit trail Counterfactuals (actionable, transparent)
End User Understand this decision Example-based (analogical reasoning)
Domain Expert Validate reasoning Concept-based (high-level abstractions)

Real example: Healthcare AI

  • Doctor: Wants to see which symptoms drove diagnosis (features)
  • Patient: Wants to know what they can do to change outcome (counterfactuals)
  • Hospital Admin: Needs consistent explanations across patients (SHAP global)
  • Regulator: Requires audit trail of decisions (all of the above)

One explanation method can’t serve all. I now build multi-explanation systems.

Lesson: Choose explanation method based on who’s asking, not which paper cited it most.

Surprise #4: Counterfactuals Are Underrated

Only 15% of surveyed papers focused on counterfactual explanations.

Yet in user studies I ran later: Counterfactuals were preferred 2:1 over feature importance.

Why Counterfactuals Resonate

Feature importance says: “Credit score (0.65) and income (0.23) contributed most”

Counterfactual says: “If your credit score were 720 instead of 650, you’d be approved”

Which is more useful?

The Implementation That Changed My Practice

def generate_actionable_counterfactual(model, input, outcome="approved"):
    # Find minimal changes to flip outcome
    changes = optimizer.find_minimal_changes(
        model=model,
        current=input,
        target_outcome=outcome,
        constraints={
            'age': 'immutable',           # Can't change
            'credit_score': 'slow',       # Takes time
            'income': 'actionable',       # Can increase
            'debt_ratio': 'actionable'    # Can reduce
        }
    )

    # Present in human terms
    return f"""
    To get approved:
    - Reduce debt-to-income ratio from 0.45 to 0.35 (most impactful)
    - OR increase credit score from 650 to 720 (takes 6-12 months)

    Changing income alone won't be enough.
    """

Lesson: Feature importance tells you “why”. Counterfactuals tell you “what to do”.

Users prefer actionable insights.

Surprise #5: The Evaluation Metric Chaos

Identified 5 categories of XAI evaluation metrics, often contradicting each other:

  1. Fidelity: How well explanation reflects model
  2. Interpretability: Human comprehensibility
  3. Stability: Robustness to perturbations
  4. Actionability: Usefulness for decisions
  5. Fairness: Bias detection

The problem: A method can score high on fidelity but low on interpretability.

Example: The SHAP Paradox

SHAP on deep neural net:

  • ✅ High fidelity (game-theoretically sound)
  • ❌ Low interpretability (hundreds of features with small values)
  • ✅ Good stability (consistent across similar inputs)
  • ❌ Low actionability (most features not changeable)

My solution: Multi-metric dashboards

class ExplanationQuality:
    def evaluate(self, explanation, model, input):
        return {
            'fidelity': self.compute_fidelity(explanation, model),
            'comprehensibility': self.user_study_score(explanation),
            'stability': self.perturbation_robustness(explanation, input),
            'actionability': self.count_actionable_features(explanation),
            'fairness': self.check_demographic_parity(explanation)
        }

    def recommend(self, scores):
        if scores['comprehensibility'] < 0.6:
            return "Simplify explanation (fewer features)"
        if scores['stability'] < 0.7:
            return "Explanation unreliable, investigate model"
        # ...

Lesson: Don’t optimize for one metric. Track multiple dimensions and make trade-offs explicit.

Surprise #6: Healthcare Dominates (28% of Papers)

Why healthcare leads:

  • Regulatory requirements (GDPR, HIPAA)
  • High stakes (life/death decisions)
  • Doctor-AI collaboration culture

What other domains can learn from healthcare XAI:

  1. Uncertainty quantification is mandatory
    • Don’t just give diagnosis, give confidence
    • Show when model is uncertain
  2. Explanations must cite evidence
    • “Based on symptoms A, B, C (see medical literature X, Y)”
    • Traceability to source knowledge
  3. Domain expert validation
    • Doctors review explanations for medical plausibility
    • Catch hallucinations early

How I applied this to finance:

Instead of:

"Loan denied. Credit score too low."

Now:

"Loan denied (confidence: 78%)
Reason: Credit score 650 below threshold 680
Reference: Credit policy section 3.2
Uncertainty: Borderline case, request manual review"

Lesson: Borrow evaluation practices from high-stakes domains, even if yours isn’t life-or-death.

The Gaps Nobody’s Talking About

After surveying 202 papers, here’s what’s not being researched (but should be):

Gap 1: Cultural Context (0.5% of papers)

Do explanations work the same across cultures?

  • Western preference: causal “why” explanations
  • Some cultures prefer: analogical or narrative explanations

No one is studying this. Huge opportunity.

Gap 2: Temporal Explanations (4% of papers)

Most XAI: “Why this prediction now?”

Needed: “How did the model’s reasoning change over time?”

Use case: Fraud detection

  • “Account flagged due to unusual spending”
  • Better: “Account was normal until March 15, then spending pattern shifted from X to Y, triggering fraud signal”

Gap 3: Interactive Refinement (8% of papers)

Current: Model → Explanation → User (one-way)

Needed: User can probe, ask follow-ups, refine explanation

Example conversation:

System: "Rejected due to high debt ratio"
User: "What if I pay off my car loan?"
System: "Then debt ratio becomes 0.30, you'd be approved"
User: "What if I only pay half?"
System: "Debt ratio becomes 0.37, still rejected but closer"

Very few systems do this. Huge UX opportunity.

Practical Takeaways for Builders

1. Start with Stakeholder Needs, Not Techniques

Before choosing SHAP/LIME/etc., ask:

  • Who needs this explanation?
  • What decision are they making?
  • What’s their technical background?

Then pick method.

2. Validate with Humans Early

Don’t wait for perfect fidelity scores. Show real users early, iterate.

3. Combine Multiple Explanation Types

One size doesn’t fit all:

  • Feature importance (for debugging)
  • Counterfactuals (for action)
  • Examples (for intuition)

4. Measure What Matters

Track:

  • User task completion time
  • Decision confidence improvement
  • Error reduction when using explanations

Not just:

  • Computational fidelity
  • Algorithmic consistency

5. Build Explanation Quality Into Your ML Pipeline

# Don't bolt on XAI at the end
model = train_model(data)
explanation = generate_shap(model)  # ❌ Afterthought

# Design for explainability from start
model = train_interpretable_model(data)  # ✅ Built-in
validate_explanation_quality(model)      # ✅ Tested
monitor_explanation_drift(model)         # ✅ Maintained

What I’m Doing Differently Now

  1. Multi-explanation dashboards instead of single method
  2. Human evaluation from day 1 (even if just 5 users)
  3. Explanation quality metrics alongside model performance
  4. Stakeholder-specific views of the same explanation
  5. Counterfactuals as default for user-facing systems

Want to Dive Deeper?

Read the full survey: ACM Computing Surveys paper

Related projects using these insights:


What surprised you most about XAI in your work?

I’d love to hear: LinkedIn GitHub