Why I Spent a Year Reading XAI Papers
When I started the literature survey that became our ACM Computing Surveys paper, I thought I’d find consensus on what makes a “good” explanation.
I was wrong.
After 202 papers, 17 technique benchmarks, and countless debates with co-authors, here’s what I actually learned—the insights that changed how I build explainable systems.
Surprise #1: We’re Explaining the Wrong Thing
What papers focus on: How models make predictions
What users actually need: Why they should trust this prediction
The Example That Opened My Eyes
Paper shows: “SHAP says feature X contributed 0.82 to the prediction”
User asks: “But is feature X even relevant to this decision? Should it be?”
The gap: Fidelity (accurate model representation) ≠ Trustworthiness (should I believe this?)
What Changed in My Work
Before survey:
# Generate explanation
shap_values = explainer(model, input)
display(shap_values) # Show feature importance
After survey:
# Generate explanation WITH context
shap_values = explainer(model, input)
# Add domain validation
relevant_features = validate_against_domain_knowledge(shap_values)
# Add uncertainty
confidence = estimate_explanation_stability(input, shap_values)
# Present meaningfully
display_with_context(
features=relevant_features,
confidence=confidence,
what_to_trust="These features align with medical guidelines",
what_to_question="Feature Y unexpected, worth verifying"
)
Lesson: Explanations need context, not just attribution scores.
Surprise #2: Only 34% Include Human Evaluation
Of 202 papers:
- 66%: Pure computational metrics (fidelity, consistency, etc.)
- 34%: Actually tested with humans
- 12%: Tested with domain experts (doctors, lawyers, etc.)
Why This Matters
Paper claimed: “Our method achieves 94% fidelity”
Reality: Users found it completely incomprehensible.
Fidelity measures: Does explanation match model? Comprehensibility asks: Can humans understand it?
The Humbling Experience
For one of my projects, I was proud of my SHAP implementation:
- ✅ High fidelity scores
- ✅ Computationally stable
- ✅ Fast inference
Then I showed it to actual users:
- ❌ “What do these numbers mean?”
- ❌ “Why should I care about feature X?”
- ❌ “This doesn’t help me decide”
Fix: Added three layers:
- Translate: Feature names → plain English
- Contextualize: Importance scores → comparative statements (“Higher than 80% of cases”)
- Actionable: “What would change the outcome?”
Lesson: If you’re not testing with real users, you’re not building XAI—you’re building model introspection tools.
Surprise #3: SHAP vs. LIME Isn’t the Right Question
Papers endlessly compare:
- “SHAP is more consistent”
- “LIME is faster”
- “SHAP has theoretical guarantees”
What the survey revealed: Pick based on your stakeholder, not the method’s theoretical properties.
The Stakeholder Framework I Now Use
| Stakeholder | Needs | Best Method |
|---|---|---|
| Data Scientist | Debug model, find bugs | SHAP (global + local insights) |
| Regulator | Verify fairness, audit trail | Counterfactuals (actionable, transparent) |
| End User | Understand this decision | Example-based (analogical reasoning) |
| Domain Expert | Validate reasoning | Concept-based (high-level abstractions) |
Real example: Healthcare AI
- Doctor: Wants to see which symptoms drove diagnosis (features)
- Patient: Wants to know what they can do to change outcome (counterfactuals)
- Hospital Admin: Needs consistent explanations across patients (SHAP global)
- Regulator: Requires audit trail of decisions (all of the above)
One explanation method can’t serve all. I now build multi-explanation systems.
Lesson: Choose explanation method based on who’s asking, not which paper cited it most.
Surprise #4: Counterfactuals Are Underrated
Only 15% of surveyed papers focused on counterfactual explanations.
Yet in user studies I ran later: Counterfactuals were preferred 2:1 over feature importance.
Why Counterfactuals Resonate
Feature importance says: “Credit score (0.65) and income (0.23) contributed most”
Counterfactual says: “If your credit score were 720 instead of 650, you’d be approved”
Which is more useful?
The Implementation That Changed My Practice
def generate_actionable_counterfactual(model, input, outcome="approved"):
# Find minimal changes to flip outcome
changes = optimizer.find_minimal_changes(
model=model,
current=input,
target_outcome=outcome,
constraints={
'age': 'immutable', # Can't change
'credit_score': 'slow', # Takes time
'income': 'actionable', # Can increase
'debt_ratio': 'actionable' # Can reduce
}
)
# Present in human terms
return f"""
To get approved:
- Reduce debt-to-income ratio from 0.45 to 0.35 (most impactful)
- OR increase credit score from 650 to 720 (takes 6-12 months)
Changing income alone won't be enough.
"""
Lesson: Feature importance tells you “why”. Counterfactuals tell you “what to do”.
Users prefer actionable insights.
Surprise #5: The Evaluation Metric Chaos
Identified 5 categories of XAI evaluation metrics, often contradicting each other:
- Fidelity: How well explanation reflects model
- Interpretability: Human comprehensibility
- Stability: Robustness to perturbations
- Actionability: Usefulness for decisions
- Fairness: Bias detection
The problem: A method can score high on fidelity but low on interpretability.
Example: The SHAP Paradox
SHAP on deep neural net:
- ✅ High fidelity (game-theoretically sound)
- ❌ Low interpretability (hundreds of features with small values)
- ✅ Good stability (consistent across similar inputs)
- ❌ Low actionability (most features not changeable)
My solution: Multi-metric dashboards
class ExplanationQuality:
def evaluate(self, explanation, model, input):
return {
'fidelity': self.compute_fidelity(explanation, model),
'comprehensibility': self.user_study_score(explanation),
'stability': self.perturbation_robustness(explanation, input),
'actionability': self.count_actionable_features(explanation),
'fairness': self.check_demographic_parity(explanation)
}
def recommend(self, scores):
if scores['comprehensibility'] < 0.6:
return "Simplify explanation (fewer features)"
if scores['stability'] < 0.7:
return "Explanation unreliable, investigate model"
# ...
Lesson: Don’t optimize for one metric. Track multiple dimensions and make trade-offs explicit.
Surprise #6: Healthcare Dominates (28% of Papers)
Why healthcare leads:
- Regulatory requirements (GDPR, HIPAA)
- High stakes (life/death decisions)
- Doctor-AI collaboration culture
What other domains can learn from healthcare XAI:
- Uncertainty quantification is mandatory
- Don’t just give diagnosis, give confidence
- Show when model is uncertain
- Explanations must cite evidence
- “Based on symptoms A, B, C (see medical literature X, Y)”
- Traceability to source knowledge
- Domain expert validation
- Doctors review explanations for medical plausibility
- Catch hallucinations early
How I applied this to finance:
Instead of:
"Loan denied. Credit score too low."
Now:
"Loan denied (confidence: 78%)
Reason: Credit score 650 below threshold 680
Reference: Credit policy section 3.2
Uncertainty: Borderline case, request manual review"
Lesson: Borrow evaluation practices from high-stakes domains, even if yours isn’t life-or-death.
The Gaps Nobody’s Talking About
After surveying 202 papers, here’s what’s not being researched (but should be):
Gap 1: Cultural Context (0.5% of papers)
Do explanations work the same across cultures?
- Western preference: causal “why” explanations
- Some cultures prefer: analogical or narrative explanations
No one is studying this. Huge opportunity.
Gap 2: Temporal Explanations (4% of papers)
Most XAI: “Why this prediction now?”
Needed: “How did the model’s reasoning change over time?”
Use case: Fraud detection
- “Account flagged due to unusual spending”
- Better: “Account was normal until March 15, then spending pattern shifted from X to Y, triggering fraud signal”
Gap 3: Interactive Refinement (8% of papers)
Current: Model → Explanation → User (one-way)
Needed: User can probe, ask follow-ups, refine explanation
Example conversation:
System: "Rejected due to high debt ratio"
User: "What if I pay off my car loan?"
System: "Then debt ratio becomes 0.30, you'd be approved"
User: "What if I only pay half?"
System: "Debt ratio becomes 0.37, still rejected but closer"
Very few systems do this. Huge UX opportunity.
Practical Takeaways for Builders
1. Start with Stakeholder Needs, Not Techniques
Before choosing SHAP/LIME/etc., ask:
- Who needs this explanation?
- What decision are they making?
- What’s their technical background?
Then pick method.
2. Validate with Humans Early
Don’t wait for perfect fidelity scores. Show real users early, iterate.
3. Combine Multiple Explanation Types
One size doesn’t fit all:
- Feature importance (for debugging)
- Counterfactuals (for action)
- Examples (for intuition)
4. Measure What Matters
Track:
- User task completion time
- Decision confidence improvement
- Error reduction when using explanations
Not just:
- Computational fidelity
- Algorithmic consistency
5. Build Explanation Quality Into Your ML Pipeline
# Don't bolt on XAI at the end
model = train_model(data)
explanation = generate_shap(model) # ❌ Afterthought
# Design for explainability from start
model = train_interpretable_model(data) # ✅ Built-in
validate_explanation_quality(model) # ✅ Tested
monitor_explanation_drift(model) # ✅ Maintained
What I’m Doing Differently Now
- Multi-explanation dashboards instead of single method
- Human evaluation from day 1 (even if just 5 users)
- Explanation quality metrics alongside model performance
- Stakeholder-specific views of the same explanation
- Counterfactuals as default for user-facing systems
Want to Dive Deeper?
Read the full survey: ACM Computing Surveys paper
Related projects using these insights:
- Student Risk Prediction - Multi-stakeholder explanations
- Clone-Me Console - Warrant Inspector (explanation UI)
What surprised you most about XAI in your work?
| I’d love to hear: LinkedIn | GitHub |