Black-box Model Interpretation
Published:
**2D PDP (Interaction Plots):** For **two features**, PDP generates a **heatmap** showing interactions.
::: :::: ::: footer Friedman, J. H. (2001). [*Greedy function approximation: A gradient boosting machine*](https://doi.org/10.1214/aos/1013203451). Annals of Statistics. ::: ::: {.notes} PDPs are one of the oldest and most intuitive model-agnostic interpretation methods. The key idea: marginalize over all other features to isolate the effect of one (or two) features on the prediction. The figure is three PDPs side by side from Molnar, all for the bike model: temperature, humidity and wind speed, sharing a y axis of predicted bike count. Temperature climbs from about 3,400 rentals at freezing to roughly 5,000 near 22 degrees, then eases off, the sweet spot the green annotation marks. Humidity is flat until about 60 percent and then falls away. Wind speed declines gently throughout. The black tick marks along the bottom of each panel are a rug plot showing where the data actually is, which is how you know the far right of each curve rests on very few days. The visualization immediately reveals this pattern that might be hidden in model coefficients. For two features, PDP generates a 2D plot (heatmap) showing interactions - the more complex the contour lines, the stronger the interaction between features. For example, a 2D PDP of temperature Γ humidity might show that high temperature increases rentals only when humidity is low. PDPs answer: "On average, how does changing this feature affect predictions?" ::: ## PDP Visualization: Interpreting Feature Effects :::: {.columns} ::: {.column width="33%"} **1. Monotonic/Linear** π¦π¦π¨π§π§
π¦π¦π¨π§π§
π©π©π¨π¨π¨
π©π©π¨π¨π¨
**Complex contours** = strong feature interactionπ¦π¦π¨π§π§
π©π©π¨π¨π¨
π©π©π¨π¨π¨
β
β
β
Straight diagonal line β Feature β β Prediction Feature has consistent positive (or negative) effect ::: ::: {.column width="33%"} **2. Non-linear/Sweet Spot**β
β
β±βΎβ²
β± β²
β± β²
Curve peaks and drops Optimal range exists (like temperature example) ::: ::: {.column width="33%"} **3. Flat Line**β± β²
β± β²
βββββββ
**The Problem:** PDP shows the **average effect** across all instances. But what if the effect differs between: - Cold days vs. warm days? - New customers vs. loyal customers? - Low-income vs. high-income applicants? **PDP cannot tell you!**
::: ::: {.column width="50%"} **The Solution:** Move from **global** to **local** explanations. Instead of asking: > "How does temperature affect rentals **on average**?" Ask: > "Why did the model predict **this specific value** for **this particular day**?" **β Enter LIME**
::: :::: ::: {.notes} This slide bridges the conceptual gap between PDP and LIME. PDP's key limitation - "hidden heterogeneous effects from averaging" - means we lose important information about how features affect different types of instances differently. For example, temperature might increase bike rentals on sunny days but decrease them on rainy days, and PDP would just show the average. This motivates the need for local explanations that focus on individual instances. LIME addresses this by building interpretable surrogate models around specific predictions, allowing us to understand "why THIS prediction for THIS instance?" rather than just global averages. ::: ## Local Interpretable Model-agnostic Explanations (LIME) Training local surrogate models to explain *individual* predictions :::: {.columns} ::: {.column width="60%"} {height="760" fig-align="center"} ::: ::: {.column width="40%"} {height="760" fig-align="center"} ::: :::: ::: footer Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). [*"Why Should I Trust You?" Explaining the Predictions of Any Classifier*](https://arxiv.org/pdf/1602.04938.pdf). KDD. ::: ::: {.notes} LIME represents a paradigm shift from global to local explanations. The key insight: even if the global decision boundary is complex (left image shows a highly non-linear boundary), we can approximate it LOCALLY with a simple linear model. Think of it like approximating a curve with a tangent line - it only works near the point of tangency. The title "Why Should I Trust You?" captures the motivation: users need to understand individual predictions to build trust. LIME became hugely influential because it's model-agnostic, produces human-interpretable explanations, and works for any data type (tabular, text, images). ::: ## LIME: Training Local Surrogate Models The idea is quite intuitive. **First, forget about the training data and imagine you only have the black box model where you can input data points and get the predictions of the model. You can probe the box as often as you want. Your goal is to understand why the machine learning model made a certain prediction. LIME tests what happens to the predictions when you give variations of your data into the machine learning model.** **LIME generates a new dataset consisting of perturbed samples and the corresponding predictions of the black box model.** **On this new dataset LIME then trains an interpretable model, which is weighted by the proximity of the sampled instances to the instance of interest.** The interpretable model can be anything from the interpretable models chapter, for example Lasso or a decision tree. **The learned model should be a good approximation of the machine learning model predictions locally, but it does not have to be a good global approximation. This kind of accuracy is also called *local fidelity*.** Mathematically, local surrogate models with interpretability constraint can be expressed as follows: $$\text{explanation}(x) = \arg\min_{g \in G} L(f, g, \pi_x) + \Omega(g)$$ ::: footer Section 9.2, on Molnar's book. ::: ::: {.notes} This is the paragraph from Molnar worth reading slowly, because it defines the method's whole stance. Forget the training data. You have a box: you feed it inputs, it returns predictions, and you can probe it as often as you like. That is the only access LIME assumes, which is exactly why it is model agnostic. The procedure in one sentence: generate variations of your instance, ask the box what it predicts for each, and fit a simple interpretable model to that new dataset, weighting each sample by how close it is to the instance you care about. Then the key term: local fidelity. The surrogate does not need to approximate the model anywhere else. It has to be right near this one point. A linear model that is hopeless globally can be an excellent description of a curved boundary over a small neighbourhood, the same way a tangent line describes a curve near the point of tangency. Unpack the objective on the slide. Three parts: L is the fidelity loss, how badly the surrogate g reproduces the black box f, measured only over the neighbourhood defined by the proximity kernel pi_x; omega of g penalises the surrogate's own complexity, which is what keeps the explanation short; and the arg min over a family G of interpretable models is where you choose between a sparse linear model and a shallow tree. Every knob students will later complain about, kernel width, number of features, choice of surrogate, is one of those three symbols. ::: ## LIME: The Algorithm ### Algorithm 1. Pick an input that you want an explanation for. 2. Sample the neighbors of the selected input (i.e. perturbation). 3. Train a linear classifier on the neighbors. 4. The weights on the linear classifier is the explanation. ::: {.notes} The four-line version of the previous slide, and the one to put on the board. Pick an instance. Perturb around it. Fit a weighted linear model on the perturbed set. Read the coefficients as the explanation. The next four slides walk through exactly these steps on a two-dimensional toy problem, one step per slide, so the room can watch the neighbourhood form. ::: ## LIME Step 1: The Black-box Decision Boundary :::: {.columns} ::: {.column width="60%"} {height="760" fig-align="center"} ::: ::: {.column width="40%"} Random forest predictions given features x1 and x2. Predicted classes: 1 (dark) or 0 (light). ::: :::: ::: {.notes} Step one, panel A from Molnar. This is the black box we are trying to explain: a random forest on two features, x1 and x2. Light blue is the region predicted class 1, grey is class 0, and the blue region is a narrow inverted V, two thin wedges meeting near x1 equals zero. The shape is the point. No linear model could describe this boundary globally. Ask the room to imagine fitting a single straight line to separate blue from grey, and let them conclude it is hopeless. That global hopelessness is what makes the local trick interesting. ::: ## LIME Step 2: Sample Around the Instance :::: {.columns} ::: {.column width="60%"} {height="760" fig-align="center"} ::: ::: {.column width="40%"} Instance of interest (big yellow dot) and data sampled from a normal distribution (small dots). ::: :::: ::: {.notes} Step two, panel B. The big yellow dot near x1 equals 1, x2 equals minus 0.5 is the instance we want explained, and it sits in the grey region, so the model predicts class 0 for it. The small black dots are the perturbed samples, drawn here from a normal distribution over the feature space. Note what LIME is doing: it is not using the training data at all, it is manufacturing its own dataset and labelling it with the black box. Note also that the samples are spread across the whole plot, not just near the yellow dot. That is deliberate, and the next slide is what fixes it. ::: ## LIME Step 3: Weight by Proximity :::: {.columns} ::: {.column width="60%"} {height="760" fig-align="center"} ::: ::: {.column width="40%"} Assign higher weight to points near the instance of interest. I.e., $weight(p) = \sqrt{\frac{e^{-d^2}}{w^2}}$ where $d$ is the distance between $p$ and the instantce of interest, and $w$ is the kernel width (self-defined). ::: :::: ::: {.notes} Step three, panel C. The same samples, now drawn with size proportional to their weight. The weights come from an exponential kernel on distance, the formula on the right, so points near the yellow instance count heavily and far points fade almost to nothing. You can see the black mass concentrated around the instance. The kernel width w is the single most consequential parameter in LIME and it is chosen by the user. Too wide and you are fitting a global model again, and the explanation blurs. Too narrow and only a handful of samples carry any weight, so the fit is noisy and changes every time you run it. There is no principled default. This is the honest answer to the instability complaint on the pros and cons slide, and it is worth saying out loud now rather than later. ::: ## LIME Step 4: Fit a Local Linear Model :::: {.columns} ::: {.column width="60%"} {height="760" fig-align="center"} ::: ::: {.column width="40%"} Use both the samples and sample weights to train a linear classifier. Signs of the grid show the classifications of the locally learned model from the weighted samples. The red line marks the decision boundary (P(class=1) = 0.5). The official implementation uses a Ridge Classifier as the linear model for explanation. ::: :::: ::: {.notes} Step four, panel D. A ridge classifier is fitted to the weighted samples. The plus and minus signs on the grid show what the local surrogate predicts, and the red line is its decision boundary at probability one half. Read the line against the true boundary. Globally it is wrong: it slices straight through the left wedge. Locally, in the neighbourhood of the yellow dot, it tracks the right edge of the blue wedge closely. That is local fidelity, and it is the whole claim of the method. The coefficients of that red line are the explanation. Two numbers here, one per feature, and their signs and magnitudes are what a user would be shown as the bar chart. ::: ## LIME: The Weighted Ridge Objective :::: {.columns} ::: {.column width="50%"} {height="640" fig-align="center"} $s_i$ = sample weight, $\lambda$ = regularization term ::: ::: {.column width="50%"} **Ridge Classifier** $$minimize \sum_{i=1}^{M} s_i(y_i - \hat{y}_i)^2$$ $$= \sum_{i=1}^{M} s_i(y_i - \sum_{j=0}^{p} w_j \times x_{ij})^2 + \lambda \sum_{j=0}^{p} w_j^2$$ $w_j$ = trained weight to explain the importance of feature j The higher the $\lambda$, the more sparse the $w$ (more zeros) will become. ::: :::: ::: footer Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). [*"Why Should I Trust You?" Explaining the Predictions of Any Classifier*](https://arxiv.org/pdf/1602.04938.pdf). KDD. ::: ::: {.notes} The objective written out. It is weighted ridge regression: squared error between the surrogate and the black box, each term multiplied by that sample's proximity weight s_i, plus an L2 penalty on the coefficients. Two things to point at. The s_i is where the previous slide's kernel enters the algebra, so the geometry and the formula are the same idea. And lambda controls how much the coefficients shrink, which is the sparsity knob from the objective two slides back, the omega term. In the reference implementation the surrogate is a ridge classifier, and feature selection is applied on top to cut the explanation down to the few features the user actually sees. Note the sign convention: w_j is what gets reported as the importance of feature j. ::: ## LIME Visualization: Bar Charts for Local Explanations **Primary LIME Output: Sparse Bar Charts** :::: {.columns} ::: {.column width="50%"} **Case 1: High Rental Day** *Prediction: ABOVE Trend* β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ Temp > 20Β°C
β¬β¬β¬β¬β¬β¬β¬ Windspeed Low
β¬β¬β¬ Holiday = False
β Warm temperature strongly supports high rentals
β Low wind moderately supports
β Non-holiday slightly opposes
::: ::: {.column width="50%"} **Case 2: Low Rental Day** *Prediction: BELOW Trend*β Low wind moderately supports
β Non-holiday slightly opposes
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ Weather: Rain
β¬β¬β¬β¬β¬β¬β¬β¬ Temp < 5Β°C
β¬β¬ Weekday = True
β Rain strongly opposes high rentals
β Cold temperature opposes
β Weekday weakly supports
::: :::: ::: {.notes} Emphasize the visualization: **Color** (Green supports the prediction, Red opposes it), **Length of Bar** (magnitude of influence), and **Sparsity** (only a few features are shownβthis makes it easy for a non-expert to trust). The bar chart is the key deliverable of LIME - it provides a contrastive explanation showing which features push toward vs away from the prediction. Compare the two cases: Case 1 is dominated by green (supporting features), while Case 2 is dominated by red (opposing features). This visual contrast makes the explanation immediately understandable. ::: ## Local Surrogate (LIME): Bike Rental Example **Task:** Predict if bike rentals will be above or below trend ```{python} #| echo: false #| output: false import pandas as pd import numpy as np from sklearn.ensemble import RandomForestClassifier import lime import lime.lime_tabular import matplotlib.pyplot as plt # Load bike sharing dataset url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00275/Bike-Sharing-Dataset.zip" import urllib.request import zipfile import io # Download and extract response = urllib.request.urlopen(url) zip_file = zipfile.ZipFile(io.BytesIO(response.read())) zip_file.extractall("/tmp/bike_data") # Load day.csv bike_df = pd.read_csv("/tmp/bike_data/day.csv") # Create features feature_names = ['season', 'weathersit', 'temp', 'hum', 'windspeed', 'workingday'] X = bike_df[feature_names].values y = (bike_df['cnt'] > bike_df['cnt'].median()).astype(int) # Above/below median # Train random forest rf_model = RandomForestClassifier(n_estimators=100, random_state=42) rf_model.fit(X, y) # Create LIME explainer explainer = lime.lime_tabular.LimeTabularExplainer( training_data=X, feature_names=feature_names, class_names=['BELOW trend', 'ABOVE trend'], mode='classification', random_state=42 ) # Find a high rental day (warm, good weather) high_idx = bike_df[(bike_df['temp'] > 0.7) & (bike_df['weathersit'] == 1)].index[0] high_instance = X[high_idx] exp_high = explainer.explain_instance(high_instance, rf_model.predict_proba, num_features=5) # Find a low rental day (cold, bad weather) low_idx = bike_df[(bike_df['temp'] < 0.3) & (bike_df['weathersit'] == 3)].index[0] low_instance = X[low_idx] exp_low = explainer.explain_instance(low_instance, rf_model.predict_proba, num_features=5) # Store predictions pred_high = rf_model.predict_proba([high_instance])[0] pred_low = rf_model.predict_proba([low_instance])[0] ``` :::: {.columns} ::: {.column width="50%"} **High Rental Day** Prediction: **{python} f"{pred_high[1]:.2f}"** probability ABOVE trend ```{python} #| echo: false #| fig-width: 6.5 #| fig-height: 4.5 fig = exp_high.as_pyplot_figure(label=1) ax = fig.gca() ax.set_xlabel('Feature Contribution to ABOVE Trend', fontsize=13) ax.tick_params(labelsize=12) ax.set_title('LIME Explanation', fontsize=15, fontweight='bold') fig.tight_layout() fig ```β Cold temperature opposes
β Weekday weakly supports
β High temperature strongly supports
β Good weather condition supports
::: ::: {.column width="50%"} **Low Rental Day** Prediction: **{python} f"{pred_low[0]:.2f}"** probability BELOW trend ```{python} #| echo: false #| fig-width: 6.5 #| fig-height: 4.5 # For low rental, also explain label=1 (ABOVE) to show negative contributions fig = exp_low.as_pyplot_figure(label=1) ax = fig.gca() ax.set_xlabel('Feature Contribution to ABOVE Trend', fontsize=13) ax.tick_params(labelsize=12) ax.set_title('LIME Explanation', fontsize=15, fontweight='bold') fig.tight_layout() fig ```β Good weather condition supports
β Bad weather strongly opposes
β Low temperature opposes
::: :::: ::: {.notes} This slide shows actual LIME explanations for bike rental predictions. The bar charts display feature contributions - green bars indicate features supporting the prediction, orange/red bars indicate features opposing it. Notice how the high rental day is dominated by positive contributions (warm temp, good weather), while the low rental day is dominated by negative contributions (bad weather, cold temp). The model predictions are shown at the top - one instance has high probability of being ABOVE trend, the other BELOW trend. This is exactly the kind of local explanation LIME provides - understanding which features matter most for each specific prediction. ::: ::: footer Bike Sharing Dataset from UCI Machine Learning Repository. Fanaee-T, H., & Gama, J. (2014). Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 2(2-3), 113-127. ::: ## LIME on Images: Superpixels as Features **Training local surrogate models to explain individual predictions** {width=90%} ::: footer Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). [*"Why Should I Trust You?" Explaining the Predictions of Any Classifier*](https://arxiv.org/pdf/1602.04938.pdf). KDD. Figure 4. ::: ::: {.notes} This figure demonstrates LIME's model-agnostic nature applied to image classification with Google's Inception neural network. The key adaptation for images is using superpixels (coherent image regions) as interpretable features instead of individual pixels. Notice how LIME provides instance-specific explanations: when explaining "Electric guitar" (p=0.32), it highlights the guitar; when explaining "Acoustic guitar" (p=0.24), it highlights acoustic features; when explaining "Labrador" (p=0.21), it highlights the dog's face. This shows that different predictions for the same image get different explanations - LIME adapts to explain what matters for each specific prediction. The gray regions are turned off (superpixels not contributing to that prediction), while colored regions are what the local linear model identifies as most important. ::: ## LIME for Debugging: The Husky and the Wolf **Raw data and explanation of a bad model's prediction in the "Husky vs Wolf" task** {width=70%} ::: footer Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). [*"Why Should I Trust You?" Explaining the Predictions of Any Classifier*](https://arxiv.org/pdf/1602.04938.pdf). KDD. Figure 11. ::: ::: {.notes} This is one of the most famous examples from the LIME paper, demonstrating its value for model debugging. A husky is incorrectly classified as a wolf. The LIME explanation reveals why: the model is focusing on the background snow rather than the animal's features. This exposes a serious flaw - the model learned a spurious correlation between wolves and snowy backgrounds in the training data, rather than learning actual wolf features. Without LIME, this model might have seemed accurate on test data, but would fail catastrophically in the real world when encountering wolves without snow or huskies with snow. This illustrates LIME's critical role in building trust and identifying when NOT to trust a model. The explanation shows which superpixels the model is using - and in this case, it's using the wrong ones. ::: ## LIME: Code Example **Dataset:** California Housing (same model as PDP example) ```{python} #| echo: true #| output: false import lime import lime.lime_tabular import numpy as np # Reload California Housing data for this example housing = fetch_california_housing() X_housing, y_housing = housing.data, housing.target # Retrain model on California Housing model = GradientBoostingRegressor( n_estimators=100, max_depth=4, learning_rate=0.1, random_state=0 ).fit(X_housing, y_housing) # Create LIME explainer using training data explainer = lime.lime_tabular.LimeTabularExplainer( training_data=X_housing, feature_names=housing.feature_names, mode='regression', random_state=0 ) # Select an instance to explain (e.g., instance 0) instance_idx = 0 instance = X_housing[instance_idx] # Generate explanation for this instance # predict_fn should return predictions exp = explainer.explain_instance( instance, model.predict, num_features=5 # Show top 5 features ) ``` ::: footer [LIME Documentation](https://lime-ml.readthedocs.io/) ::: ::: {.notes} LIME provides local explanations by training a simple interpretable model around a specific instance. The LimeTabularExplainer is initialized with the training data to understand feature distributions. The key parameter is `num_features` which controls sparsity - we only show the top 5 most important features for this prediction. Unlike PDP which gives global effects, LIME explains why the model made this specific prediction for this specific instance. The `predict` function is passed to LIME so it can probe the model with perturbed versions of the instance. ::: ## LIME: Output Visualization ```{python} #| echo: false #| fig-width: 12 #| fig-height: 7 # Get the explanation as a matplotlib figure # Increase font sizes for better readability fig = exp.as_pyplot_figure() ax = fig.gca() for text in ax.texts: text.set_fontsize(14) ax.tick_params(labelsize=14) ax.set_xlabel(ax.get_xlabel(), fontsize=16) ax.set_title(ax.get_title(), fontsize=18) fig.tight_layout() fig ``` ::: {style="font-size: 1.3em;"} **Explanation for Instance 0:** Predicted value: `{python} f"{model.predict([instance])[0]:.2f}"` (in units of $100k) The bar chart shows which features push the prediction higher (positive) or lower (negative) for this specific house. ::: ::: {.notes} The output shows a horizontal bar chart with the most important features for this specific prediction. Positive bars (pointing right) indicate features that increase the predicted house value, while negative bars (pointing left) decrease it. The length of each bar represents the magnitude of the feature's contribution. Notice this is a LOCAL explanation - it only applies to this one house. If we explained a different house, we'd likely get different important features. This is the key difference from PDP which shows global average effects. The sparse representation (only 5 features) makes it easy for users to understand and trust the prediction. ::: ## LIME: Pros and Cons ::: {.fragment} **Pros** - Explanations are short (= selective) and possibly contrastive. * we can control the sparsity of weight coefficients in the regressions method. - Very easy to use. ::: ::: {.fragment} **Cons** - Unstable results due to sampling. - Hard to weight similar neighbors in a high dimensional dataset. - Many parameters for data scientists to hide biases. ::: ::: {.notes} Close LIME on the trade, and be blunt about the cons because students will hit all three in the lab. Short and contrastive is a genuine strength: you control how many features appear, and the bar chart reads as a sentence. Very easy to use, one object and one call. Unstable results due to sampling is the big one. Run LIME twice on the same instance with a different seed and you can get different features. That is not a bug, it follows from step two. Distance in high dimensions is the second: with fifty features almost everything is far from everything, so the proximity kernel stops distinguishing neighbours and the local neighbourhood is no longer local. And the third is the uncomfortable one, many parameters means a motivated analyst can tune kernel width and feature count until the explanation says what they want. Hold that thought for the adversarial slides in the SHAP section. ::: ## SHAP (SHapley Additive exPlanations) :::: {.columns} ::: {.column width="60%"} {height="760" fig-align="center"} ::: ::: {.column width="40%"} SHAP (Lundberg and Lee 2017a) is a game-theory-inspired method created to explain predictions made by machine learning models. SHAP generates one value per input feature (also known as SHAP values) that indicates how the feature contributes to the prediction of the specified data point. ::: :::: ::: footer Lundberg, S. M., & Lee, S. I. (2017). [*A Unified Approach to Interpreting Model Predictions*](https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf). NeurIPS. | Molnar, C. (2024). [*SHAP Book*](https://christophmolnar.com/books/shap/). ::: ::: {.notes} SHAP brings rigorous game theory to ML interpretability. The key innovation: apply Shapley values from cooperative game theory to feature attribution. This provides a principled answer to "how much did each feature contribute?" with mathematical guarantees (efficiency, symmetry, dummy, additivity axioms). SHAP unified several existing methods under one framework. The image shows a typical SHAP force plot - red pushes prediction higher, blue pushes lower. Unlike LIME which is heuristic, SHAP has strong theoretical foundations. Trade-off: computational cost is higher, but you get provably fair attributions. ::: ## A Short History of Shapley Values and SHAP **Three key milestones:** * **1953**: The introduction of Shapley values in game theory * **2010**: The initial steps toward applying Shapley values in machine learning * **2017**: The advent of SHAP, a turning point in machine learning ::: {.notes} The roadmap for the next block. Three dates: 1953 for the maths, 2010 for the first attempt to apply it to machine learning, 2017 for the version everyone now uses. The gap between the second and third is the interesting part, and it is a lesson about software rather than about theory. ::: ## Lloyd Shapley's Pursuit of Fairness (1953) :::: {.columns .smaller} ::: {.column width="60%"} **Lloyd Shapley** - "The greatest game theorist of all time" - PhD at Princeton (post-WWII): "Additive and Non-Additive Set Functions" - 1953 paper: "A Value for n-Person Games" - **2012 Nobel Prize in Economics** (with Alvin Roth) for work in market design and matching theory **The Problem:** How to fairly divide payouts among players who contribute differently to a cooperative game? **The Solution:** Shapley values provide a mathematical method for fair distribution based on marginal contributions ::: ::: {.column width="40%"} **Applications of Shapley values:** - Political science - Economics - Computer science - Dividing profits among shareholders - Allocating costs among collaborators - Assigning credit in research projectsβ Low temperature opposes
**Key insight**: By 1953, Shapley values were well-established in game theory, but machine learning was still in its infancy.
::: :::: ::: footer Shapley, L. S. (1953). A value for n-person games. Contributions to the Theory of Games, 2(28), 307-317. ::: ::: {.notes} Shapley's question had nothing to do with machine learning. A group cooperates, they generate some total value, and the question is how to split it fairly when members contribute differently and in combination. His answer was to define fairness by a set of properties and then show that exactly one allocation satisfies them, which is the uniqueness result we come back to in a few slides. The Nobel was in 2012, shared with Alvin Roth, and it was for matching and market design rather than for this paper specifically. Shapley died in 2016, the year before SHAP appeared. The point to leave them with: this is a sixty-year-old piece of economics that turned out to answer a question machine learning did not know it was asking. ::: ## Early Days in Machine Learning (2010) **2010: Erik Ε trumbelj and Igor Kononenko propose using Shapley values for ML** - Paper: "An efficient explanation of individual classifications using game theory" - 2014 follow-up: Further methodology development **Why didn't it gain immediate traction?** :::: {.columns .smaller} ::: {.column width="50%"} β **Barriers to adoption:** 1. Explainable AI/Interpretable ML not widely recognized yet 2. **No code released** with the papers 3. Estimation method too slow for images/text 4. Limited awareness outside specialized communities ::: ::: {.column width="50%"} β
**What was needed:** 1. Growing demand for interpretability 2. Open-source implementation 3. Faster computation methods 4. High-profile publication venue 5. Integration with popular ML frameworks ::: :::: ::: footer Ε trumbelj, E., & Kononenko, I. (2010). An efficient explanation of individual classifications using game theory. JMLR, 11, 1-18. ::: ::: {.notes} Ε trumbelj and Kononenko got there first, in JMLR, seven years before SHAP, with essentially the right idea including the sampling approximation. Almost nobody used it. Walk the two columns as a case study in why good ideas fail to spread. Interpretability was not yet a named field with a conference track. No code shipped with the paper. The estimator was too slow for images or text, which were the domains generating excitement. And JMLR, while excellent, was not where the deep learning community was looking in 2010. This is worth a minute for a class full of people about to write research software. The 2017 paper's advantage was not primarily mathematical. ::: ## The SHAP Cambrian Explosion (2017) :::: {.columns} ::: {.column width="42%"} **2016: LIME catalyzes the field** Ribeiro et al. introduce Local Interpretable Model-Agnostic Explanations β growing interest in model interpretability **2017: Lundberg and Lee publish SHAP at NeurIPS** "A Unified Approach to Interpreting Model Predictions" ::: ::: {.column width="58%"} **Key contributions beyond the 2010 work:** 1. **Kernel SHAP**: new estimation method using weighted linear regression 2. **Unification framework**: connected SHAP to LIME, DeepLIFT and Layer-Wise Relevance Propagation 3. **Open-source `shap` package**: wide range of features and plotting capabilities 4. **High-profile venue**: published at a major ML conference ::: :::: ::: {.notes} What changed. LIME in 2016 created the demand by making interpretability a visible topic. Then Lundberg and Lee published at NeurIPS with four things the 2010 work lacked. Kernel SHAP, an estimator that recasts the Shapley computation as a weighted linear regression, which is also the formal bridge to LIME. The unification argument, showing that LIME, DeepLIFT and layer-wise relevance propagation are all special cases of one additive attribution framework, which is what the title means by unified. A maintained open-source package with good plots. And a venue the whole field reads. The unification claim is the intellectually interesting one: it says these methods are not competitors but different estimators of the same quantity. ::: ## Why SHAP Gained Popularity :::: {.columns .smaller} ::: {.column width="50%"} **Critical success factors:** β Published at prestigious venue (NeurIPS)β Pioneering work in rapidly growing field
β **Open-source Python package**, which enabled integration
β Ongoing development by original authors
β Strong community contributions
β Comprehensive visualization tools **2020 breakthrough:** TreeSHAP - Efficient computation for tree-based models - Enabled SHAP for state-of-the-art models - Made global interpretations possible ::: ::: {.column width="50%"} **Naming conventions:** - **Shapley values**: Original game theory method (1953) - **SHAP**: Application to machine learning (2017) - **SHAP values**: Resulting feature importance values - **`shap`**: The Python library implementation
"SHAP" became a brand name like Post-it or Band-Aid - well-established in the community and distinguishes game theory from ML application.
::: :::: ::: footer Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. NeurIPS. | Lundberg, S. M., et al. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1), 56-67. ::: ::: {.notes} Two lists. On the left, why it spread, and note that four of the six reasons are about software and community rather than about mathematics. TreeSHAP in 2020 is the one to emphasise: exact Shapley values for tree ensembles in polynomial rather than exponential time, published in Nature Machine Intelligence. That is what made SHAP usable on the gradient boosting models that win most tabular competitions, and it is why global SHAP summary plots became routine. On the right, the vocabulary, which trips people up in writing. Shapley values are the 1953 concept, SHAP is the application to a prediction, SHAP values are the outputs, and shap in lower case is the package. Students should use the terms precisely in their project reports. ::: ## Theory of Shapley Values **Who's going to pay for that taxi?** Alice, Bob, and Charlie have dinner together and share a taxi ride home. **The total cost is $51.** **The question is: How should they divide the costs fairly?** :::: {.columns .smaller} ::: {.column width="50%"} | Passengers | Cost | Note | |------------|------|------| | β
| $0 | No taxi, no costs | | {Alice} | $15 | Standard fare | | {Bob} | $25 | Luxury taxi | | {Charlie} | $38 | Lives further away | | {Alice, Bob} | $25 | Bob gets his way | | {Alice, Charlie} | $41 | Drop Alice first | | {Bob, Charlie} | $51 | Drop Bob first | | {Alice, Bob, Charlie} | $51 | Full fare | ::: ::: {.column width="50%"} **Key observations:** - Alice alone: $15 - Bob alone: $25 (insists on luxury) - Charlie alone: $38 (lives farther) - All three: $51 **Naive approach:** $51 Γ· 3 = $17 each **Problem:** Is this fair? Alice is subsidizing the others! We need a principled way to divide costs based on **marginal contributions**.
::: :::: ::: {.notes} The taxi example beautifully illustrates why Shapley values are "fair". A naive split ($17 each) is unfair - Alice subsidizes the others. Shapley values compute each person's marginal contribution across all possible orderings of joining the taxi. This ensures: (1) Total cost is exactly split, (2) Symmetric players pay equally, (3) Non-contributors pay nothing, (4) Values are additive. The same logic applies to features in ML: how much does each feature "contribute" to the prediction, measured fairly across all possible feature coalitions? ::: ## Calculating Marginal Contributions **Marginal Contribution** = Value with player β Value without player
**Example:** Charlie joining Bob's taxi = $51 - $25 = **$26** | Addition | To Coalition | Cost Before | Cost After | Marginal Contribution | |----------|--------------|-------------|------------|---------------------| | Alice | β
| $0 | $15 | **$15** | | Alice | {Bob} | $25 | $25 | **$0** | | Alice | {Charlie} | $38 | $41 | **$3** | | Alice | {Bob, Charlie} | $51 | $51 | **$0** | | Bob | β
| $0 | $25 | **$25** | | Bob | {Alice} | $15 | $25 | **$10** | | Bob | {Charlie} | $38 | $51 | **$13** | | Bob | {Alice, Charlie} | $41 | $51 | **$10** | | Charlie | β
| $0 | $38 | **$38** | | Charlie | {Alice} | $15 | $41 | **$26** | | Charlie | {Bob} | $25 | $51 | **$26** | | Charlie | {Alice, Bob} | $25 | $51 | **$26** | ::: {.notes} The mechanical step. A marginal contribution is what the total changes by when one player joins a specific group. Charlie joining Bob costs 51 minus 25, so 26. Work two rows out loud with the room. Alice joining the empty taxi contributes 15. Alice joining Bob contributes nothing at all, because Bob's luxury taxi already costs 25 and adding Alice does not change it. That zero is not a rounding artefact, it is the reason a single number cannot describe Alice's contribution: it depends entirely on who is already in the car. Count the rows: each of three players has four possible coalitions to join, so twelve numbers. In general each of p players faces two to the power p minus one coalitions, which is where the exponential cost comes from later. ::: ## Weighted Average via Permutations **How to weight these marginal contributions?** Consider all possible permutations (orderings) of passengers joining the taxi: :::: {.columns .smaller} ::: {.column width="50%"} **All 3! = 6 permutations:** 1. Alice, Bob, Charlie 2. Alice, Charlie, Bob 3. Bob, Alice, Charlie 4. Bob, Charlie, Alice 5. Charlie, Alice, Bob 6. Charlie, Bob, Alice Each permutation defines which players are "already in the taxi" when each player joins. ::: ::: {.column width="50%"} **For Alice:** - **2 times** added to β
(empty taxi) - **1 time** added to {Bob} - **1 time** added to {Charlie} - **2 times** added to {Bob, Charlie} **Weights determined by permutation frequency!** Coalition size matters: smaller coalitions get higher weight.
::: :::: ::: {.notes} Now the weighting, which is the part people skip and then cannot reconstruct. We have twelve marginal contributions and we need one number per player, so we have to average. The Shapley answer is to average over orderings rather than over coalitions. Imagine the three of them getting into the taxi one at a time. There are six possible orders. In each order, each player joins whoever is already there. Count how often Alice faces each situation: in two of the six orders she is first, so she joins the empty set; in one she joins Bob alone; in one she joins Charlie alone; and in two she joins both. That is where the weights come from. They are not arbitrary, they are the frequency with which each coalition arises if every arrival order is equally likely. Small and large coalitions get more weight than middling ones because there are fewer orderings that produce them. ::: ## Averaging Marginal Contributions **Alice's Shapley value:** $$\phi_{Alice} = \frac{1}{6}(2 \cdot \$15 + 1 \cdot \$0 + 1 \cdot \$3 + 2 \cdot \$0) = \$5.50$$ **Bob's Shapley value:** $$\phi_{Bob} = \frac{1}{6}(2 \cdot \$25 + 1 \cdot \$10 + 1 \cdot \$13 + 2 \cdot \$10) = \$15.50$$ **Charlie's Shapley value:** $$\phi_{Charlie} = \frac{1}{6}(2 \cdot \$38 + 1 \cdot \$26 + 1 \cdot \$26 + 2 \cdot \$26) = \$30.00$$ **Verification:** $5.50 + $15.50 + $30.00 = **$51.00** β
::: {.notes} Apply the weights and out come the three numbers. Alice pays 5.50, Bob 15.50, Charlie 30. Stop on the verification line, because it is the efficiency axiom made concrete: the three shares add to exactly 51, the actual fare. Nothing is left over and nothing is invented. Then compare with the naive split of 17 each from two slides ago. Alice pays a third of that, and the reason is visible in the table: she adds almost nothing to any coalition that already contains someone. Ask the room whether 5.50 feels fair. Most people say yes once they have seen the marginal contributions, and that shift is exactly the argument for using this in model explanation. ::: ## General Shapley Value Formula **Game Theory Notation:** | Term | Math | Taxi Example | |------|------|--------------| | Player | $j \in \{1, \ldots, N\}$ | Alice, Bob, Charlie | | Coalition | $S \subseteq N$ | {Alice, Bob} | | Value Function | $v(S)$ | Cost of coalition | | Shapley Value | $\phi_j$ | Fair share for player $j$ | **The Formula:** $$\phi_j = \sum_{S \subseteq N \setminus \{j\}} \frac{|S|!(N - |S| - 1)!}{N!} \left( v(S \cup \{j\}) - v(S) \right)$$ **The Shapley value is the weighted average of a player's marginal contributions to all possible coalitions.**
::: {.notes} The general formula, and every symbol in it has already appeared in the taxi example. The sum ranges over all coalitions not containing player j. The bracket is the marginal contribution of j to that coalition. The fraction in front is the permutation weight we just counted, written combinatorially: the number of orderings in which exactly the members of S precede j, divided by the total number of orderings. Do not let it look like a new idea. Point back at Alice: the fraction is how we got two sixths, one sixth, one sixth, two sixths. The sum over all subsets is also where the computational problem lives, and the next section on SHAP is largely about approximating it. ::: ## The Four Axioms Behind Shapley Values :::: {.columns} ::: {.column width="50%"} **1. Efficiency** The contributions sum to the total payout: $$\sum_{j \in N} \phi_j = v(N)$$ *In taxi: $5.50 + $15.50 + $30 = $51* β
**2. Symmetry** If players have identical marginal contributions, they get equal payouts: If $v(S \cup \{j\}) = v(S \cup \{k\})$ for all $S$, then $\phi_j = \phi_k$ *If Bob didn't need luxury taxi, he'd pay same as Alice*
::: ::: {.column width="50%"} **3. Dummy (Null Player)** Players who contribute nothing get nothing: If $v(S \cup \{j\}) = v(S)$ for all $S$, then $\phi_j = 0$ *If Dora the dog rides free, her share = $0*
**4. Additivity** For two games with value functions $v_1$ and $v_2$: $$\phi_{j,v_1+v_2} = \phi_{j,v_1} + \phi_{j,v_2}$$ *Can split taxi + ice cream costs separately, then add*
::: :::: **These four axioms uniquely determine the Shapley value formula** (Shapley, 1953)
::: {.notes} Why anyone should accept this particular average rather than some other. Shapley's argument is axiomatic: state four properties any fair allocation ought to have, and show that exactly one allocation satisfies all four. Efficiency, the shares sum to the total. Symmetry, two players who contribute identically to every coalition get the same share, so the answer does not depend on labels or ordering. Dummy, a player who never changes any coalition's value gets zero. Additivity, if you run two independent games, the fair share in the combined game is the sum of the two shares. The uniqueness is the punchline, and it is the strongest claim any method in this lecture can make. LIME has no comparable result. Flag that additivity looks like the dullest axiom but is the one that makes TreeSHAP possible: compute per tree, then add. Worth one honest caveat: uniqueness is conditional on accepting the four axioms, and the critics later in the lecture attack whether they are the right four for feature attribution. ::: ## From Shapley Values to SHAP **A Machine Learning Example** You have trained a model $f$ to predict apartment prices. :::: {.columns} ::: {.column width="50%"} **For a specific apartment $x^{(i)}$:** - Area: 50 mΒ² (538 sq ft) - Floor: 2nd floor - Park: Nearby - Cats: Banned ::: ::: {.column width="50%"} **Predictions:** - $f(x^{(i)}) = \text{β¬}300,000$ (this apartment) - $\mathbb{E}[f(X)] = \text{β¬}310,000$ (average) - **Difference: -β¬10,000** ::: :::: **Goal:** Explain how each feature value contributed to the -β¬10,000 difference from average.
::: {.notes} The translation. A trained model f, one apartment, and a prediction of 300,000 against an average prediction of 310,000 across all apartments. The quantity to be explained is the difference, minus 10,000. Not the prediction itself, the gap between this prediction and the average. That reframing is what turns explanation into a division problem: we have a fixed amount of deviation and we need to hand it out among the features. Ask which feature they would blame for the 10,000 before revealing anything. Fifty square metres is small, the second floor is unremarkable, the park is a plus, and banning cats is a minus. The point is that reasonable people disagree, which is why we want a principled rule. ::: ## Viewing a Prediction as a Coalitional Game **Key insight:** Each feature value is a "player" in a game where the "payout" is the prediction. | Game Theory Concept | Machine Learning Translation | Notation | |---------------------|------------------------------|----------| | Player | Feature index | $j$ | | Coalition | Set of features | $S \subseteq \{1, \ldots, p\}$ | | Not in coalition | Features not in $S$ | $C = \{1, \ldots, p\} \setminus S$ | | Coalition size | Number of features in $S$ | $\|S\|$ | | Total players | Number of features | $p$ | | **Total payout** | **Prediction - Average** | $f(x^{(i)}) - \mathbb{E}[f(X)]$ | | **Value function** | **Prediction for coalition** | $v_{f,x^{(i)}}(S)$ | | **Shapley value** | **SHAP value (contribution)** | $\phi_j^{(i)}$ | We translate concepts from game theory to machine learning predictions β **SHAP**
::: {.notes} The dictionary between the two halves of this section, and it is worth walking every row rather than flashing it. A player becomes a feature. A coalition becomes a subset of features. The payout becomes the prediction minus the average prediction, which is the quantity we just agreed to divide. The value function becomes the model's output when only the features in S are known. That last row is the one with a subtlety, and the next slide is entirely about it. A trained model does not accept partial input. You cannot ask it for a prediction using only the park and the floor. So the translation is not free, and how you fill in the missing features is where all the later controversy comes from. ::: ## The SHAP Value Function **How do we handle "absent" features in a coalition?** $$v_{f,x^{(i)}}(S) = \int f(x_S^{(i)} \cup X_C) d\mathbb{P}_{X_C} - \mathbb{E}[f(X)]$$ :::: {.columns .smaller} ::: {.column width="50%"} **Key components:** 1. $x_S^{(i)}$: Known feature values (in coalition $S$) 2. $X_C$: Unknown features (random variables) 3. $\int \ldots d\mathbb{P}_{X_C}$: **Marginalization** - integrate over distribution 4. $-\mathbb{E}[f(X)]$: Ensures $v(\emptyset) = 0$ **Marginalization:** Treat absent features as random variables, weight predictions by their likelihood ::: ::: {.column width="50%"} **Apartment example:** For coalition $S = \{\text{park}, \text{floor}\}$: $$v(\{park, floor\}) = \int f(x_{park}, X_{cat}, X_{area}, x_{floor}) d\mathbb{P}_{cat,area}$$ - $x_{park} = \text{nearby}$ (known) - $x_{floor} = 2$ (known) - $X_{cat}$, $X_{area}$ (random - integrated over) **Present** features input directly; **absent** features marginalized out
::: :::: ::: {.notes} Here is the answer to that problem, and it is the technical heart of SHAP. To evaluate a coalition, plug in the known feature values and integrate the model's output over the distribution of the unknown ones. Absent does not mean zero or missing, it means averaged out. Read the four components on the left. The subtraction of the expected prediction at the end is bookkeeping: it makes the empty coalition worth zero, which is what the game requires. Then the honest part. The integral is over the marginal distribution of the absent features, treating them as independent of the present ones. For the apartment that means averaging over apartment sizes drawn without regard to the floor or the neighbourhood, which generates combinations that do not exist. This is the same independence assumption we criticised in PDP, reappearing in the method with the strongest theory. The correlation slides later in the lecture come back to precisely this integral, so plant the flag here. ::: ## SHAP Visualization: Force Plot **Visualizing the Efficiency Axiom** **Base Value** (E[f(x)]) = $300K **Output Value** f(x) = $450K
::: {.notes} This visualization directly maps the Efficiency Axiom from the theory section: the contributions (Οα΅’) of all features (red/blue blocks) must sum up exactly from the Base Value (average prediction) to the Final Output (instance prediction). This is a powerful, rigorous local explanation. Red blocks push the prediction higher; blue blocks push it lower. The horizontal layout shows the additive nature - each feature's contribution stacks to reach the final prediction. This is fundamentally different from LIME which uses sampling; SHAP provides exact attributions based on game theory. ::: ## Putting It All Together: The SHAP Formula **Combining all terms into the SHAP equation:** $$\phi_j^{(i)} = \sum_{S \subseteq \{1,\ldots,p\} \setminus j} \frac{|S|! (p - |S| - 1)!}{p!} \cdot \left( \int f(x_{S \cup j}^{(i)} \cup X_{C \setminus j}) d\mathbb{P}_{X_{C \setminus j}} - \int f(x_S^{(i)} \cup X_C) d\mathbb{P}_{X_C} \right)$$ β¬β¬β¬β¬β¬β¬ **+$80K** Area=85mΒ² (high pushes price up)
β¬β¬β¬β¬ **+$45K** Location=Downtown
β¬β¬β¬ **+$30K** Year=2020 (new)
β¬β¬ **-$15K** Cat-banned=True
β¬ **-$10K** Floor=1 (ground floor)
**Sum: $300K + $80K + $45K + $30K - $15K - $10K = $450K β**β¬β¬β¬β¬ **+$45K** Location=Downtown
β¬β¬β¬ **+$30K** Year=2020 (new)
β¬β¬ **-$15K** Cat-banned=True
β¬ **-$10K** Floor=1 (ground floor)
**In words:** The SHAP value $\phi_j^{(i)}$ of feature $j$ is the **weighted average marginal contribution** of feature value $x_j^{(i)}$ across all possible coalitions of features.
**This is Shapley values with an ML-specific value function!** ::: {.notes} This formula looks complex but it's the same Shapley formula we saw before, just with the ML-specific value function plugged in. The integral handles absent features through marginalization. In practice, we can't compute this exactly (we don't know the true distribution), so we estimate it. ::: ## Interpreting SHAP Values Through Axioms **Since SHAP values are Shapley values, they satisfy the four axioms:** :::: {.columns .smaller} ::: {.column width="50%"} **1. Efficiency** $$\sum_{j=1}^p \phi_j^{(i)} = f(x^{(i)}) - \mathbb{E}[f(X)]$$ **Implications:** - SHAP values sum to deviation from average - Attributions on same scale as output - Unlike gradients, which don't sum to prediction - Enables force plot visualizations **2. Symmetry** Features with equal contributions get equal SHAP values **Implications:** - Feature order doesn't matter - Essential for feature importance ranking - Unlike breakdown method which depends on order ::: ::: {.column width="50%"} **3. Dummy** Unused features get SHAP value of zero **Implications:** - If feature doesn't affect prediction β $\phi_j = 0$ - Makes sense for sparse models - E.g., Lasso with $\beta_j = 0$ β $\phi_j^{(i)} = 0$ for all $i$ **4. Additivity** For ensemble models: $\phi_j(v_1 + v_2) = \phi_j(v_1) + \phi_j(v_2)$ **Implications:** - For random forest: compute SHAP per tree, then average - For additive ensembles: sum individual SHAP values - Enables TreeSHAP efficiency ::: :::: ::: footer Ε trumbelj & Kononenko (2010, 2014); Lundberg & Lee (2017) ::: ::: {.notes} Now reap the axioms. Each one buys something concrete for visualization, which is why this slide matters for this course rather than for a statistics course. Efficiency means the attributions sum to the prediction minus the baseline, so they live on the same scale as the output and can be stacked end to end. That is what makes the force plot and the waterfall plot possible and honest. Gradients do not have this property, which is why saliency maps cannot be summed to anything meaningful. Symmetry means the ranking in a summary plot is well defined and does not depend on the order features happen to appear in the data frame. Contrast that with breakdown methods, which do depend on order. Dummy means an unused feature gets exactly zero, so sparsity in the model shows up as sparsity in the explanation. Additivity is what lets you compute SHAP per tree in a forest and add. The through line: every visualization in the SHAP library is licensed by one of these four properties. Design and theory are not separate here. ::: ## Key Differences: LIME vs. SHAP :::: {.columns .smaller} ::: {.column width="50%"} **LIME** - Heuristic local approximation - Sparse explanations (few features) - Faster to compute - Good for lay audiences - No theoretical guarantees Best when: You want **simple**, **sparse** explanations with only most important features highlighted
::: ::: {.column width="50%"} **SHAP** - Rigorous game-theoretic foundation - Always uses **all features** - Computationally expensive - Satisfies fairness axioms - Unique solution Best when: You need **complete**, **theoretically grounded** feature attributions with fairness guarantees
::: :::: **Important:** SHAP values measure contribution to deviation from average, NOT the effect of removing features from training. ::: footer Source: Molnar, C. (2024). Interpretable Machine Learning. ::: ::: {.notes} The comparison the class will actually use when choosing a tool for their project. LIME is a heuristic, gives you a handful of features, is fast, and is easy to hand to a non-expert. It has no guarantee that two runs agree or that the numbers mean anything precise. SHAP has a uniqueness theorem, attributes to every feature, and costs a great deal more compute. The density is a real cost in practice: twenty-three features means twenty-three bars, and nobody reads twenty-three bars. The line at the bottom is the most commonly misread thing in this lecture, so say it slowly. A SHAP value is not the answer to what happens if I delete this feature and retrain. It is the contribution of this feature's value to this prediction's deviation from the average, under a fixed trained model. People write the first sentence in papers when they mean the second. ::: ## SHAP Visualization: Global Feature Analysis (Summary Plot) **From Local to Global: Understanding Feature Importance Across All Predictions** :::: {.columns .smaller} ::: {.column width="60%"} **Feature Importance** (Y-axis) β
::: ::: {.column width="40%"} **Dual Interpretation:** 1. **Feature Importance** (Y-Axis vertical spread) - Wider spread = more important - Area has largest impact 2. **Direction of Effect** (Color) - Red (high) on right = positive impact - Blue (low) on left = negative impact **Example Reading:** - High area (red) β strong positive SHAP - Low area (blue) β negative SHAP - Clear correlation! ::: :::: ::: {.notes} Explain the dual interpretation of this plot: (1) **Feature Importance (Y-Axis)**: Vertical spread/density of points indicates a feature's overall importance (like traditional feature importance). Features at the top matter more globally across all predictions. (2) **Correlation (Color)**: The color coding (Red for high feature value, Blue for low) shows the relationship. For example, 'Red points on the far right' means high feature values are associated with high positive impact on the prediction. Area shows both high importance AND clear directionality - bigger apartments cost more. Compare this to cat-banned which has mixed effects and lower importance. This plot aggregates SHAP values across many instances to give a global view. ::: ## SHAP: Pros and Cons ::: {.fragment} **Pros** - Fairly distributed feature importance to a prediction - Contrastive explanations (can compare an instance to a subset or even to a single data point) - Solid theory ::: ::: {.fragment} **Cons** - A lot of computing time - Not sparse explanations (every feature is important) ::: ::: {.notes} The SHAP trade, and it mirrors the LIME one deliberately, so put them side by side in the room's head. Fair distribution with a theorem behind it, contrastive by construction because everything is measured against a baseline you choose, and a solid theory that unifies several earlier methods. Against that: computation, exponential in the number of features unless you have a model-specific shortcut like TreeSHAP. And dense explanations, because the axioms require every feature to receive a value, including the ones worth 0.001. Sparsity has to be imposed afterwards by the visualization, which is a design decision rather than a property of the method, and that is our problem as visualization people rather than theirs as theorists. ::: ## SHAP: Force Plots for Two Patients  ::: {.notes} Two force plots from Molnar's book, explaining a cervical cancer risk model for two different women. Same model, same features, two very different explanations, which is the case for local methods in one image. Read the top one. The baseline, the average predicted risk across the dataset, is 0.066. This woman's predicted risk is 0.06, so slightly below average. The red arrows push right, toward higher risk: an STD diagnosis, six years with an IUD, two recorded STDs. The blue arrows push left: seven years of hormonal contraceptives and being 42. The two sides nearly cancel, which is why the final number is so close to the baseline. A pure importance ranking would have told you these features matter; only the force plot tells you they matter in opposite directions and almost exactly offset. The bottom one runs from the same base value out to 0.71, and it is almost all red. Age 51 and 34 years of smoking are the two widest bands. Two things to point at as visualization. The widths are the SHAP values and they tile the distance from base value to output exactly, which is the efficiency axiom drawn to scale. And features with small contributions get thin slivers with no room for a label, which is the density problem from the previous slide showing up in the rendering. ::: ## SHAP limitations :::: {.columns} ::: {.column width="50%"} {height="760" fig-align="center"} ::: ::: {.column width="50%"} **Key Limitations:** - **Computational Cost**: Exponential complexity O(2^n) for exact calculation - **Feature Independence Assumption**: Unrealistic in correlated datasets - **Interpretation Challenges**: Values represent marginal contributions, not causal effects - **Instability**: Small perturbations can lead to different explanations ::: :::: ::: footer Kumar, I. E., et al. (2020). [*Problems with Shapley-value-based explanations as feature importance measures*](http://proceedings.mlr.press/v119/kumar20e/kumar20e.pdf). ICML. ::: ::: {.notes} Despite SHAP's theoretical elegance, it has important limitations that practitioners must understand. The Kumar et al. paper provides crucial critiques: (1) SHAP assumes features can be independently varied, but real features are often correlated (e.g., house size and bedrooms), (2) SHAP values show associative, not causal relationships - they tell you "what" contributes but not "why", (3) Computational cost grows exponentially with features (though approximations like TreeSHAP and KernelSHAP help), (4) Explanations can be unstable under small data perturbations. Bottom line: SHAP is a powerful tool but not a magic bullet. Always validate explanations against domain knowledge. ::: ## SHAP Limitations: Not Always Human-Friendly :::: {.columns .smaller} ::: {.column width="50%"} **What Makes Explanations Human-Friendly?** *(Miller 2019)* Human-friendly explanations should be: 1. **Concise**: Focus on key factors (2-3 features) 2. **Contrastive**: Compare to specific alternatives 3. **Abnormal**: Highlight unusual causes **But SHAP:** - β **Not concise**: Always uses ALL features - β **Not strongly contrastive**: Averages over many coalitions, diluting contrast - β **Doesn't highlight abnormal**: Treats all features equally **Area (mΒ²)** ββββββββββββββ
**Location** ββββββββββ
**Year Built** βββββββββ
**Cat-banned** βββββ
**Floor** βββββ
β Negative Impact **SHAP Value** Positive Impact β β Blue = Low feature value β Red = High feature value**Location** ββββββββββ
**Year Built** βββββββββ
**Cat-banned** βββββ
**Floor** βββββ
**β οΈ Warning**: Don't present SHAP values to end-users as straightforward information. They require explanation and training to understand properly.
::: ::: {.column width="50%"} **SHAP Doesn't Enable User Actions** **Example: Corn Yield Prediction** - Model predicts low yield for a field - SHAP: "Fertilizer use had +0.3 positive impact" - Farmer asks: **"Should I add more fertilizer?"** - SHAP: π€· **Cannot answer!** SHAP explains the **current state** vs. average, not **how to change** outcomes.
**For Actionable Recommendations:** - Use **counterfactual explanations**: "If fertilizer increased by 20%, yield would increase by 5%" - Design models with **causal relationships** - Use **representative training data** ::: :::: ::: footer Miller, T. (2019). [*Explanation in artificial intelligence: Insights from the social sciences*](https://www.sciencedirect.com/science/article/pii/S0004370218305988). Artificial Intelligence, 267, 1-38. ::: ::: {.notes} This slide addresses critical usability limitations of SHAP. Research on human cognition shows that good explanations are concise (2-3 key factors), contrastive (compared to specific alternatives), and focus on abnormal causes. SHAP fails on all three: it always attributes to ALL features (not concise), averages over many coalitions which dilutes contrastiveness, and treats all features equally without highlighting what's unusual. The corn yield example illustrates another major limitation: SHAP doesn't enable action. It tells you fertilizer had a positive contribution compared to the average field, but doesn't answer "should I add more?" This is because SHAP measures deviation from background data, not causal effects of changes. For actionable insights, you need counterfactual explanations that explicitly model "what if" scenarios. Bottom line: SHAP is a technical tool for ML practitioners, not an end-user explanation method. ::: ## SHAP Limitations: Misinterpretation and Adversarial Risks :::: {.columns .smaller} ::: {.column width="50%"} **Common Misinterpretations** **β Wrong Interpretation:** "SHAP value = difference in prediction if feature removed from training" **β Correct Interpretation:** "SHAP value = contribution to deviation from mean prediction, given current feature values"
**SHAP Is NOT:** - β A **surrogate model** (unlike LIME) - Cannot predict effects of feature changes - Example: "If salary increases β¬300, credit score increases 5 points" β SHAP can't do this! - β **Training-data independent** - Requires background data for computation - Privacy/access concerns in production ::: ::: {.column width="50%"} **Adversarial Risk: You Can Fool SHAP** *(Slack et al. 2020)* Unscrupulous data scientists can create **intentionally misleading** SHAP explanations to conceal biases: **Attack Scenario:** 1. Train a **biased model** (e.g., using race for loan decisions) 2. Add **decoy features** that appear important to SHAP 3. Manipulate **background data** distribution 4. Result: SHAP shows "fair" explanations for discriminatory predictions!
**Philosophical Issue: No Consensus** - What should "feature importance" mean? - Are SHAP axioms the "right" definition? - Process is backwards: method β interpretation "We use a mathematically coherent method that generates an 'explanation' and then attempt to decipher its meaning." - Molnar (2024)
::: :::: ::: footer Slack, D., et al. (2020). [*Fooling LIME and SHAP*](https://arxiv.org/abs/1911.02508). AIES. | Molnar, C. (2024). *Interpretable Machine Learning*. ::: ::: {.notes} This slide covers critical risks in deploying SHAP. FIRST, misinterpretation is common and dangerous. SHAP values are NOT the effect of removing features from training - they're contributions to deviation from the mean prediction given current values. This subtle distinction matters for correct interpretation. SHAP also isn't a surrogate model like LIME - you can't use it to simulate "what if" changes. SECOND, adversarial risk: Slack et al. demonstrated that malicious data scientists can intentionally create misleading SHAP explanations. They can add decoy features, manipulate background data, and train models that appear fair to SHAP while being discriminatory. This is particularly dangerous for auditing AI systems - you can't blindly trust SHAP explanations without deeper investigation. THIRD, philosophical issue: there's no consensus in the field about what "feature importance" should mean. SHAP defines it through axioms, but are these the right axioms? For linear/additive models, SHAP aligns with coefficients, which validates the approach. But for complex interactions, interpretation becomes murky. The field often works backwards: we apply mathematically elegant methods and then try to understand what they mean, rather than defining what we want first. ::: ## Deep Dive: The Correlation Problem **Why Correlated Features Break SHAP's Independence Assumption** :::: {.columns .smaller} ::: {.column width="55%"} **The Core Issue:** SHAP simulates feature absence by replacing them with **sampled values from background data** (marginal distribution) **Problems with Correlated Features:** 1. **Unrealistic data points**: Combining features independently creates impossible combinations - Example: 2-meter tall person weighing 10 kg - Example: Rain without clouds 2. **Extrapolation**: Model evaluated on data it never saw during training 3. **Misleading explanations**: SHAP values based on unrealistic scenarios ::: ::: {.column width="45%"} **Philosophical Question:** If two features share information (e.g., rainfall β cloudiness), can we truly isolate the effect of one while ignoring the other?
**The Trade-off:** - **Marginal sampling**: True to the **model** - **Conditional sampling**: True to the **data** Your choice depends on interpretation goals! ::: :::: ::: footer Molnar, C. (2024). *Interpretable Machine Learning*. Chapter 11: The Correlation Problem. ::: ::: {.notes} The correlation problem is subtle but critical. SHAP's marginal sampling strategy assumes features can be varied independently, but this assumption fails when features are correlated. When we replace a feature value with one sampled from the background data without considering correlations, we create data points that may never occur in reality. For example, if apartment size and number of rooms are correlated, SHAP might evaluate a 20mΒ² apartment with 5 bedrooms - physically impossible. The model was never trained on such data and might produce unreliable predictions. This raises a philosophical question: what does "feature importance" even mean when features share information? The rainfall/cloudiness example illustrates this perfectly - if it rained, it must have been cloudy, so how do we isolate rain's effect? ::: ## Visualizing the Correlation Problem ```{python} #| echo: true #| output: true #| fig-width: 14 #| fig-height: 6 import numpy as np import matplotlib.pyplot as plt np.random.seed(42) # Create two highly correlated features (Ο = 0.9) p = 0.9 mean = [0, 0] cov = [[1, p], [p, 1]] n = 100 x1, x2 = np.random.multivariate_normal(mean, cov, n).T # Point to explain point = (-1.7, -1.7) m = 15 # number of samples # Marginal sampling: sample x2 independently (ignores correlation) x2_marg = np.random.choice(x2, size=m) # Conditional sampling: sample x2 given x1 (respects correlation) x2_cond = np.random.normal(loc=p*point[0], scale=np.sqrt(1-p**2), size=m) # Create side-by-side plots fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6)) # Left: Marginal sampling (SHAP default) ax1.scatter(x1, x2, color='black', alpha=0.2, s=50, label='Training data') ax1.scatter(np.repeat(point[0], m), x2_marg, color='blue', s=100, alpha=0.7, label='Marginal samples') ax1.scatter(point[0], point[1], color='red', s=200, marker='*', label='Point to explain', zorder=10) ax1.set_xlabel('Feature Xβ', fontsize=14) ax1.set_ylabel('Feature Xβ', fontsize=14) ax1.set_title('Marginal Sampling (SHAP Default)\nβ οΈ Creates unrealistic data points', fontsize=16, fontweight='bold') ax1.legend(fontsize=12) ax1.grid(alpha=0.3) ax1.set_xlim(-2.5, 2.5) ax1.set_ylim(-2.5, 2.5) # Right: Conditional sampling ax2.scatter(x1, x2, color='black', alpha=0.2, s=50, label='Training data') ax2.scatter(np.repeat(point[0], m), x2_cond, color='green', s=100, alpha=0.7, label='Conditional samples') ax2.scatter(point[0], point[1], color='red', s=200, marker='*', label='Point to explain', zorder=10) ax2.set_xlabel('Feature Xβ', fontsize=14) ax2.set_ylabel('Feature Xβ', fontsize=14) ax2.set_title('Conditional Sampling P(Xβ|Xβ)\nβ Respects feature correlation', fontsize=16, fontweight='bold') ax2.legend(fontsize=12) ax2.grid(alpha=0.3) ax2.set_xlim(-2.5, 2.5) ax2.set_ylim(-2.5, 2.5) plt.tight_layout() plt.show() ``` ::: {.notes} This visualization demonstrates the correlation problem concretely. We simulate two features with high correlation (Ο=0.9) - notice the diagonal cloud of training data points. The red star is the point we want to explain at (-1.7, -1.7). LEFT PLOT shows marginal sampling (SHAP's default): when computing SHAP for Xβ, we fix it at -1.7 and sample Xβ independently from the background data (blue points). Notice these blue points spread across the entire range of Xβ, creating data points far outside the training distribution - these are unrealistic! RIGHT PLOT shows conditional sampling: we sample Xβ given Xβ=-1.7, respecting their correlation. Green points cluster near the red star, staying within the training distribution. Marginal sampling evaluates the model on out-of-distribution data, potentially yielding misleading SHAP values. Conditional sampling avoids this but changes the interpretation (no longer pure Shapley values). ::: ## Solutions to the Correlation Problem :::: {.columns} ::: {.column width="50%"} **Solution 1: Reduce Correlation in Model** - **Feature selection**: Remove redundant correlated features - **Feature engineering**: Transform features to decorrelate - Example: "rent" + "rooms" β "rent per mΒ²" - **Combine features**: Daily rainfall vs. morning/afternoon rain - **Dimensionality reduction**: PCA (but loses interpretability) **β Benefit**: Improves both model performance AND interpretability
::: ::: {.column width="50%"} **Solution 2: Use Conditional Sampling** Available in SHAP library: ```python # Linear models explainer = shap.LinearExplainer( model, X_train, feature_perturbation='correlation_dependent' ) # Tree models explainer = shap.TreeExplainer( model, feature_perturbation='tree_path_dependent' ) ``` **β οΈ Warning**: Changes the value function - no longer pure Shapley values! Use when goal is to understand **data** rather than audit **model**.
::: :::: ::: footer Aas, K., et al. (2021). [*Explaining individual predictions when features are dependent*](https://www.jmlr.org/papers/volume22/20-1223/20-1223.pdf). JMLR, 22(213), 1-34. ::: ::: {.notes} There are two main approaches to handle the correlation problem. SOLUTION 1: Prevent the problem by reducing feature correlations during model development. This is the cleanest approach - use feature selection methods that identify and remove correlated features, apply feature engineering to decorrelate (e.g., instead of absolute rent and size, use rent per square meter), or combine redundant features. This not only helps SHAP but often improves model generalization. SOLUTION 2: Use conditional sampling when computing SHAP values. The SHAP library provides feature_perturbation options for this. However, understand the trade-off: conditional sampling changes the underlying game-theoretic foundation. The resulting values aren't true Shapley values anymore, and interpretation changes. Marginal sampling (default) is "true to the model" - it shows what the model learned, including potential artifacts from correlation. Conditional sampling is "true to the data" - it respects real-world distributions. Choose based on your goal: model auditing β marginal; data understanding β conditional. ::: ## SHAP in Practice: Wine Quality Prediction **A Complete Example with Linear Models** :::: {.columns .smaller} ::: {.column width="50%"} **Dataset: UCI Wine Quality** - White wine physicochemical properties - Target: Quality score (0-10) - 11 features: alcohol, acidity, sugar, etc. - 4,898 samples **Why Linear Models?** - Inherently interpretable (coefficients) - SHAP values simplify elegantly - Perfect for teaching SHAP concepts ::: ::: {.column width="50%"} **Learning Objectives:** 1. Load and explore wine dataset 2. Train linear regression model 3. Compute SHAP values efficiently 4. Create three key visualizations: - Waterfall plot (local) - Summary plot (global) - Dependence plot (relationships) 5. Verify SHAP = coefficients ::: :::: ::: footer Cortez, P., et al. (2009). Modeling wine preferences by data mining from physicochemical properties. *Decision Support Systems*, 47(4), 547-553. ::: ::: {.notes} We'll now work through a complete SHAP example using the UCI Wine Quality dataset. This example is ideal for teaching because: (1) Linear models are inherently interpretable, making it easier to understand what SHAP is computing, (2) SHAP has efficient exact computation for linear models (no approximation needed), (3) We can verify our understanding by comparing SHAP values to model coefficients. The dataset predicts wine quality from physicochemical properties - a realistic regression task with clear feature interpretations. ::: ## Loading the Wine Quality Dataset ```{python} #| echo: true #| output: true import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression import shap # Load wine quality dataset url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv' wine = pd.read_csv(url, sep=";") print(f"Dataset shape: {wine.shape}") print(f"\nFeatures: {list(wine.columns[:-1])}") print(f"\nQuality distribution:\n{wine['quality'].value_counts().sort_index()}") print(f"\nFirst few rows:\n{wine.head(3)}") ``` ::: {.notes} We load the white wine quality dataset directly from UCI. Notice the dataset has 4,898 wines with 11 physicochemical features (fixed acidity, volatile acidity, citric acid, residual sugar, chlorides, free sulfur dioxide, total sulfur dioxide, density, pH, sulphates, alcohol) and one target variable (quality). The quality scores range from 3 to 9, with most wines scoring 5-6. This is a real-world regression problem with interpretable features. ::: ## Training a Linear Regression Model ```{python} #| echo: true #| output: true # Prepare features and target X = wine.drop('quality', axis=1) y = wine['quality'] # Split data X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Train linear regression model model = LinearRegression() model.fit(X_train, y_train) # Evaluate model train_score = model.score(X_train, y_train) test_score = model.score(X_test, y_test) print(f"Train RΒ² score: {train_score:.3f}") print(f"Test RΒ² score: {test_score:.3f}") print(f"\nModel coefficients:") for feature, coef in zip(X.columns, model.coef_): print(f" {feature:25s}: {coef:+.4f}") print(f" Intercept: {model.intercept_:.4f}") ``` ::: {.notes} We split the data 80/20 and train a simple linear regression model. The RΒ² scores (~0.28) indicate moderate predictive power - typical for wine quality prediction. The coefficients show interpretable relationships: alcohol has the strongest positive effect (+0.28), volatile acidity has a strong negative effect (-1.86), and density has a large negative coefficient (-32.67). These coefficients represent the linear relationship between each feature and quality. Soon we'll see how SHAP values relate to these coefficients. ::: ## Computing SHAP Values with LinearExplainer ```{python} #| echo: true #| output: true # Create SHAP explainer for linear models explainer = shap.LinearExplainer(model, X_train) # Compute SHAP values for test set shap_values = explainer(X_test) print(f"SHAP values shape: {shap_values.values.shape}") print(f"Base value (average prediction): {shap_values.base_values[0]:.3f}") print(f"\nSHAP values for first test instance:") for feature, value in zip(X.columns, shap_values.values[0]): print(f" {feature:25s}: {value:+.4f}") print(f"\nPrediction for first instance: {model.predict(X_test.iloc[[0]])[0]:.3f}") print(f"Sum: base_value + sum(shap_values) = {shap_values.base_values[0] + shap_values.values[0].sum():.3f}") ``` ::: {.notes} We use SHAP's LinearExplainer, which computes exact SHAP values efficiently for linear models. The explainer is initialized with the model and training data (to compute the base value). For each test instance, SHAP returns: (1) base_value: the average prediction across training data, (2) shap_values: the contribution of each feature to deviation from base_value. Notice that base_value + sum(shap_values) equals the model's prediction - this is the additivity property. For linear models, SHAP values have a simple form: shap_i = coef_i Γ (x_i - mean(x_i)). This makes linear models perfect for understanding SHAP. ::: ## SHAP Waterfall Plot: Local Explanation ```{python} #| echo: true #| output: true #| fig-width: 12 #| fig-height: 8 # Select an interesting wine (high quality prediction) idx = np.argmax(model.predict(X_test)) print(f"Selected wine index: {idx}") print(f"Predicted quality: {model.predict(X_test.iloc[[idx]])[0]:.2f}") print(f"Actual quality: {y_test.iloc[idx]}") # Create waterfall plot shap.plots.waterfall(shap_values[idx]) ``` ::: {.notes} The waterfall plot shows how SHAP builds up from the base value (average prediction ~5.88) to the final prediction for a specific wine. Read from bottom to top: we start at E[f(X)], then each feature either pushes the prediction higher (red, right) or lower (blue, left). For this high-quality wine, alcohol content adds the most positive contribution, followed by density and volatile acidity. The sum of all contributions equals f(x) - E[f(X)]. This visualization answers: "Why did the model predict THIS value for THIS wine?" It's perfect for explaining individual predictions to stakeholders. ::: ## SHAP Summary Plot: Global Feature Importance ```{python} #| echo: true #| output: true #| fig-width: 12 #| fig-height: 8 # Create summary plot (beeswarm) shap.plots.beeswarm(shap_values) ``` ::: {.notes} The summary plot (also called beeswarm plot) aggregates SHAP values across ALL test instances to show global patterns. The Y-axis shows features ranked by importance (vertical spread). The X-axis shows SHAP values (impact on prediction). Color indicates feature value (red = high, blue = low). Key insights: (1) Alcohol is most important - high values (red) consistently push predictions higher (right), (2) Volatile acidity is second most important - high values push predictions lower (left), (3) Density shows strong negative relationship when high. Compare this to feature importance from tree models - SHAP provides the same ranking but with directionality. This plot answers: "Which features matter most GLOBALLY across all wines?" ::: ## SHAP Dependence Plot: Feature Relationships ```{python} #| echo: true #| output: true #| fig-width: 12 #| fig-height: 8 # Create dependence plot for alcohol shap.plots.scatter(shap_values[:, 'alcohol']) ``` ::: {.notes} The dependence plot shows the relationship between feature value (X-axis) and SHAP value (Y-axis) for a specific feature. For alcohol, we see a clear LINEAR relationship - as alcohol increases, SHAP value increases proportionally. This makes perfect sense for a linear model! The color indicates interaction effects with another feature (automatically selected by SHAP). The vertical spread at each X value shows variance across instances. This plot answers: "HOW does this feature affect predictions?" For linear models, you'll always see straight lines. For non-linear models (trees, neural networks), these plots reveal non-linear patterns, thresholds, and interactions. ::: ## Verifying SHAP Values for Linear Models ```{python} #| echo: true #| output: true # For linear models: SHAP = coefficient Γ (x - mean(x)) # Let's verify this relationship print("Verification: SHAP values = coefficient Γ (feature - mean)") print("\nFor the first test instance:") print(f"{'Feature':<25} {'Coefficient':>12} {'Value':>10} {'Mean':>10} {'(Val-Mean)':>12} {'SHAP Value':>12} {'Match?':>8}") print("-" * 105) for i, feature in enumerate(X.columns): coef = model.coef_[i] value = X_test.iloc[0][feature] mean = X_train[feature].mean() expected_shap = coef * (value - mean) actual_shap = shap_values.values[0][i] match = "β" if np.isclose(expected_shap, actual_shap, rtol=1e-3) else "β" print(f"{feature:<25} {coef:>12.4f} {value:>10.2f} {mean:>10.2f} {value-mean:>12.2f} {actual_shap:>12.4f} {match:>8}") ``` ::: {.notes} This verification slide demonstrates the elegant simplicity of SHAP for linear models. For each feature, SHAP value equals: coefficient Γ (feature_value - mean_feature_value). This shows that: (1) SHAP respects the linear model's coefficients (direction and magnitude), (2) SHAP measures deviation from the mean (why predictions differ from average), (3) Features with large coefficients AND large deviations have high SHAP values. This is why linear models are perfect for teaching SHAP - you can directly see what SHAP computes. For non-linear models, SHAP still satisfies the same axioms, but the computation is more complex. The key insight: SHAP provides feature attributions that respect the model's actual behavior. ::: ## Comparative Trade-offs: Choosing the Right XAI Tool | **Criterion** | **PDP***Partial Dependence Plot* | **LIME**
*Local Interpretable Model-agnostic Explanations* | **SHAP**
*SHapley Additive exPlanations* | |--------------|---------|----------|----------| | **Scope** | Global (feature-level) | Local (instance-level) | Both (local + global) | | **Primary Visualization** | Line plot (2D curve) | Bar chart (sparse) | Force plot, Summary plot | | **Interpretability** | β β β β β Intuitive | β β β β β Easy to understand | β β β ββ Requires training | | **Computational Cost** | β β β β β Moderate | β β β ββ Fast sampling | β β βββ Expensive | | **Theoretical Foundation** | Statistical (marginal effects) | Heuristic (local approximation) | Game theory (Shapley values) | | **Feature Coverage** | One at a time | Sparse (selected) | All features (dense) | | **Best For** | Understanding global trends | Quick local explanations | Rigorous feature attribution | | **Key Limitation** | Assumes independence | Unstable (sampling) | Computational cost | ::: {.notes} This table provides a decision guide for practitioners. **Choose PDP** when you need to understand how a feature affects predictions globally - great for presentations to stakeholders. **Choose LIME** when you need fast, interpretable explanations for individual predictions - ideal for production systems where speed matters. **Choose SHAP** when you need theoretically grounded attributions and can afford the computational cost - best for research or high-stakes decisions. In practice, use multiple methods: PDP for global insights, LIME for quick checks, SHAP for final validation. Remember: no method is perfect - always validate against domain knowledge and use visualizations to communicate effectively. ::: ## Next Week (Oct 13): Clustering, and the Project Discussion :::: {.columns} ::: {.column width="50%"} ### Topics - Unsupervised learning, k-means and DBSCAN - The manifold hypothesis and intrinsic dimensionality - PCA, SVD and Local Linear Embedding - **Default project overview** - **Project discussion:** proposal guidelines, timeline, evaluation ### Lab - Interactive clustering and cluster exploration ::: ::: {.column width="50%"} ### Required reading - [Wolfram Clustering Tutorial](https://www.wolfram.com/language/introduction-machine-learning/clustering/) ### Optional - [A Tutorial on Principal Component Analysis](https://arxiv.org/abs/1404.1100) - Shlens, 2014 - [Scikit-learn PCA User Guide](https://scikit-learn.org/stable/modules/decomposition.html#pca) ::: :::: ::: {.notes} Where this lecture sits. Two weeks ago we asked whether a model is any good, last week we opened up models that are readable by construction, and today we explained models we cannot see. All three assumed labelled data and a prediction to explain. Next week drops that assumption: no labels, no prediction, just structure. Clustering and dimensionality reduction are where visualization stops being a way to check a model and becomes the analysis itself, because a projection is the only way anyone ever looks at a high-dimensional space. Practical: the default project and the full project discussion are both next week, and the 4-page proposal is due October 20, so next week is the one to come to with questions. :::