{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"shap-and-partial-dependence","title":"Explaining one prediction, and the shape of one feature","url":"https://addaly.com/learn/machine-learning-foundations/shap-and-partial-dependence","position":62,"of":92,"minutes":9,"answer":"SHAP divides one prediction's departure from the average among the features with an exact additive decomposition, partial dependence shows a feature's average learned shape.","keyIdea":"SHAP divides one prediction's departure from the average among the features with an exact additive decomposition, partial dependence shows a feature's average learned shape, and both assume feature independence in ways that break when the columns are correlated."},"body":{"format":"markdown","text":"## Two different questions\n\n\"Which features does this model rely on overall?\" was the last lesson. Two more specific questions come up constantly and need different tools.\n\n**Why did *this* application get declined?** A local explanation, for one row.\n\n**How does the prediction change as *this* feature changes?** A global shape, for one column.\n\n## SHAP: dividing the credit for one prediction\n\nSHAP assigns each feature a number saying how much it pushed this particular prediction away from the average prediction. The numbers add up exactly:\n\n```text\nbase value (average prediction)          0.12\n+ income = 240,000                      -0.04\n+ late_payments = 3                     +0.19\n+ account_age_days = 45                 +0.08\n+ everything else                       -0.02\n= this prediction                        0.33\n```\n\nThat is a readable answer to \"why 0.33\", and it is the format regulators tend to accept for adverse action notices.\n\nThe underlying idea comes from cooperative game theory. Shapley values answer: if features are players cooperating to produce a prediction, what is each one's fair share of the result? The fair division is defined by averaging a feature's marginal contribution over every possible order in which features could be added. That is the only allocation satisfying a small set of reasonable axioms, which is why SHAP has a stronger theoretical footing than most explanation methods.\n\nComputing it exactly requires evaluating all `2^n` subsets. `TreeSHAP` does it exactly for tree ensembles in polynomial time, which is why SHAP became standard alongside gradient boosting specifically.\n\n```python\nimport shap\nexplainer = shap.TreeExplainer(model)\nsv = explainer.shap_values(X_val)\nshap.summary_plot(sv, X_val)\n```\n\n## What SHAP does not tell you\n\nThree limits, stated because they are routinely ignored.\n\n**It explains the model, not the world.** A SHAP value of +0.19 for `late_payments = 3` means this model's output was 0.19 higher because of that value. It does not mean reducing late payments by three would lower the person's real risk. Module 1's distinction, again.\n\n**Correlated features share credit in ways that depend on the method.** With `height_cm` and `height_inches` in the model, how the credit divides between them is partly an artefact of the algorithm's assumptions rather than a fact about the model.\n\n**The standard approximation assumes feature independence.** When features are dependent, the method evaluates the model on combinations that never occur in reality: an income of ₹40 lakh with a postcode where nobody earns that. The model's output on impossible inputs then influences the attribution. There are variants that condition on the data distribution instead, and they are slower and less used.\n\n## Partial dependence: the shape of one feature\n\nA different question. Take every row, set `income` to 200,000 for all of them, average the predictions. Repeat for 250,000, 300,000, and so on. Plot the curve.\n\n```python\nfrom sklearn.inspection import PartialDependenceDisplay\nPartialDependenceDisplay.from_estimator(model, X_val, [\"income\", \"age\"])\n```\n\nThe result shows the average relationship the model learned. It is frequently the most informative single plot you can produce about a tree ensemble, and it regularly reveals things nobody expected: a threshold effect at a round number that turns out to be a policy rule, a relationship that reverses direction, a plateau where a linear model assumed a slope.\n\nIts flaw is the same independence assumption. If income and postcode are correlated, setting income to 40 lakh for every row creates impossible people, and the average includes the model's behaviour on them.\n\n**ALE plots** (accumulated local effects) fix this by computing the effect locally, using only rows where the feature is already near the value being examined. They are more trustworthy on correlated data and less widely used. The `alibi` library implements them, free.\n\n## ICE: the curves the average hides\n\nPartial dependence plots an average, and averages hide heterogeneity. An **individual conditional expectation** plot draws one line per row instead.\n\nIf all the lines are parallel, the average is a fair summary. If half slope up and half slope down, the average is flat and completely misleading: there is an interaction, and the effect of this feature depends on something else in the row.\n\n```python\nPartialDependenceDisplay.from_estimator(model, X_val, [\"income\"],\n                                        kind=\"both\")\n```\n\n`kind=\"both\"` draws the ICE lines and the average together. It costs nothing and it is the check that stops you reporting a flat average over two opposite populations.\n\n## Choosing between them\n\n- **One decision to justify to a person**: SHAP.\n- **Understanding the model's overall behaviour on one feature**: partial dependence, with ICE lines.\n- **Which features matter at all**: permutation importance, from the previous lesson.\n- **Correlated features and you need it right**: ALE, and read the caveats.\n\nAnd the standing rule underneath all four: these describe the model. If somebody is going to act on the explanation to change the world, that requires an experiment, and no amount of attribution substitutes for one."},"check":{"question":"A partial dependence plot for 'discount' is flat, suggesting discount has no effect. The ICE plot shows half the lines sloping up steeply and half sloping down. What is happening?","options":[{"text":"The model has not converged, so its predictions are unstable across rows.","correct":false,"tempting":"Instability would produce noisy, disordered lines rather than two coherent opposing groups."},{"text":"The partial dependence computation used too few grid points, flattening genuine curvature.","correct":false,"tempting":"Coarse grids smooth a curve, but they cannot average two opposite slopes into flatness across individual rows."},{"text":"Discount raises the prediction for one group of customers and lowers it for another, so the average cancels; there is an interaction the partial dependence plot cannot show.","correct":true},{"text":"The ICE lines are computed on training data and the partial dependence on validation data, so the two disagree.","correct":false,"tempting":"Both are computed on the same dataset by the same call, so a train-validation mismatch is not the source."}]},"prev":{"slug":"feature-importance-and-its-lies","title":"Feature importance, and the ways it misleads","url":"https://addaly.com/learn/machine-learning-foundations/feature-importance-and-its-lies"},"next":{"slug":"clustering-with-k-means","title":"k-means, and what it assumes about your data","url":"https://addaly.com/learn/machine-learning-foundations/clustering-with-k-means"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Explaining one prediction, and the shape of one feature', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/shap-and-partial-dependence (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Explaining one prediction, and the shape of one feature","courseTitle":"Machine Learning, Foundations","module":{"index":6,"slug":"trees-and-ensembles","title":"Trees, ensembles, and the algorithms that win on tables"},"answer":null,"keyIdea":"SHAP divides one prediction’s departure from the average among the features with an exact additive decomposition, partial dependence shows a feature’s average learned shape, and both assume feature independence in ways that break when the columns are correlated.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-different-questions","runs":[{"text":"Two different questions"}]},{"t":"p","runs":[{"text":"“Which features does this model rely on overall?” was the last lesson. Two more specific questions come up constantly and need different tools."}]},{"t":"p","runs":[{"text":"**Why did "},{"text":"this","i":true},{"text":" application get declined?** A local explanation, for one row."}]},{"t":"p","runs":[{"text":"**How does the prediction change as "},{"text":"this","i":true},{"text":" feature changes?** A global shape, for one column."}]},{"t":"h2","id":"shap-dividing-the-credit-for-one-prediction","runs":[{"text":"SHAP: dividing the credit for one prediction"}]},{"t":"p","runs":[{"text":"SHAP assigns each feature a number saying how much it pushed this particular prediction away from the average prediction. The numbers add up exactly:"}]},{"t":"code","lang":"text","label":"Text","text":"base value (average prediction)          0.12\n+ income = 240,000                      -0.04\n+ late_payments = 3                     +0.19\n+ account_age_days = 45                 +0.08\n+ everything else                       -0.02\n= this prediction                        0.33"},{"t":"p","runs":[{"text":"That is a readable answer to “why 0.33”, and it is the format regulators tend to accept for adverse action notices."}]},{"t":"p","runs":[{"text":"The underlying idea comes from cooperative game theory. Shapley values answer: if features are players cooperating to produce a prediction, what is each one’s fair share of the result? The fair division is defined by averaging a feature’s marginal contribution over every possible order in which features could be added. That is the only allocation satisfying a small set of reasonable axioms, which is why SHAP has a stronger theoretical footing than most explanation methods."}]},{"t":"p","runs":[{"text":"Computing it exactly requires evaluating all "},{"text":"2^n","code":true},{"text":" subsets. "},{"text":"TreeSHAP","code":true},{"text":" does it exactly for tree ensembles in polynomial time, which is why SHAP became standard alongside gradient boosting specifically."}]},{"t":"code","lang":"python","label":"Python","text":"import shap\nexplainer = shap.TreeExplainer(model)\nsv = explainer.shap_values(X_val)\nshap.summary_plot(sv, X_val)"},{"t":"h2","id":"what-shap-does-not-tell-you","runs":[{"text":"What SHAP does not tell you"}]},{"t":"p","runs":[{"text":"Three limits, stated because they are routinely ignored."}]},{"t":"p","runs":[{"text":"It explains the model, not the world.","b":true},{"text":" A SHAP value of +0.19 for "},{"text":"late_payments = 3","code":true},{"text":" means this model’s output was 0.19 higher because of that value. It does not mean reducing late payments by three would lower the person’s real risk. Module 1’s distinction, again."}]},{"t":"p","runs":[{"text":"Correlated features share credit in ways that depend on the method.","b":true},{"text":" With "},{"text":"height_cm","code":true},{"text":" and "},{"text":"height_inches","code":true},{"text":" in the model, how the credit divides between them is partly an artefact of the algorithm’s assumptions rather than a fact about the model."}]},{"t":"p","runs":[{"text":"The standard approximation assumes feature independence.","b":true},{"text":" When features are dependent, the method evaluates the model on combinations that never occur in reality: an income of ₹40 lakh with a postcode where nobody earns that. The model’s output on impossible inputs then influences the attribution. There are variants that condition on the data distribution instead, and they are slower and less used."}]},{"t":"h2","id":"partial-dependence-the-shape-of-one-feature","runs":[{"text":"Partial dependence: the shape of one feature"}]},{"t":"p","runs":[{"text":"A different question. Take every row, set "},{"text":"income","code":true},{"text":" to 200,000 for all of them, average the predictions. Repeat for 250,000, 300,000, and so on. Plot the curve."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.inspection import PartialDependenceDisplay\nPartialDependenceDisplay.from_estimator(model, X_val, [\"income\", \"age\"])"},{"t":"p","runs":[{"text":"The result shows the average relationship the model learned. It is frequently the most informative single plot you can produce about a tree ensemble, and it regularly reveals things nobody expected: a threshold effect at a round number that turns out to be a policy rule, a relationship that reverses direction, a plateau where a linear model assumed a slope."}]},{"t":"p","runs":[{"text":"Its flaw is the same independence assumption. If income and postcode are correlated, setting income to 40 lakh for every row creates impossible people, and the average includes the model’s behaviour on them."}]},{"t":"p","runs":[{"text":"ALE plots","b":true},{"text":" (accumulated local effects) fix this by computing the effect locally, using only rows where the feature is already near the value being examined. They are more trustworthy on correlated data and less widely used. The "},{"text":"alibi","code":true},{"text":" library implements them, free."}]},{"t":"h2","id":"ice-the-curves-the-average-hides","runs":[{"text":"ICE: the curves the average hides"}]},{"t":"p","runs":[{"text":"Partial dependence plots an average, and averages hide heterogeneity. An "},{"text":"individual conditional expectation","b":true},{"text":" plot draws one line per row instead."}]},{"t":"p","runs":[{"text":"If all the lines are parallel, the average is a fair summary. If half slope up and half slope down, the average is flat and completely misleading: there is an interaction, and the effect of this feature depends on something else in the row."}]},{"t":"code","lang":"python","label":"Python","text":"PartialDependenceDisplay.from_estimator(model, X_val, [\"income\"],\n                                        kind=\"both\")"},{"t":"p","runs":[{"text":"kind=\"both\"","code":true},{"text":" draws the ICE lines and the average together. It costs nothing and it is the check that stops you reporting a flat average over two opposite populations."}]},{"t":"h2","id":"choosing-between-them","runs":[{"text":"Choosing between them"}]},{"t":"ul","items":[[{"text":"One decision to justify to a person","b":true},{"text":": SHAP."}],[{"text":"Understanding the model’s overall behaviour on one feature","b":true},{"text":": partial dependence, with ICE lines."}],[{"text":"Which features matter at all","b":true},{"text":": permutation importance, from the previous lesson."}],[{"text":"Correlated features and you need it right","b":true},{"text":": ALE, and read the caveats."}]]},{"t":"p","runs":[{"text":"And the standing rule underneath all four: these describe the model. If somebody is going to act on the explanation to change the world, that requires an experiment, and no amount of attribution substitutes for one."}]}],"check":{"question":[{"text":"A partial dependence plot for ‘discount’ is flat, suggesting discount has no effect. The ICE plot shows half the lines sloping up steeply and half sloping down. What is happening?"}],"options":[{"text":[{"text":"The model has not converged, so its predictions are unstable across rows."}],"correct":false,"tempting":[{"text":"Instability would produce noisy, disordered lines rather than two coherent opposing groups."}]},{"text":[{"text":"The partial dependence computation used too few grid points, flattening genuine curvature."}],"correct":false,"tempting":[{"text":"Coarse grids smooth a curve, but they cannot average two opposite slopes into flatness across individual rows."}]},{"text":[{"text":"Discount raises the prediction for one group of customers and lowers it for another, so the average cancels; there is an interaction the partial dependence plot cannot show."}],"correct":true,"tempting":null},{"text":[{"text":"The ICE lines are computed on training data and the partial dependence on validation data, so the two disagree."}],"correct":false,"tempting":[{"text":"Both are computed on the same dataset by the same call, so a train-validation mismatch is not the source."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"feature-importance-and-its-lies","title":"Feature importance, and the ways it misleads","minutes":9},"next":{"slug":"clustering-with-k-means","title":"k-means, and what it assumes about your data","minutes":9}}}