⬡ Hub
Skip to content

XGBoost: Feature Importance and Model Interpretation

Understanding why an XGBoost model makes certain predictions, and which features contribute most to those predictions, is crucial for debugging, gaining trust in the model, and extracting business insights. XGBoost provides built-in tools for feature importance and various techniques can be employed for deeper model interpretation.

1. Feature Importance

XGBoost can calculate feature importance based on several metrics:

  • weight (default): The number of times a feature is used in a tree across all boosting rounds.
  • gain: The average gain of splits which use the feature. This is the improvement in accuracy brought by a feature to the branches it is on.
  • cover: The average coverage of splits which use the feature. Coverage refers to the relative number of observations concerned by a feature.
  • total_gain: Total gain across all splits the feature is used in.
  • total_cover: Total cover across all splits the feature is used in.

Getting Feature Importance

import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import pandas as pd

# Generate synthetic data
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=42)
# Assign feature names
feature_names = [f'feature_{i}' for i in range(X.shape[1])]
X_df = pd.DataFrame(X, columns=feature_names) # Use DataFrame for named features

X_train, X_test, y_train, y_test = train_test_split(X_df, y, test_size=0.2, random_state=42)

# Train an XGBoost Classifier
model = xgb.XGBClassifier(objective='binary:logistic',
                          n_estimators=100,
                          learning_rate=0.1,
                          eval_metric='logloss',
                          use_label_encoder=False,
                          random_state=42)
model.fit(X_train, y_train)

# Get feature importances
# .feature_importances_ (Scikit-learn API) provides 'gain' by default
print("Feature Importances (gain, Scikit-learn API):\n", model.feature_importances_)

# Using get_booster().get_score() for different types of importance
# Requires feature names to be set in DMatrix or DataFrame columns
importance_weight = model.get_booster().get_score(importance_type='weight')
importance_gain = model.get_booster().get_score(importance_type='gain')
importance_cover = model.get_booster().get_score(importance_type='cover')

print("\nFeature Importances (weight, Native API):\n", importance_weight)
print("Feature Importances (gain, Native API):\n", importance_gain)
print("Feature Importances (cover, Native API):\n", importance_cover)

# Sort and display
sorted_importance = sorted(importance_gain.items(), key=lambda x: x[1], reverse=True)
print("\nSorted Feature Importances (gain):\n", sorted_importance)

Visualizing Feature Importance

import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import pandas as pd

# (Assume model and X_train are already defined and trained as above)

# Plotting feature importance using XGBoost's built-in function
plt.figure(figsize=(10, 6))
xgb.plot_importance(model, importance_type='gain', ax=plt.gca(), max_num_features=10)
plt.title('XGBoost Feature Importance (Type: Gain)')
plt.show()

plt.figure(figsize=(10, 6))
xgb.plot_importance(model, importance_type='weight', ax=plt.gca(), max_num_features=10)
plt.title('XGBoost Feature Importance (Type: Weight)')
plt.show()

2. Tree Visualization

Visualizing individual trees within the ensemble can provide insights into the model's decision-making process for specific features.

import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# (Assume model and X_train are already defined and trained as above)

# Plot the first tree in the ensemble
# Requires graphviz and pydot to be installed (pip install graphviz pydot)
# And graphviz executable added to PATH.
try:
    plt.figure(figsize=(20, 15))
    xgb.plot_tree(model, num_trees=0, rankdir='LR') # num_trees=0 for the first tree
    plt.title('First Tree in XGBoost Ensemble', fontsize=20)
    plt.show()

    # If you want to save it as a file
    # fig, ax = plt.subplots(figsize=(20, 15))
    # xgb.plot_tree(model, num_trees=0, rankdir='LR', ax=ax)
    # plt.savefig('xgboost_tree_0.png')

except Exception as e:
    print(f"\nCould not plot tree. Ensure Graphviz is installed and in PATH. Error: {e}")

3. SHAP (SHapley Additive exPlanations) Values

SHAP is a game theory-based approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using Shapley values. SHAP provides a powerful way to understand both global feature importance and individual prediction explanations.

import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import shap # pip install shap
import matplotlib.pyplot as plt

# Generate synthetic data
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=42)
feature_names = [f'feature_{i}' for i in range(X.shape[1])]
X_df = pd.DataFrame(X, columns=feature_names)

X_train, X_test, y_train, y_test = train_test_split(X_df, y, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(objective='binary:logistic',
                          n_estimators=100,
                          learning_rate=0.1,
                          use_label_encoder=False,
                          eval_metric='logloss',
                          random_state=42)
model.fit(X_train, y_train)

# Create a SHAP explainer object
explainer = shap.TreeExplainer(model)

# Calculate SHAP values for the test set
shap_values = explainer.shap_values(X_test)

# --- Global Feature Importance (SHAP Summary Plot) ---
print("\nSHAP Summary Plot (Global Feature Importance):")
shap.summary_plot(shap_values, X_test, plot_type="bar", show=False)
plt.title('SHAP Feature Importance (Global)')
plt.show()

shap.summary_plot(shap_values, X_test, show=False) # Beeswarm plot
plt.title('SHAP Summary Plot (Beeswarm)')
plt.show()

# --- Local Explanation (SHAP Force Plot for a single instance) ---
# Explain the prediction of the first instance in the test set
print("\nSHAP Force Plot (Local Explanation for first instance):")
shap.initjs() # For JS visualization in notebooks
shap.force_plot(explainer.expected_value, shap_values[0,:], X_test.iloc[0,:])

# For command-line environments, this may not render directly,
# but the concept is to show how each feature pushes the prediction
# from the base value to the model's output for that specific instance.

Further Topics:

  • Partial Dependence Plots (PDP)
  • Individual Conditional Expectation (ICE) plots
  • LIME (Local Interpretable Model-agnostic Explanations)
  • Understanding feature interaction effects.

Model interpretability is becoming increasingly important for transparency, fairness, and building trust in machine learning systems. XGBoost, combined with tools like SHAP, provides powerful capabilities to achieve this.