Feature importance

Creates a data.table of feature importances.

Usage

xgb.importance(
  model = NULL,
  feature_names = getinfo(model, "feature_name"),
  trees = NULL
)

Arguments

model

Object of class xgb.Booster.

feature_names

Character vector used to overwrite the feature names of the model. The default is NULL (use original feature names).

trees

An integer vector of (base-1) tree indices that should be included into the importance calculation (only for the "gbtree" booster). The default (NULL) parses all trees. It could be useful, e.g., in multiclass classification to get feature importances for each class separately.

Value

A data.table with the following columns:

For a tree model:

• Features: Names of the features used in the model.

• Gain: Fractional contribution of each feature to the model based on the total gain of this feature's splits. Higher percentage means higher importance.

• Cover: Metric of the number of observation related to this feature.

• Frequency: Percentage of times a feature has been used in trees.

For a linear model:

• Features: Names of the features used in the model.

• Weight: Linear coefficient of this feature.

• Class: Class label (only for multiclass models). For objects of class xgboost (as produced by xgboost()), it will be a factor, while for objects of class xgb.Booster (as produced by xgb.train()), it will be a zero-based integer vector.

If feature_names is not provided and model doesn't have feature_names, the index of the features will be used instead. Because the index is extracted from the model dump (based on C++ code), it starts at 0 (as in C/C++ or Python) instead of 1 (usual in R).

Details

This function works for both linear and tree models.

For linear models, the importance is the absolute magnitude of linear coefficients. To obtain a meaningful ranking by importance for linear models, the features need to be on the same scale (which is also recommended when using L1 or L2 regularization).

Examples

# binary classification using "gbtree":
data("ToothGrowth")
x <- ToothGrowth[, c("len", "dose")]
y <- ToothGrowth$supp
model_tree_binary <- xgboost(
  x, y,
  nrounds = 5L,
  nthreads = 1L,
  booster = "gbtree",
  max_depth = 2L
)
xgb.importance(model_tree_binary)
#>    Feature      Gain    Cover Frequency
#>     <char>     <num>    <num>     <num>
#> 1:     len 0.6285416 0.600436 0.4615385
#> 2:    dose 0.3714584 0.399564 0.5384615

# binary classification using "gblinear":
model_tree_linear <- xgboost(
  x, y,
  nrounds = 5L,
  nthreads = 1L,
  booster = "gblinear",
  learning_rate = 0.3
)
xgb.importance(model_tree_linear)
#>    Feature      Weight
#>     <char>       <num>
#> 1:    dose  0.08156057
#> 2:     len -0.01104625

# multi-class classification using "gbtree":
data("iris")
x <- iris[, c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width")]
y <- iris$Species
model_tree_multi <- xgboost(
  x, y,
  nrounds = 5L,
  nthreads = 1L,
  booster = "gbtree",
  max_depth = 3
)
# all classes clumped together:
xgb.importance(model_tree_multi)
#>         Feature     Gain     Cover Frequency
#>          <char>    <num>     <num>     <num>
#> 1: Petal.Length 0.664454 0.6940665 0.6595745
#> 2:  Petal.Width 0.335546 0.3059335 0.3404255
# inspect importances separately for each class:
num_classes <- 3L
nrounds <- 5L
xgb.importance(
  model_tree_multi, trees = seq(from = 1, by = num_classes, length.out = nrounds)
)
#>         Feature  Gain Cover Frequency
#>          <char> <num> <num>     <num>
#> 1: Petal.Length     1     1         1
xgb.importance(
  model_tree_multi, trees = seq(from = 2, by = num_classes, length.out = nrounds)
)
#>         Feature      Gain     Cover Frequency
#>          <char>     <num>     <num>     <num>
#> 1:  Petal.Width 0.6144668 0.2933525      0.25
#> 2: Petal.Length 0.3855332 0.7066475      0.75
xgb.importance(
  model_tree_multi, trees = seq(from = 3, by = num_classes, length.out = nrounds)
)
#>         Feature      Gain     Cover Frequency
#>          <char>     <num>     <num>     <num>
#> 1: Petal.Length 0.5726846 0.5483562       0.5
#> 2:  Petal.Width 0.4273154 0.4516438       0.5

# multi-class classification using "gblinear":
model_linear_multi <- xgboost(
  x, y,
  nrounds = 5L,
  nthreads = 1L,
  booster = "gblinear",
  learning_rate = 0.2
)
xgb.importance(model_linear_multi)
#>          Feature        Weight      Class
#>           <char>         <num>     <fctr>
#>  1:  Petal.Width -0.2877993584     setosa
#>  2: Petal.Length -0.0906287134     setosa
#>  3:  Sepal.Width  0.0657290220     setosa
#>  4: Sepal.Length  0.0016006248     setosa
#>  5:  Petal.Width  0.0335206985 versicolor
#>  6:  Sepal.Width -0.0243656412 versicolor
#>  7: Petal.Length  0.0218368899 versicolor
#>  8: Sepal.Length  0.0006736242 versicolor
#>  9:  Petal.Width  0.1891384423  virginica
#> 10: Petal.Length  0.0509033017  virginica
#> 11:  Sepal.Width -0.0322071426  virginica
#> 12: Sepal.Length -0.0007461226  virginica