Callback for collecting coefficients history of a gblinear booster

Usage

xgb.cb.gblinear.history(sparse = FALSE)

Arguments

sparse: When set to FALSE/TRUE, a dense/sparse matrix is used to store the result. Sparse format is useful when one expects only a subset of coefficients to be non-zero, when using the "thrifty" feature selector with fairly small number of top features selected per iteration.

Value

An xgb.Callback object, which can be passed to xgb.train() or xgb.cv().

Details

To keep things fast and simple, gblinear booster does not internally store the history of linear model coefficients at each boosting iteration. This callback provides a workaround for storing the coefficients' path, by extracting them after each training iteration.

This callback will construct a matrix where rows are boosting iterations and columns are feature coefficients (same order as when calling coef.xgb.Booster, with the intercept corresponding to the first column).

When there is more than one coefficient per feature (e.g. multi-class classification), the result will be reshaped into a vector where coefficients are arranged first by features and then by class (e.g. first 1 through N coefficients will be for the first class, then coefficients N+1 through 2N for the second class, and so on).

If the result has only one coefficient per feature in the data, then the resulting matrix will have column names matching with the feature names, otherwise (when there's more than one coefficient per feature) the names will be composed as 'column name' + ':' + 'class index' (so e.g. column 'c1' for class '0' will be named 'c1:0').

With xgb.train(), the output is either a dense or a sparse matrix. With with xgb.cv(), it is a list (one element per each fold) of such matrices.

Function xgb.gblinear.history provides an easy way to retrieve the outputs from this callback.

Examples

#### Binary classification:

## Keep the number of threads to 1 for examples
nthread <- 1
data.table::setDTthreads(nthread)

# In the iris dataset, it is hard to linearly separate Versicolor class from the rest
# without considering the 2nd order interactions:
x <- model.matrix(Species ~ .^2, iris)[, -1]
colnames(x)
#>  [1] "Sepal.Length"              "Sepal.Width"              
#>  [3] "Petal.Length"              "Petal.Width"              
#>  [5] "Sepal.Length:Sepal.Width"  "Sepal.Length:Petal.Length"
#>  [7] "Sepal.Length:Petal.Width"  "Sepal.Width:Petal.Length" 
#>  [9] "Sepal.Width:Petal.Width"   "Petal.Length:Petal.Width" 
dtrain <- xgb.DMatrix(
  scale(x),
  label = 1 * (iris$Species == "versicolor"),
  nthread = nthread
)
param <- xgb.params(
  booster = "gblinear",
  objective = "reg:logistic",
  eval_metric = "auc",
  reg_lambda = 0.0003,
  reg_alpha = 0.0003,
  nthread = nthread
)

# For 'shotgun', which is a default linear updater, using high learning_rate values may result in
# unstable behaviour in some datasets. With this simple dataset, however, the high learning
# rate does not break the convergence, but allows us to illustrate the typical pattern of
# "stochastic explosion" behaviour of this lock-free algorithm at early boosting iterations.
bst <- xgb.train(
  c(param, list(learning_rate = 1.)),
  dtrain,
  evals = list(tr = dtrain),
  nrounds = 200,
  callbacks = list(xgb.cb.gblinear.history())
)
#> [1]	tr-auc:0.818000 
#> [2]	tr-auc:0.840000 
#> [3]	tr-auc:0.857200 
#> [4]	tr-auc:0.874600 
#> [5]	tr-auc:0.888200 
#> [6]	tr-auc:0.900800 
#> [7]	tr-auc:0.910800 
#> [8]	tr-auc:0.922600 
#> [9]	tr-auc:0.930800 
#> [10]	tr-auc:0.940800 
#> [11]	tr-auc:0.948600 
#> [12]	tr-auc:0.956000 
#> [13]	tr-auc:0.961400 
#> [14]	tr-auc:0.966400 
#> [15]	tr-auc:0.970400 
#> [16]	tr-auc:0.973000 
#> [17]	tr-auc:0.977200 
#> [18]	tr-auc:0.979000 
#> [19]	tr-auc:0.981400 
#> [20]	tr-auc:0.983800 
#> [21]	tr-auc:0.986000 
#> [22]	tr-auc:0.987200 
#> [23]	tr-auc:0.989200 
#> [24]	tr-auc:0.990000 
#> [25]	tr-auc:0.991400 
#> [26]	tr-auc:0.992000 
#> [27]	tr-auc:0.992800 
#> [28]	tr-auc:0.993200 
#> [29]	tr-auc:0.993600 
#> [30]	tr-auc:0.994000 
#> [31]	tr-auc:0.994400 
#> [32]	tr-auc:0.995000 
#> [33]	tr-auc:0.995200 
#> [34]	tr-auc:0.996000 
#> [35]	tr-auc:0.996000 
#> [36]	tr-auc:0.996400 
#> [37]	tr-auc:0.996800 
#> [38]	tr-auc:0.997400 
#> [39]	tr-auc:0.997600 
#> [40]	tr-auc:0.997600 
#> [41]	tr-auc:0.997600 
#> [42]	tr-auc:0.997800 
#> [43]	tr-auc:0.998000 
#> [44]	tr-auc:0.998000 
#> [45]	tr-auc:0.998200 
#> [46]	tr-auc:0.998200 
#> [47]	tr-auc:0.998200 
#> [48]	tr-auc:0.998200 
#> [49]	tr-auc:0.998200 
#> [50]	tr-auc:0.998200 
#> [51]	tr-auc:0.998200 
#> [52]	tr-auc:0.998400 
#> [53]	tr-auc:0.998200 
#> [54]	tr-auc:0.998200 
#> [55]	tr-auc:0.998200 
#> [56]	tr-auc:0.998200 
#> [57]	tr-auc:0.998200 
#> [58]	tr-auc:0.998200 
#> [59]	tr-auc:0.998200 
#> [60]	tr-auc:0.998200 
#> [61]	tr-auc:0.998200 
#> [62]	tr-auc:0.998200 
#> [63]	tr-auc:0.998200 
#> [64]	tr-auc:0.998400 
#> [65]	tr-auc:0.998400 
#> [66]	tr-auc:0.998600 
#> [67]	tr-auc:0.998600 
#> [68]	tr-auc:0.998600 
#> [69]	tr-auc:0.998600 
#> [70]	tr-auc:0.998600 
#> [71]	tr-auc:0.998600 
#> [72]	tr-auc:0.998600 
#> [73]	tr-auc:0.998600 
#> [74]	tr-auc:0.998600 
#> [75]	tr-auc:0.998600 
#> [76]	tr-auc:0.998600 
#> [77]	tr-auc:0.998600 
#> [78]	tr-auc:0.998600 
#> [79]	tr-auc:0.998600 
#> [80]	tr-auc:0.998600 
#> [81]	tr-auc:0.998600 
#> [82]	tr-auc:0.998600 
#> [83]	tr-auc:0.998600 
#> [84]	tr-auc:0.998600 
#> [85]	tr-auc:0.998600 
#> [86]	tr-auc:0.998600 
#> [87]	tr-auc:0.998600 
#> [88]	tr-auc:0.998600 
#> [89]	tr-auc:0.998600 
#> [90]	tr-auc:0.998600 
#> [91]	tr-auc:0.998600 
#> [92]	tr-auc:0.998600 
#> [93]	tr-auc:0.998600 
#> [94]	tr-auc:0.998600 
#> [95]	tr-auc:0.998600 
#> [96]	tr-auc:0.998600 
#> [97]	tr-auc:0.998600 
#> [98]	tr-auc:0.998600 
#> [99]	tr-auc:0.998600 
#> [100]	tr-auc:0.998600 
#> [101]	tr-auc:0.998600 
#> [102]	tr-auc:0.998600 
#> [103]	tr-auc:0.998600 
#> [104]	tr-auc:0.998600 
#> [105]	tr-auc:0.998600 
#> [106]	tr-auc:0.998600 
#> [107]	tr-auc:0.998600 
#> [108]	tr-auc:0.998600 
#> [109]	tr-auc:0.998600 
#> [110]	tr-auc:0.998600 
#> [111]	tr-auc:0.998600 
#> [112]	tr-auc:0.998600 
#> [113]	tr-auc:0.998600 
#> [114]	tr-auc:0.998600 
#> [115]	tr-auc:0.998600 
#> [116]	tr-auc:0.998600 
#> [117]	tr-auc:0.998600 
#> [118]	tr-auc:0.998600 
#> [119]	tr-auc:0.998600 
#> [120]	tr-auc:0.998600 
#> [121]	tr-auc:0.998600 
#> [122]	tr-auc:0.998600 
#> [123]	tr-auc:0.998600 
#> [124]	tr-auc:0.998600 
#> [125]	tr-auc:0.998600 
#> [126]	tr-auc:0.998600 
#> [127]	tr-auc:0.998600 
#> [128]	tr-auc:0.998600 
#> [129]	tr-auc:0.998600 
#> [130]	tr-auc:0.998600 
#> [131]	tr-auc:0.998600 
#> [132]	tr-auc:0.998600 
#> [133]	tr-auc:0.998600 
#> [134]	tr-auc:0.998600 
#> [135]	tr-auc:0.998600 
#> [136]	tr-auc:0.998600 
#> [137]	tr-auc:0.998600 
#> [138]	tr-auc:0.998600 
#> [139]	tr-auc:0.998600 
#> [140]	tr-auc:0.998600 
#> [141]	tr-auc:0.998600 
#> [142]	tr-auc:0.998400 
#> [143]	tr-auc:0.998400 
#> [144]	tr-auc:0.998400 
#> [145]	tr-auc:0.998400 
#> [146]	tr-auc:0.998400 
#> [147]	tr-auc:0.998400 
#> [148]	tr-auc:0.998400 
#> [149]	tr-auc:0.998400 
#> [150]	tr-auc:0.998400 
#> [151]	tr-auc:0.998400 
#> [152]	tr-auc:0.998400 
#> [153]	tr-auc:0.998400 
#> [154]	tr-auc:0.998400 
#> [155]	tr-auc:0.998400 
#> [156]	tr-auc:0.998400 
#> [157]	tr-auc:0.998400 
#> [158]	tr-auc:0.998400 
#> [159]	tr-auc:0.998400 
#> [160]	tr-auc:0.998400 
#> [161]	tr-auc:0.998400 
#> [162]	tr-auc:0.998400 
#> [163]	tr-auc:0.998400 
#> [164]	tr-auc:0.998400 
#> [165]	tr-auc:0.998400 
#> [166]	tr-auc:0.998400 
#> [167]	tr-auc:0.998400 
#> [168]	tr-auc:0.998400 
#> [169]	tr-auc:0.998400 
#> [170]	tr-auc:0.998400 
#> [171]	tr-auc:0.998400 
#> [172]	tr-auc:0.998400 
#> [173]	tr-auc:0.998400 
#> [174]	tr-auc:0.998400 
#> [175]	tr-auc:0.998400 
#> [176]	tr-auc:0.998400 
#> [177]	tr-auc:0.998400 
#> [178]	tr-auc:0.998400 
#> [179]	tr-auc:0.998400 
#> [180]	tr-auc:0.998400 
#> [181]	tr-auc:0.998400 
#> [182]	tr-auc:0.998400 
#> [183]	tr-auc:0.998400 
#> [184]	tr-auc:0.998400 
#> [185]	tr-auc:0.998400 
#> [186]	tr-auc:0.998400 
#> [187]	tr-auc:0.998400 
#> [188]	tr-auc:0.998400 
#> [189]	tr-auc:0.998400 
#> [190]	tr-auc:0.998400 
#> [191]	tr-auc:0.998400 
#> [192]	tr-auc:0.998400 
#> [193]	tr-auc:0.998400 
#> [194]	tr-auc:0.998400 
#> [195]	tr-auc:0.998400 
#> [196]	tr-auc:0.998400 
#> [197]	tr-auc:0.998400 
#> [198]	tr-auc:0.998400 
#> [199]	tr-auc:0.998400 
#> [200]	tr-auc:0.998400 

# Extract the coefficients' path and plot them vs boosting iteration number:
coef_path <- xgb.gblinear.history(bst)
matplot(coef_path, type = "l")


# With the deterministic coordinate descent updater, it is safer to use higher learning rates.
# Will try the classical componentwise boosting which selects a single best feature per round:
bst <- xgb.train(
  c(
    param,
    xgb.params(
      learning_rate = 0.8,
      updater = "coord_descent",
      feature_selector = "thrifty",
      top_k = 1
    )
  ),
  dtrain,
  evals = list(tr = dtrain),
  nrounds = 200,
  callbacks = list(xgb.cb.gblinear.history())
)
#> [1]	tr-auc:0.794200 
#> [2]	tr-auc:0.794200 
#> [3]	tr-auc:0.802200 
#> [4]	tr-auc:0.822000 
#> [5]	tr-auc:0.839200 
#> [6]	tr-auc:0.848400 
#> [7]	tr-auc:0.863200 
#> [8]	tr-auc:0.872200 
#> [9]	tr-auc:0.885800 
#> [10]	tr-auc:0.873800 
#> [11]	tr-auc:0.877400 
#> [12]	tr-auc:0.889000 
#> [13]	tr-auc:0.895800 
#> [14]	tr-auc:0.908000 
#> [15]	tr-auc:0.909600 
#> [16]	tr-auc:0.922800 
#> [17]	tr-auc:0.923400 
#> [18]	tr-auc:0.934000 
#> [19]	tr-auc:0.931000 
#> [20]	tr-auc:0.937400 
#> [21]	tr-auc:0.939600 
#> [22]	tr-auc:0.944400 
#> [23]	tr-auc:0.941000 
#> [24]	tr-auc:0.945600 
#> [25]	tr-auc:0.943600 
#> [26]	tr-auc:0.946200 
#> [27]	tr-auc:0.950000 
#> [28]	tr-auc:0.952600 
#> [29]	tr-auc:0.951400 
#> [30]	tr-auc:0.955000 
#> [31]	tr-auc:0.956800 
#> [32]	tr-auc:0.959200 
#> [33]	tr-auc:0.958000 
#> [34]	tr-auc:0.959800 
#> [35]	tr-auc:0.962400 
#> [36]	tr-auc:0.964800 
#> [37]	tr-auc:0.966000 
#> [38]	tr-auc:0.967600 
#> [39]	tr-auc:0.966800 
#> [40]	tr-auc:0.967800 
#> [41]	tr-auc:0.969400 
#> [42]	tr-auc:0.971800 
#> [43]	tr-auc:0.965200 
#> [44]	tr-auc:0.967200 
#> [45]	tr-auc:0.968800 
#> [46]	tr-auc:0.970600 
#> [47]	tr-auc:0.972000 
#> [48]	tr-auc:0.972600 
#> [49]	tr-auc:0.974000 
#> [50]	tr-auc:0.975400 
#> [51]	tr-auc:0.975400 
#> [52]	tr-auc:0.975000 
#> [53]	tr-auc:0.975800 
#> [54]	tr-auc:0.977000 
#> [55]	tr-auc:0.978200 
#> [56]	tr-auc:0.977200 
#> [57]	tr-auc:0.977600 
#> [58]	tr-auc:0.979800 
#> [59]	tr-auc:0.980600 
#> [60]	tr-auc:0.979600 
#> [61]	tr-auc:0.979600 
#> [62]	tr-auc:0.981800 
#> [63]	tr-auc:0.982600 
#> [64]	tr-auc:0.983400 
#> [65]	tr-auc:0.983600 
#> [66]	tr-auc:0.982600 
#> [67]	tr-auc:0.983600 
#> [68]	tr-auc:0.985000 
#> [69]	tr-auc:0.986400 
#> [70]	tr-auc:0.986200 
#> [71]	tr-auc:0.986400 
#> [72]	tr-auc:0.986200 
#> [73]	tr-auc:0.987400 
#> [74]	tr-auc:0.987800 
#> [75]	tr-auc:0.988000 
#> [76]	tr-auc:0.988600 
#> [77]	tr-auc:0.988400 
#> [78]	tr-auc:0.989000 
#> [79]	tr-auc:0.988600 
#> [80]	tr-auc:0.988800 
#> [81]	tr-auc:0.988800 
#> [82]	tr-auc:0.990000 
#> [83]	tr-auc:0.990200 
#> [84]	tr-auc:0.990400 
#> [85]	tr-auc:0.992200 
#> [86]	tr-auc:0.990200 
#> [87]	tr-auc:0.990600 
#> [88]	tr-auc:0.992200 
#> [89]	tr-auc:0.992800 
#> [90]	tr-auc:0.993000 
#> [91]	tr-auc:0.993200 
#> [92]	tr-auc:0.992400 
#> [93]	tr-auc:0.993000 
#> [94]	tr-auc:0.993400 
#> [95]	tr-auc:0.993400 
#> [96]	tr-auc:0.993600 
#> [97]	tr-auc:0.993600 
#> [98]	tr-auc:0.993200 
#> [99]	tr-auc:0.993800 
#> [100]	tr-auc:0.993800 
#> [101]	tr-auc:0.994000 
#> [102]	tr-auc:0.994200 
#> [103]	tr-auc:0.994600 
#> [104]	tr-auc:0.994600 
#> [105]	tr-auc:0.994800 
#> [106]	tr-auc:0.994200 
#> [107]	tr-auc:0.994400 
#> [108]	tr-auc:0.995000 
#> [109]	tr-auc:0.995400 
#> [110]	tr-auc:0.995400 
#> [111]	tr-auc:0.995400 
#> [112]	tr-auc:0.995200 
#> [113]	tr-auc:0.995400 
#> [114]	tr-auc:0.995600 
#> [115]	tr-auc:0.995800 
#> [116]	tr-auc:0.995000 
#> [117]	tr-auc:0.995000 
#> [118]	tr-auc:0.995000 
#> [119]	tr-auc:0.994400 
#> [120]	tr-auc:0.994800 
#> [121]	tr-auc:0.995600 
#> [122]	tr-auc:0.995800 
#> [123]	tr-auc:0.995800 
#> [124]	tr-auc:0.995800 
#> [125]	tr-auc:0.995800 
#> [126]	tr-auc:0.995800 
#> [127]	tr-auc:0.996400 
#> [128]	tr-auc:0.997000 
#> [129]	tr-auc:0.996400 
#> [130]	tr-auc:0.996400 
#> [131]	tr-auc:0.996600 
#> [132]	tr-auc:0.997000 
#> [133]	tr-auc:0.996400 
#> [134]	tr-auc:0.996400 
#> [135]	tr-auc:0.997200 
#> [136]	tr-auc:0.997000 
#> [137]	tr-auc:0.997600 
#> [138]	tr-auc:0.997600 
#> [139]	tr-auc:0.997000 
#> [140]	tr-auc:0.997200 
#> [141]	tr-auc:0.997600 
#> [142]	tr-auc:0.997600 
#> [143]	tr-auc:0.997800 
#> [144]	tr-auc:0.997800 
#> [145]	tr-auc:0.997400 
#> [146]	tr-auc:0.997600 
#> [147]	tr-auc:0.997600 
#> [148]	tr-auc:0.997800 
#> [149]	tr-auc:0.997200 
#> [150]	tr-auc:0.997200 
#> [151]	tr-auc:0.997200 
#> [152]	tr-auc:0.997400 
#> [153]	tr-auc:0.997200 
#> [154]	tr-auc:0.997800 
#> [155]	tr-auc:0.997600 
#> [156]	tr-auc:0.997800 
#> [157]	tr-auc:0.997800 
#> [158]	tr-auc:0.997800 
#> [159]	tr-auc:0.997800 
#> [160]	tr-auc:0.997800 
#> [161]	tr-auc:0.997800 
#> [162]	tr-auc:0.997600 
#> [163]	tr-auc:0.997800 
#> [164]	tr-auc:0.997600 
#> [165]	tr-auc:0.997800 
#> [166]	tr-auc:0.997600 
#> [167]	tr-auc:0.997800 
#> [168]	tr-auc:0.997800 
#> [169]	tr-auc:0.997800 
#> [170]	tr-auc:0.997800 
#> [171]	tr-auc:0.997800 
#> [172]	tr-auc:0.997800 
#> [173]	tr-auc:0.997800 
#> [174]	tr-auc:0.997800 
#> [175]	tr-auc:0.997800 
#> [176]	tr-auc:0.997800 
#> [177]	tr-auc:0.998000 
#> [178]	tr-auc:0.997800 
#> [179]	tr-auc:0.998000 
#> [180]	tr-auc:0.998000 
#> [181]	tr-auc:0.998000 
#> [182]	tr-auc:0.998000 
#> [183]	tr-auc:0.998000 
#> [184]	tr-auc:0.997800 
#> [185]	tr-auc:0.997800 
#> [186]	tr-auc:0.997800 
#> [187]	tr-auc:0.997800 
#> [188]	tr-auc:0.998000 
#> [189]	tr-auc:0.997800 
#> [190]	tr-auc:0.998000 
#> [191]	tr-auc:0.998000 
#> [192]	tr-auc:0.998000 
#> [193]	tr-auc:0.998000 
#> [194]	tr-auc:0.998000 
#> [195]	tr-auc:0.998000 
#> [196]	tr-auc:0.997800 
#> [197]	tr-auc:0.997800 
#> [198]	tr-auc:0.997800 
#> [199]	tr-auc:0.998000 
#> [200]	tr-auc:0.998000 
matplot(xgb.gblinear.history(bst), type = "l")

#  Componentwise boosting is known to have similar effect to Lasso regularization.
# Try experimenting with various values of top_k, learning_rate, nrounds,
# as well as different feature_selectors.

# For xgb.cv:
bst <- xgb.cv(
  c(
    param,
    xgb.params(
      learning_rate = 0.8,
      updater = "coord_descent",
      feature_selector = "thrifty",
      top_k = 1
    )
  ),
  dtrain,
  nfold = 5,
  nrounds = 100,
  callbacks = list(xgb.cb.gblinear.history())
)
#> [1]	train-auc:0.793060±0.023328	test-auc:0.775454±0.086055 
#> [2]	train-auc:0.793060±0.023328	test-auc:0.775454±0.086055 
#> [3]	train-auc:0.804262±0.017919	test-auc:0.781522±0.086891 
#> [4]	train-auc:0.821515±0.019493	test-auc:0.799411±0.077902 
#> [5]	train-auc:0.835118±0.019551	test-auc:0.809509±0.076119 
#> [6]	train-auc:0.847467±0.014156	test-auc:0.823381±0.073306 
#> [7]	train-auc:0.862370±0.013194	test-auc:0.839700±0.082388 
#> [8]	train-auc:0.868314±0.013269	test-auc:0.843532±0.072344 
#> [9]	train-auc:0.882742±0.012134	test-auc:0.853325±0.073367 
#> [10]	train-auc:0.878097±0.014015	test-auc:0.839837±0.065236 
#> [11]	train-auc:0.889313±0.015378	test-auc:0.875227±0.075280 
#> [12]	train-auc:0.891337±0.011002	test-auc:0.857023±0.065416 
#> [13]	train-auc:0.902791±0.010819	test-auc:0.885825±0.068979 
#> [14]	train-auc:0.907233±0.012448	test-auc:0.878204±0.060966 
#> [15]	train-auc:0.914820±0.011278	test-auc:0.905098±0.058477 
#> [16]	train-auc:0.919320±0.009285	test-auc:0.891294±0.057408 
#> [17]	train-auc:0.926405±0.007876	test-auc:0.915486±0.052004 
#> [18]	train-auc:0.927659±0.009691	test-auc:0.900044±0.054034 
#> [19]	train-auc:0.933862±0.007179	test-auc:0.921704±0.053379 
#> [20]	train-auc:0.937265±0.007191	test-auc:0.911205±0.053852 
#> [21]	train-auc:0.937510±0.007305	test-auc:0.923811±0.046189 
#> [22]	train-auc:0.936561±0.007346	test-auc:0.918103±0.052207 
#> [23]	train-auc:0.940529±0.008236	test-auc:0.919475±0.058445 
#> [24]	train-auc:0.941540±0.006579	test-auc:0.926639±0.048786 
#> [25]	train-auc:0.944965±0.006023	test-auc:0.926163±0.055860 
#> [26]	train-auc:0.947389±0.004879	test-auc:0.933296±0.044660 
#> [27]	train-auc:0.951834±0.005027	test-auc:0.932607±0.051344 
#> [28]	train-auc:0.952166±0.004347	test-auc:0.937818±0.044498 
#> [29]	train-auc:0.955169±0.004073	test-auc:0.938207±0.043216 
#> [30]	train-auc:0.954571±0.003837	test-auc:0.942718±0.045841 
#> [31]	train-auc:0.957766±0.003101	test-auc:0.941907±0.041792 
#> [32]	train-auc:0.959117±0.002933	test-auc:0.951014±0.039270 
#> [33]	train-auc:0.960919±0.004649	test-auc:0.949523±0.038817 
#> [34]	train-auc:0.959323±0.001723	test-auc:0.946533±0.040178 
#> [35]	train-auc:0.962159±0.002223	test-auc:0.949764±0.040940 
#> [36]	train-auc:0.963183±0.002142	test-auc:0.950381±0.036574 
#> [37]	train-auc:0.965269±0.002112	test-auc:0.953645±0.038673 
#> [38]	train-auc:0.964997±0.002036	test-auc:0.954421±0.041280 
#> [39]	train-auc:0.965734±0.005493	test-auc:0.953581±0.035973 
#> [40]	train-auc:0.966805±0.005332	test-auc:0.953548±0.034795 
#> [41]	train-auc:0.968330±0.004304	test-auc:0.957454±0.038927 
#> [42]	train-auc:0.968186±0.005057	test-auc:0.953612±0.037447 
#> [43]	train-auc:0.970721±0.004528	test-auc:0.960325±0.041527 
#> [44]	train-auc:0.971692±0.004703	test-auc:0.957409±0.040353 
#> [45]	train-auc:0.973163±0.004947	test-auc:0.962882±0.044139 
#> [46]	train-auc:0.973600±0.004863	test-auc:0.959935±0.041320 
#> [47]	train-auc:0.973016±0.003829	test-auc:0.960609±0.043500 
#> [48]	train-auc:0.973494±0.004160	test-auc:0.961071±0.041603 
#> [49]	train-auc:0.972523±0.002122	test-auc:0.962428±0.041341 
#> [50]	train-auc:0.974420±0.001986	test-auc:0.963564±0.041538 
#> [51]	train-auc:0.975742±0.002798	test-auc:0.963354±0.040887 
#> [52]	train-auc:0.976464±0.004398	test-auc:0.965416±0.040670 
#> [53]	train-auc:0.977318±0.004273	test-auc:0.964280±0.040534 
#> [54]	train-auc:0.978557±0.004215	test-auc:0.966552±0.040964 
#> [55]	train-auc:0.979428±0.004153	test-auc:0.964280±0.040534 
#> [56]	train-auc:0.979964±0.005074	test-auc:0.966552±0.040964 
#> [57]	train-auc:0.980773±0.005013	test-auc:0.964280±0.040534 
#> [58]	train-auc:0.980219±0.003956	test-auc:0.964280±0.040534 
#> [59]	train-auc:0.980709±0.003563	test-auc:0.966552±0.040964 
#> [60]	train-auc:0.981444±0.003426	test-auc:0.964280±0.040534 
#> [61]	train-auc:0.982428±0.003666	test-auc:0.966552±0.040964 
#> [62]	train-auc:0.983324±0.004339	test-auc:0.965206±0.040284 
#> [63]	train-auc:0.983669±0.004485	test-auc:0.968338±0.037481 
#> [64]	train-auc:0.984065±0.004544	test-auc:0.965206±0.040284 
#> [65]	train-auc:0.984232±0.004892	test-auc:0.968338±0.037481 
#> [66]	train-auc:0.985011±0.004761	test-auc:0.967884±0.035128 
#> [67]	train-auc:0.984995±0.004352	test-auc:0.971293±0.035710 
#> [68]	train-auc:0.985225±0.005118	test-auc:0.969021±0.035140 
#> [69]	train-auc:0.985587±0.005397	test-auc:0.971293±0.035710 
#> [70]	train-auc:0.986212±0.005039	test-auc:0.969021±0.035140 
#> [71]	train-auc:0.986165±0.004682	test-auc:0.968777±0.033471 
#> [72]	train-auc:0.986648±0.003878	test-auc:0.969021±0.035140 
#> [73]	train-auc:0.986906±0.004498	test-auc:0.968777±0.033471 
#> [74]	train-auc:0.987640±0.003998	test-auc:0.970872±0.034677 
#> [75]	train-auc:0.988778±0.003994	test-auc:0.970629±0.033001 
#> [76]	train-auc:0.988700±0.003395	test-auc:0.974964±0.033111 
#> [77]	train-auc:0.988599±0.004077	test-auc:0.971555±0.032960 
#> [78]	train-auc:0.988917±0.004284	test-auc:0.974753±0.032874 
#> [79]	train-auc:0.989517±0.003499	test-auc:0.970629±0.033001 
#> [80]	train-auc:0.990186±0.003209	test-auc:0.974753±0.032874 
#> [81]	train-auc:0.989834±0.003765	test-auc:0.972481±0.033048 
#> [82]	train-auc:0.990398±0.003316	test-auc:0.977952±0.033537 
#> [83]	train-auc:0.990021±0.003544	test-auc:0.977465±0.029251 
#> [84]	train-auc:0.991196±0.003111	test-auc:0.977952±0.033537 
#> [85]	train-auc:0.990861±0.003247	test-auc:0.979251±0.025559 
#> [86]	train-auc:0.991306±0.002390	test-auc:0.973407±0.033266 
#> [87]	train-auc:0.990578±0.003243	test-auc:0.980631±0.027751 
#> [88]	train-auc:0.991149±0.002685	test-auc:0.976329±0.029370 
#> [89]	train-auc:0.991396±0.002608	test-auc:0.982449±0.025924 
#> [90]	train-auc:0.991951±0.002110	test-auc:0.978391±0.029482 
#> [91]	train-auc:0.991816±0.002562	test-auc:0.982449±0.025924 
#> [92]	train-auc:0.992131±0.002253	test-auc:0.980177±0.025743 
#> [93]	train-auc:0.992381±0.002761	test-auc:0.981313±0.025708 
#> [94]	train-auc:0.992568±0.002480	test-auc:0.980177±0.025743 
#> [95]	train-auc:0.992450±0.002467	test-auc:0.981491±0.024084 
#> [96]	train-auc:0.993132±0.002176	test-auc:0.980144±0.023750 
#> [97]	train-auc:0.993245±0.002626	test-auc:0.982416±0.023949 
#> [98]	train-auc:0.993874±0.002192	test-auc:0.981070±0.023905 
#> [99]	train-auc:0.993870±0.002054	test-auc:0.983342±0.023994 
#> [100]	train-auc:0.993996±0.002017	test-auc:0.982206±0.023814 
# coefficients in the CV fold #3
matplot(xgb.gblinear.history(bst)[[3]], type = "l")



#### Multiclass classification:
dtrain <- xgb.DMatrix(scale(x), label = as.numeric(iris$Species) - 1, nthread = nthread)

param <- xgb.params(
  booster = "gblinear",
  objective = "multi:softprob",
  num_class = 3,
  reg_lambda = 0.0003,
  reg_alpha = 0.0003,
  nthread = nthread
)

# For the default linear updater 'shotgun' it sometimes is helpful
# to use smaller learning_rate to reduce instability
bst <- xgb.train(
  c(param, list(learning_rate = 0.5)),
  dtrain,
  evals = list(tr = dtrain),
  nrounds = 50,
  callbacks = list(xgb.cb.gblinear.history())
)
#> [1]	tr-mlogloss:0.582812 
#> [2]	tr-mlogloss:0.447500 
#> [3]	tr-mlogloss:0.366725 
#> [4]	tr-mlogloss:0.306350 
#> [5]	tr-mlogloss:0.258967 
#> [6]	tr-mlogloss:0.222000 
#> [7]	tr-mlogloss:0.193138 
#> [8]	tr-mlogloss:0.170405 
#> [9]	tr-mlogloss:0.152281 
#> [10]	tr-mlogloss:0.137667 
#> [11]	tr-mlogloss:0.125807 
#> [12]	tr-mlogloss:0.116022 
#> [13]	tr-mlogloss:0.107892 
#> [14]	tr-mlogloss:0.101078 
#> [15]	tr-mlogloss:0.095316 
#> [16]	tr-mlogloss:0.090437 
#> [17]	tr-mlogloss:0.086270 
#> [18]	tr-mlogloss:0.082687 
#> [19]	tr-mlogloss:0.079586 
#> [20]	tr-mlogloss:0.076887 
#> [21]	tr-mlogloss:0.074542 
#> [22]	tr-mlogloss:0.072452 
#> [23]	tr-mlogloss:0.070595 
#> [24]	tr-mlogloss:0.068935 
#> [25]	tr-mlogloss:0.067446 
#> [26]	tr-mlogloss:0.066103 
#> [27]	tr-mlogloss:0.064898 
#> [28]	tr-mlogloss:0.063817 
#> [29]	tr-mlogloss:0.062831 
#> [30]	tr-mlogloss:0.061929 
#> [31]	tr-mlogloss:0.061102 
#> [32]	tr-mlogloss:0.060343 
#> [33]	tr-mlogloss:0.059644 
#> [34]	tr-mlogloss:0.058999 
#> [35]	tr-mlogloss:0.058404 
#> [36]	tr-mlogloss:0.057852 
#> [37]	tr-mlogloss:0.057341 
#> [38]	tr-mlogloss:0.056866 
#> [39]	tr-mlogloss:0.056424 
#> [40]	tr-mlogloss:0.056012 
#> [41]	tr-mlogloss:0.055627 
#> [42]	tr-mlogloss:0.055266 
#> [43]	tr-mlogloss:0.054929 
#> [44]	tr-mlogloss:0.054615 
#> [45]	tr-mlogloss:0.054321 
#> [46]	tr-mlogloss:0.054041 
#> [47]	tr-mlogloss:0.053777 
#> [48]	tr-mlogloss:0.053528 
#> [49]	tr-mlogloss:0.053292 
#> [50]	tr-mlogloss:0.053070 

# Will plot the coefficient paths separately for each class:
matplot(xgb.gblinear.history(bst, class_index = 0), type = "l")

matplot(xgb.gblinear.history(bst, class_index = 1), type = "l")

matplot(xgb.gblinear.history(bst, class_index = 2), type = "l")


# CV:
bst <- xgb.cv(
  c(param, list(learning_rate = 0.5)),
  dtrain,
  nfold = 5,
  nrounds = 70,
  callbacks = list(xgb.cb.gblinear.history(FALSE))
)
#> [1]	train-mlogloss:0.581537±0.012604	test-mlogloss:0.589547±0.057862 
#> [2]	train-mlogloss:0.445828±0.012655	test-mlogloss:0.458894±0.062860 
#> [3]	train-mlogloss:0.364919±0.009281	test-mlogloss:0.381273±0.057068 
#> [4]	train-mlogloss:0.304609±0.006136	test-mlogloss:0.322367±0.045592 
#> [5]	train-mlogloss:0.257324±0.005571	test-mlogloss:0.274981±0.033505 
#> [6]	train-mlogloss:0.220425±0.006519	test-mlogloss:0.237491±0.024171 
#> [7]	train-mlogloss:0.191608±0.007651	test-mlogloss:0.208055±0.018819 
#> [8]	train-mlogloss:0.168915±0.008645	test-mlogloss:0.184809±0.017401 
#> [9]	train-mlogloss:0.150826±0.009464	test-mlogloss:0.166270±0.018615 
#> [10]	train-mlogloss:0.136255±0.010131	test-mlogloss:0.151248±0.020932 
#> [11]	train-mlogloss:0.124387±0.010658	test-mlogloss:0.138950±0.023507 
#> [12]	train-mlogloss:0.114627±0.011081	test-mlogloss:0.128779±0.025927 
#> [13]	train-mlogloss:0.106519±0.011396	test-mlogloss:0.120284±0.028145 
#> [14]	train-mlogloss:0.099725±0.011644	test-mlogloss:0.113187±0.030099 
#> [15]	train-mlogloss:0.093990±0.011833	test-mlogloss:0.107233±0.031786 
#> [16]	train-mlogloss:0.089123±0.011985	test-mlogloss:0.102171±0.033226 
#> [17]	train-mlogloss:0.084969±0.012091	test-mlogloss:0.097849±0.034445 
#> [18]	train-mlogloss:0.081404±0.012170	test-mlogloss:0.094121±0.035475 
#> [19]	train-mlogloss:0.078319±0.012213	test-mlogloss:0.090916±0.036341 
#> [20]	train-mlogloss:0.075630±0.012233	test-mlogloss:0.088123±0.037064 
#> [21]	train-mlogloss:0.073269±0.012239	test-mlogloss:0.085665±0.037664 
#> [22]	train-mlogloss:0.071189±0.012240	test-mlogloss:0.083510±0.038146 
#> [23]	train-mlogloss:0.069337±0.012232	test-mlogloss:0.081605±0.038552 
#> [24]	train-mlogloss:0.067690±0.012213	test-mlogloss:0.079930±0.038940 
#> [25]	train-mlogloss:0.066209±0.012193	test-mlogloss:0.078437±0.039278 
#> [26]	train-mlogloss:0.064880±0.012177	test-mlogloss:0.077115±0.039575 
#> [27]	train-mlogloss:0.063681±0.012153	test-mlogloss:0.075929±0.039822 
#> [28]	train-mlogloss:0.062591±0.012127	test-mlogloss:0.074864±0.040050 
#> [29]	train-mlogloss:0.061597±0.012104	test-mlogloss:0.073905±0.040260 
#> [30]	train-mlogloss:0.060687±0.012082	test-mlogloss:0.073036±0.040452 
#> [31]	train-mlogloss:0.059853±0.012062	test-mlogloss:0.072247±0.040631 
#> [32]	train-mlogloss:0.059086±0.012044	test-mlogloss:0.071525±0.040804 
#> [33]	train-mlogloss:0.058380±0.012028	test-mlogloss:0.070867±0.040968 
#> [34]	train-mlogloss:0.057728±0.012015	test-mlogloss:0.070270±0.041125 
#> [35]	train-mlogloss:0.057125±0.012002	test-mlogloss:0.069726±0.041277 
#> [36]	train-mlogloss:0.056564±0.011992	test-mlogloss:0.069227±0.041415 
#> [37]	train-mlogloss:0.056042±0.011986	test-mlogloss:0.068761±0.041528 
#> [38]	train-mlogloss:0.055558±0.011976	test-mlogloss:0.068351±0.041670 
#> [39]	train-mlogloss:0.055107±0.011967	test-mlogloss:0.067973±0.041801 
#> [40]	train-mlogloss:0.054686±0.011959	test-mlogloss:0.067622±0.041934 
#> [41]	train-mlogloss:0.054293±0.011952	test-mlogloss:0.067311±0.042064 
#> [42]	train-mlogloss:0.053927±0.011946	test-mlogloss:0.067038±0.042190 
#> [43]	train-mlogloss:0.053583±0.011942	test-mlogloss:0.066793±0.042313 
#> [44]	train-mlogloss:0.053260±0.011938	test-mlogloss:0.066567±0.042434 
#> [45]	train-mlogloss:0.052956±0.011934	test-mlogloss:0.066357±0.042554 
#> [46]	train-mlogloss:0.052669±0.011932	test-mlogloss:0.066160±0.042666 
#> [47]	train-mlogloss:0.052398±0.011929	test-mlogloss:0.065979±0.042781 
#> [48]	train-mlogloss:0.052143±0.011925	test-mlogloss:0.065810±0.042901 
#> [49]	train-mlogloss:0.051900±0.011920	test-mlogloss:0.065649±0.043016 
#> [50]	train-mlogloss:0.051670±0.011915	test-mlogloss:0.065485±0.043131 
#> [51]	train-mlogloss:0.051452±0.011911	test-mlogloss:0.065328±0.043243 
#> [52]	train-mlogloss:0.051244±0.011907	test-mlogloss:0.065190±0.043352 
#> [53]	train-mlogloss:0.051046±0.011903	test-mlogloss:0.065070±0.043457 
#> [54]	train-mlogloss:0.050858±0.011899	test-mlogloss:0.064961±0.043559 
#> [55]	train-mlogloss:0.050679±0.011895	test-mlogloss:0.064866±0.043659 
#> [56]	train-mlogloss:0.050507±0.011892	test-mlogloss:0.064786±0.043756 
#> [57]	train-mlogloss:0.050343±0.011889	test-mlogloss:0.064712±0.043851 
#> [58]	train-mlogloss:0.050188±0.011886	test-mlogloss:0.064644±0.043943 
#> [59]	train-mlogloss:0.050039±0.011884	test-mlogloss:0.064579±0.044033 
#> [60]	train-mlogloss:0.049896±0.011882	test-mlogloss:0.064522±0.044134 
#> [61]	train-mlogloss:0.049760±0.011880	test-mlogloss:0.064469±0.044233 
#> [62]	train-mlogloss:0.049630±0.011878	test-mlogloss:0.064419±0.044328 
#> [63]	train-mlogloss:0.049506±0.011876	test-mlogloss:0.064372±0.044419 
#> [64]	train-mlogloss:0.049386±0.011876	test-mlogloss:0.064319±0.044488 
#> [65]	train-mlogloss:0.049272±0.011877	test-mlogloss:0.064268±0.044553 
#> [66]	train-mlogloss:0.049162±0.011878	test-mlogloss:0.064222±0.044617 
#> [67]	train-mlogloss:0.049056±0.011879	test-mlogloss:0.064179±0.044681 
#> [68]	train-mlogloss:0.048954±0.011880	test-mlogloss:0.064141±0.044746 
#> [69]	train-mlogloss:0.048857±0.011881	test-mlogloss:0.064106±0.044811 
#> [70]	train-mlogloss:0.048763±0.011883	test-mlogloss:0.064074±0.044876 
# 1st fold of 1st class
matplot(xgb.gblinear.history(bst, class_index = 0)[[1]], type = "l")