A key aspect of machine learning is cross validation to evaluate the model. It repeatedly evaluate the model based on different subsets of the model and using different parameters to select the optimal parameters. The models are compared against subset of the data. The caret library is an excellent tool for performing model selection.
10.1 Feature selection
There are several approaches available for feature selection. These methods include non-deterministic methods such as genetic algorithm and simulated annealing and penalised regression. Filter methods for feature selection include the use of correlation matrix to identified correlated data.
10.2 Decision tree analysis
Decision tree method generates a logical flow diagram that resembles a tree. This triangulated diagram, with repeated partitioning of the original data into smaller groups (nodes) on a yes or no basis, resembles clinical reasoning. By way of contrast, regression methods generate significant predictors but it’s not clear how those predictors enter the sequential nature of clinical reasoning. Regression models assume that all of the variables are required at once to formulate an accurate prediction. This would make some of the elements of any model from regression analysis superfluous.
There are several different approaches to performing decision tree analyses. The most famous method CART is implemented in R as rpart. The second approaches uses chi-square test to partition the tree, available from the party library. Decision tree may also reveal complex interactions (relationship) among the predictors in a way that regression analyses do not easily reveal.
10.2.1 Information theory driven
The tree is grown using a “divide and conquer” strategy, with repeated partitioning of the original data into smaller groups (nodes) on a yes or no basis. The method uses a splitting rule built around the notion of “purity” or information grained. A node in the tree is defined as pure when all the elements belong to one class. When there is impurity in the node or high entropy, a split occurs to maximize reduction in “impurity.”
\(Entropy= \sum_{i=1}^c -p_i ln(p_i)\)
In some cases, the split may be biased toward attributes that contain many different ordinal levels or scales. Thus, the selection of an attribute as the root node may vary according to the splitting rule and the scaling of the attribute. The decision tree package rpart does tolerate certain degree of missing number because the data are split using the available data for that attribute to calculate the Gini index (rather than the entire cohort). The formula for Gini index is given below. \(p_i\) is the probability of class membership of a given variable.
\(Gini=1- \sum_{i=1}^n (p_i)^2\)
One major advantage of rpart is the presentation of the classification rules in the easily interpretable form of a tree. The hierarchical nature of the decision tree is similar to many decision processes (Phan et al. 2018). A criticism of decision tree is that it’s prone to overfitting and or preference for variable with many levels. Decision tree do not handle collinearity issues well.
library(rpart)library(rattle)
Loading required package: tibble
Loading required package: bitops
Rattle: A free graphical interface for data science with R.
Version 5.5.1 Copyright (c) 2006-2021 Togaware Pty Ltd.
Type 'rattle()' to shake, rattle, and roll your data.
#decision tree model for AML treatmenttreLeukemia<-rpart(Status~., data=Leukemia)fancyRpartPlot(treLeukemia)
10.2.2 Conditional decision tree
The conditional decision tree approach has been proposed to be superior to CART method because that method uses information criterion for partitioning and which can lead to overfitting.The scenario of overfitting describes model which works well on training data but less so with new data.The conditional approach by party is less prone to overfitting as it includes significance testing (Phan et al. 2019).
library(party)
Loading required package: grid
Loading required package: mvtnorm
Loading required package: modeltools
Loading required package: stats4
Loading required package: strucchange
Loading required package: zoo
Attaching package: 'zoo'
The following objects are masked from 'package:base':
as.Date, as.Date.numeric
#decision tree modeltreeSAH<-ctree(outcome~., data=aSAH, control =ctree_control(mincriterion=0.95, minsplit=50))plot(treeSAH,type ="simple",main ="Conditional Inference for aSAH")
10.3 Ensemble tree methods
There are several types of ensemble tree methods ranging from bagging, boosting to Bayesian methods.
10.3.1 Bagging trees
Both gradient boost machine and random forest are examples of tree-based method with the former based on boosting of the residuals of the model and the latter based on bagging with random selection (rows and columns) of multiple subsets of the data. As such random forest regression ensembles the model from multiple decision trees. The trees are created by obtaining multiple subset of the data without replacement (random selection of data by rows and columns).
Random forest avoids the problems of single decision tree analyses by aggregating the results of multiple trees obtained by performing analysis on random subsets of the original data. This method is different from the bootstrapping procedure in which the data is subset with replacement. Theoretically, decision tree can look very similar as the data structure is not significantly changed. There is a theoretical risk of overfitting with random forest and underfitting with boosting tree methods.
10.3.1.1 Random Forest
Random forest is available as randomForest or ranger or via caret. A major drawback to random forest is that the hierarchical nature of the trees is lost. As such this method is seen as a black box tool and is less commonly embraced in the medical literature. One way us to use an interpretable machine learning tool iml(Molnar, Bischl, and Casalicchio 2018) (Shapley values) tool to aid interpretation of the model. This method uses ideas from coalition game theory to fairly distribute the contribution of the coalition of covariates to the random forest model.
The machine learning models are tuned using caret library.
library(caret)
Loading required package: ggplot2
Loading required package: lattice
data("BreastCancer",package ="mlbench")#The Breast Cancer data contains NA as well as factors#note Class is benign or malignant of class factor#column Bare.nuclei removed due to NABreastCancer<-BreastCancer[,-c(1,7)]#split data using caTools. #The next example will use createDataPartition from caretset.seed(123)split = caTools::sample.split(BreastCancer$Class, SplitRatio =0.75)Train =subset(BreastCancer, split ==TRUE)Test =subset(BreastCancer, split ==FALSE)# specify that resampling method is rf_control <-trainControl(## 10-fold CVmethod ="cv",number =10)#scaling data is performed here under preProcess#note that ranger handles the outcome variable as factorrf <- caret::train(Class ~ ., data = Train, method ="ranger",trControl=rf_control,preProcess =c("center", "scale"),tuneLength =10, verbose=F)summary(rf)
Length Class Mode
predictions 525 factor numeric
num.trees 1 -none- numeric
num.independent.variables 1 -none- numeric
mtry 1 -none- numeric
min.node.size 1 -none- numeric
prediction.error 1 -none- numeric
forest 9 ranger.forest list
confusion.matrix 4 table numeric
splitrule 1 -none- character
num.random.splits 1 -none- numeric
treetype 1 -none- character
call 9 -none- call
importance.mode 1 -none- character
num.samples 1 -none- numeric
replace 1 -none- logical
xNames 71 -none- character
problemType 1 -none- character
tuneValue 3 data.frame list
obsLevels 2 -none- character
param 1 -none- list
Setting levels: control = benign, case = malignant
Setting direction: controls < cases
roc_rf
Call:
roc.default(response = BreastCancer$Class, predictor = as.numeric(pred_rf))
Data: as.numeric(pred_rf) in 458 controls (BreastCancer$Class benign) < 241 cases (BreastCancer$Class malignant).
Area under the curve: 0.9843
10.3.1.2 Random survival forest with rfsrc
Random survival forest example is provided below using rfsrc library. The survex library is used for explanation on the model. This library is also available as a learner in the mlr3verse.
library(survival)
Attaching package: 'survival'
The following object is masked from 'package:caret':
cluster
library(survminer)
Loading required package: ggpubr
Attaching package: 'survminer'
The following object is masked from 'package:survival':
myeloma
library(randomForestSRC)
randomForestSRC 3.2.2
Type rfsrc.news() to see new features, changes, and bug fixes.
library(survex)library(dplyr)
Attaching package: 'dplyr'
The following object is masked from 'package:survex':
explain
The following object is masked from 'package:party':
where
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
#time in days#status censored=1, dead=2#sex:Male=1 Female=2cancer2<- cancer %>%mutate(status=ifelse(status==1,0,1)) %>%rename(Dead=status, Days=time)time=cancer2$Daysstatus=cancer2$DeadRF<-rfsrc(Surv(Days, Dead) ~ age+sex+ph.ecog+ph.karno+wt.loss, data = cancer2)#specify library to avoid confusion with dplyrexplainer<-survex::explain(RF)
Preparation of a new explainer is initiated
-> model label : rfsrc ( [33m default [39m )
-> data : 213 rows 5 cols ( extracted from the model )
-> target variable : 213 values ( 151 events and 62 censored , censoring rate = 0.291 ) ( extracted from the model )
-> times : 50 unique time points , min = 5 , mean = 304.1224 , median = 263.16 , max = 835.44
-> times : ( generated from y as 50 time points being consecutive quantiles (0.00, 0.02, ..., 0.98) )
-> predict function : sum over the predict_cumulative_hazard_function will be used ( [33m default [39m )
-> predict survival function : stepfun based on predict.rfsrc()$survival will be used ( [33m default [39m )
-> predict cumulative hazard function : stepfun based on predict.rfsrc()$chf will be used ( [33m default [39m )
-> model_info : package randomForestSRC , ver. 3.2.2 , task survival ( [33m default [39m )
A new explainer has been created!
Plot a single tree from the random survival forest model.
plot(get.tree(RF,4))
Dynamic AUC
y <- explainer$ytimes <- explainer$timessurv <- explainer$predict_survival_function(RF, explainer$data, times)cd_auc(y, surv = surv, times = times)
Random forest can be used for performing survival analysis using ranger, randomforestSRC. The example below is an example using the lung cancer trial data.
#data from survival package on NCCTG lung cancer trial#https://stat.ethz.ch/R-manual/R-devel/library/survival/html/lung.htmldata(cancer, package="survival")#time in days#status censored=1, dead=2#sex:Male=1 Female=2library(ranger)
Attaching package: 'ranger'
The following object is masked from 'package:rattle':
importance
Boosting trees take a different approach from bagging by sequentially working on the weak learners to improve the residual of the model. This statement means that the model uses the features early and sequentially whereas the averaging from bagging is done towards the end.
10.3.2.1 Gradient Boost Machine
Gradient boost machine works by training weak learners and improves on the model at subsequent iteration. It is available from caret.
library(gbm)
Loaded gbm 2.1.8.1
#the breast cancer data from random forest is used here# specify that the resampling method is gbm_control <-trainControl(## 10-fold CVmethod ="repeatedcv",number =10)#scaling data is performed here under preProcess#note that ranger handles the outcome variable as factorgbm <- caret::train(Class ~ ., data = Train, method ="gbm",trControl=gbm_control,preProcess =c("center", "scale"),tuneLength =10)
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1522 nan 0.1000 0.0657
2 1.0452 nan 0.1000 0.0520
3 0.9465 nan 0.1000 0.0486
4 0.8699 nan 0.1000 0.0376
5 0.8031 nan 0.1000 0.0314
6 0.7480 nan 0.1000 0.0279
7 0.6949 nan 0.1000 0.0264
8 0.6500 nan 0.1000 0.0206
9 0.6110 nan 0.1000 0.0178
10 0.5772 nan 0.1000 0.0173
20 0.3674 nan 0.1000 0.0043
40 0.2328 nan 0.1000 -0.0004
60 0.1811 nan 0.1000 -0.0003
80 0.1553 nan 0.1000 -0.0002
100 0.1395 nan 0.1000 -0.0000
120 0.1248 nan 0.1000 -0.0004
140 0.1138 nan 0.1000 0.0003
160 0.1063 nan 0.1000 -0.0006
180 0.1042 nan 0.1000 -0.0003
200 0.0970 nan 0.1000 -0.0013
220 0.0955 nan 0.1000 -0.0010
240 0.0905 nan 0.1000 -0.0000
260 0.0870 nan 0.1000 -0.0001
280 0.0838 nan 0.1000 0.0003
300 0.0798 nan 0.1000 -0.0006
320 0.0756 nan 0.1000 -0.0002
340 0.0742 nan 0.1000 -0.0009
360 0.0718 nan 0.1000 -0.0008
380 0.0701 nan 0.1000 -0.0005
400 0.0692 nan 0.1000 -0.0012
420 0.0675 nan 0.1000 -0.0006
440 0.0662 nan 0.1000 -0.0006
460 0.0648 nan 0.1000 -0.0010
480 0.0633 nan 0.1000 -0.0004
500 0.0621 nan 0.1000 -0.0006
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1503 nan 0.1000 0.0720
2 1.0312 nan 0.1000 0.0637
3 0.9344 nan 0.1000 0.0459
4 0.8459 nan 0.1000 0.0426
5 0.7714 nan 0.1000 0.0350
6 0.7103 nan 0.1000 0.0297
7 0.6549 nan 0.1000 0.0252
8 0.6086 nan 0.1000 0.0224
9 0.5655 nan 0.1000 0.0203
10 0.5309 nan 0.1000 0.0142
20 0.3027 nan 0.1000 0.0035
40 0.1678 nan 0.1000 0.0004
60 0.1251 nan 0.1000 -0.0007
80 0.1006 nan 0.1000 -0.0000
100 0.0839 nan 0.1000 -0.0003
120 0.0722 nan 0.1000 -0.0004
140 0.0644 nan 0.1000 -0.0007
160 0.0563 nan 0.1000 0.0002
180 0.0519 nan 0.1000 -0.0002
200 0.0462 nan 0.1000 -0.0006
220 0.0417 nan 0.1000 -0.0006
240 0.0373 nan 0.1000 -0.0001
260 0.0346 nan 0.1000 -0.0006
280 0.0306 nan 0.1000 -0.0003
300 0.0281 nan 0.1000 -0.0002
320 0.0247 nan 0.1000 -0.0002
340 0.0232 nan 0.1000 -0.0002
360 0.0220 nan 0.1000 -0.0002
380 0.0202 nan 0.1000 -0.0002
400 0.0186 nan 0.1000 -0.0002
420 0.0169 nan 0.1000 -0.0002
440 0.0157 nan 0.1000 -0.0001
460 0.0148 nan 0.1000 -0.0002
480 0.0137 nan 0.1000 -0.0001
500 0.0124 nan 0.1000 -0.0001
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1380 nan 0.1000 0.0744
2 1.0108 nan 0.1000 0.0622
3 0.9064 nan 0.1000 0.0503
4 0.8191 nan 0.1000 0.0399
5 0.7521 nan 0.1000 0.0326
6 0.6867 nan 0.1000 0.0318
7 0.6333 nan 0.1000 0.0260
8 0.5844 nan 0.1000 0.0241
9 0.5419 nan 0.1000 0.0179
10 0.5079 nan 0.1000 0.0152
20 0.2765 nan 0.1000 0.0045
40 0.1446 nan 0.1000 -0.0006
60 0.0992 nan 0.1000 -0.0004
80 0.0768 nan 0.1000 -0.0003
100 0.0569 nan 0.1000 -0.0006
120 0.0445 nan 0.1000 -0.0003
140 0.0362 nan 0.1000 -0.0001
160 0.0296 nan 0.1000 -0.0004
180 0.0250 nan 0.1000 -0.0003
200 0.0216 nan 0.1000 -0.0003
220 0.0186 nan 0.1000 -0.0002
240 0.0160 nan 0.1000 -0.0002
260 0.0137 nan 0.1000 -0.0002
280 0.0120 nan 0.1000 -0.0001
300 0.0104 nan 0.1000 -0.0002
320 0.0088 nan 0.1000 0.0000
340 0.0076 nan 0.1000 -0.0001
360 0.0066 nan 0.1000 -0.0001
380 0.0054 nan 0.1000 -0.0000
400 0.0048 nan 0.1000 -0.0000
420 0.0042 nan 0.1000 -0.0001
440 0.0035 nan 0.1000 -0.0001
460 0.0030 nan 0.1000 -0.0000
480 0.0026 nan 0.1000 -0.0000
500 0.0023 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1380 nan 0.1000 0.0745
2 1.0151 nan 0.1000 0.0609
3 0.9111 nan 0.1000 0.0507
4 0.8246 nan 0.1000 0.0388
5 0.7497 nan 0.1000 0.0348
6 0.6838 nan 0.1000 0.0318
7 0.6301 nan 0.1000 0.0260
8 0.5768 nan 0.1000 0.0258
9 0.5342 nan 0.1000 0.0171
10 0.4962 nan 0.1000 0.0160
20 0.2672 nan 0.1000 0.0028
40 0.1320 nan 0.1000 -0.0007
60 0.0829 nan 0.1000 -0.0009
80 0.0599 nan 0.1000 -0.0007
100 0.0443 nan 0.1000 -0.0002
120 0.0336 nan 0.1000 -0.0005
140 0.0261 nan 0.1000 -0.0002
160 0.0201 nan 0.1000 -0.0002
180 0.0165 nan 0.1000 -0.0002
200 0.0123 nan 0.1000 -0.0000
220 0.0096 nan 0.1000 -0.0001
240 0.0078 nan 0.1000 -0.0000
260 0.0061 nan 0.1000 -0.0000
280 0.0050 nan 0.1000 -0.0000
300 0.0040 nan 0.1000 -0.0000
320 0.0032 nan 0.1000 -0.0000
340 0.0025 nan 0.1000 -0.0000
360 0.0019 nan 0.1000 -0.0000
380 0.0016 nan 0.1000 -0.0000
400 0.0013 nan 0.1000 -0.0000
420 0.0010 nan 0.1000 -0.0000
440 0.0009 nan 0.1000 -0.0000
460 0.0008 nan 0.1000 -0.0000
480 0.0007 nan 0.1000 -0.0000
500 0.0005 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1362 nan 0.1000 0.0781
2 1.0087 nan 0.1000 0.0624
3 0.8981 nan 0.1000 0.0531
4 0.8123 nan 0.1000 0.0414
5 0.7390 nan 0.1000 0.0332
6 0.6771 nan 0.1000 0.0292
7 0.6219 nan 0.1000 0.0254
8 0.5681 nan 0.1000 0.0241
9 0.5241 nan 0.1000 0.0211
10 0.4851 nan 0.1000 0.0180
20 0.2645 nan 0.1000 0.0039
40 0.1160 nan 0.1000 -0.0017
60 0.0678 nan 0.1000 0.0004
80 0.0452 nan 0.1000 -0.0000
100 0.0320 nan 0.1000 -0.0005
120 0.0241 nan 0.1000 -0.0002
140 0.0187 nan 0.1000 -0.0002
160 0.0139 nan 0.1000 -0.0001
180 0.0105 nan 0.1000 -0.0000
200 0.0078 nan 0.1000 -0.0001
220 0.0063 nan 0.1000 -0.0001
240 0.0045 nan 0.1000 -0.0000
260 0.0034 nan 0.1000 -0.0000
280 0.0027 nan 0.1000 -0.0000
300 0.0022 nan 0.1000 -0.0001
320 0.0019 nan 0.1000 -0.0000
340 0.0014 nan 0.1000 -0.0000
360 0.0011 nan 0.1000 -0.0000
380 0.0008 nan 0.1000 -0.0000
400 0.0006 nan 0.1000 -0.0000
420 0.0005 nan 0.1000 -0.0000
440 0.0004 nan 0.1000 -0.0000
460 0.0003 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0002 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1368 nan 0.1000 0.0740
2 1.0143 nan 0.1000 0.0581
3 0.9146 nan 0.1000 0.0474
4 0.8273 nan 0.1000 0.0431
5 0.7535 nan 0.1000 0.0343
6 0.6916 nan 0.1000 0.0293
7 0.6344 nan 0.1000 0.0249
8 0.5814 nan 0.1000 0.0239
9 0.5391 nan 0.1000 0.0180
10 0.5006 nan 0.1000 0.0150
20 0.2654 nan 0.1000 0.0034
40 0.1111 nan 0.1000 -0.0015
60 0.0611 nan 0.1000 -0.0007
80 0.0404 nan 0.1000 0.0000
100 0.0252 nan 0.1000 -0.0001
120 0.0189 nan 0.1000 -0.0006
140 0.0139 nan 0.1000 -0.0002
160 0.0103 nan 0.1000 -0.0001
180 0.0071 nan 0.1000 -0.0001
200 0.0053 nan 0.1000 -0.0000
220 0.0039 nan 0.1000 -0.0000
240 0.0027 nan 0.1000 0.0000
260 0.0020 nan 0.1000 -0.0000
280 0.0015 nan 0.1000 -0.0000
300 0.0010 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1362 nan 0.1000 0.0753
2 1.0097 nan 0.1000 0.0619
3 0.9048 nan 0.1000 0.0515
4 0.8157 nan 0.1000 0.0408
5 0.7377 nan 0.1000 0.0370
6 0.6757 nan 0.1000 0.0299
7 0.6180 nan 0.1000 0.0273
8 0.5687 nan 0.1000 0.0226
9 0.5281 nan 0.1000 0.0168
10 0.4904 nan 0.1000 0.0161
20 0.2650 nan 0.1000 0.0044
40 0.1114 nan 0.1000 -0.0012
60 0.0652 nan 0.1000 -0.0011
80 0.0388 nan 0.1000 -0.0002
100 0.0227 nan 0.1000 -0.0003
120 0.0159 nan 0.1000 -0.0003
140 0.0118 nan 0.1000 -0.0003
160 0.0084 nan 0.1000 -0.0001
180 0.0062 nan 0.1000 -0.0000
200 0.0046 nan 0.1000 -0.0001
220 0.0032 nan 0.1000 -0.0001
240 0.0023 nan 0.1000 -0.0000
260 0.0017 nan 0.1000 -0.0000
280 0.0010 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1338 nan 0.1000 0.0751
2 1.0053 nan 0.1000 0.0581
3 0.9014 nan 0.1000 0.0491
4 0.8204 nan 0.1000 0.0394
5 0.7485 nan 0.1000 0.0339
6 0.6793 nan 0.1000 0.0328
7 0.6213 nan 0.1000 0.0279
8 0.5710 nan 0.1000 0.0228
9 0.5266 nan 0.1000 0.0200
10 0.4876 nan 0.1000 0.0182
20 0.2600 nan 0.1000 0.0037
40 0.1073 nan 0.1000 0.0005
60 0.0607 nan 0.1000 -0.0004
80 0.0374 nan 0.1000 -0.0009
100 0.0245 nan 0.1000 -0.0000
120 0.0169 nan 0.1000 -0.0003
140 0.0099 nan 0.1000 -0.0002
160 0.0069 nan 0.1000 0.0000
180 0.0048 nan 0.1000 -0.0000
200 0.0038 nan 0.1000 -0.0001
220 0.0027 nan 0.1000 0.0000
240 0.0017 nan 0.1000 -0.0000
260 0.0012 nan 0.1000 -0.0000
280 0.0009 nan 0.1000 -0.0000
300 0.0007 nan 0.1000 -0.0000
320 0.0005 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1314 nan 0.1000 0.0719
2 1.0112 nan 0.1000 0.0607
3 0.9147 nan 0.1000 0.0448
4 0.8263 nan 0.1000 0.0416
5 0.7520 nan 0.1000 0.0375
6 0.6875 nan 0.1000 0.0291
7 0.6299 nan 0.1000 0.0272
8 0.5805 nan 0.1000 0.0224
9 0.5349 nan 0.1000 0.0193
10 0.4987 nan 0.1000 0.0153
20 0.2555 nan 0.1000 0.0051
40 0.0999 nan 0.1000 0.0014
60 0.0533 nan 0.1000 -0.0008
80 0.0325 nan 0.1000 -0.0004
100 0.0219 nan 0.1000 -0.0001
120 0.0159 nan 0.1000 0.0000
140 0.0099 nan 0.1000 -0.0003
160 0.0079 nan 0.1000 -0.0003
180 0.0054 nan 0.1000 -0.0002
200 0.0033 nan 0.1000 -0.0000
220 0.0022 nan 0.1000 -0.0000
240 0.0018 nan 0.1000 -0.0001
260 0.0013 nan 0.1000 -0.0000
280 0.0010 nan 0.1000 0.0000
300 0.0007 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0001 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1379 nan 0.1000 0.0704
2 1.0121 nan 0.1000 0.0599
3 0.9038 nan 0.1000 0.0500
4 0.8196 nan 0.1000 0.0421
5 0.7441 nan 0.1000 0.0339
6 0.6851 nan 0.1000 0.0267
7 0.6277 nan 0.1000 0.0271
8 0.5790 nan 0.1000 0.0206
9 0.5351 nan 0.1000 0.0204
10 0.4976 nan 0.1000 0.0161
20 0.2644 nan 0.1000 0.0048
40 0.1094 nan 0.1000 0.0002
60 0.0598 nan 0.1000 -0.0009
80 0.0374 nan 0.1000 0.0001
100 0.0245 nan 0.1000 -0.0002
120 0.0168 nan 0.1000 -0.0003
140 0.0118 nan 0.1000 -0.0003
160 0.0087 nan 0.1000 -0.0002
180 0.0062 nan 0.1000 0.0000
200 0.0043 nan 0.1000 -0.0000
220 0.0030 nan 0.1000 -0.0000
240 0.0021 nan 0.1000 0.0000
260 0.0016 nan 0.1000 -0.0000
280 0.0012 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1483 nan 0.1000 0.0660
2 1.0427 nan 0.1000 0.0513
3 0.9512 nan 0.1000 0.0461
4 0.8777 nan 0.1000 0.0373
5 0.8136 nan 0.1000 0.0320
6 0.7544 nan 0.1000 0.0285
7 0.7039 nan 0.1000 0.0226
8 0.6653 nan 0.1000 0.0171
9 0.6268 nan 0.1000 0.0173
10 0.5916 nan 0.1000 0.0158
20 0.3829 nan 0.1000 0.0056
40 0.2474 nan 0.1000 0.0008
60 0.2045 nan 0.1000 0.0008
80 0.1797 nan 0.1000 -0.0003
100 0.1651 nan 0.1000 -0.0008
120 0.1514 nan 0.1000 -0.0006
140 0.1451 nan 0.1000 -0.0007
160 0.1360 nan 0.1000 -0.0004
180 0.1307 nan 0.1000 -0.0002
200 0.1278 nan 0.1000 -0.0006
220 0.1229 nan 0.1000 -0.0018
240 0.1167 nan 0.1000 -0.0009
260 0.1144 nan 0.1000 -0.0021
280 0.1119 nan 0.1000 -0.0006
300 0.1065 nan 0.1000 -0.0010
320 0.1034 nan 0.1000 -0.0010
340 0.0993 nan 0.1000 -0.0008
360 0.0994 nan 0.1000 -0.0019
380 0.0979 nan 0.1000 -0.0005
400 0.0972 nan 0.1000 -0.0005
420 0.0957 nan 0.1000 -0.0008
440 0.0936 nan 0.1000 -0.0011
460 0.0926 nan 0.1000 -0.0005
480 0.0923 nan 0.1000 -0.0004
500 0.0918 nan 0.1000 -0.0006
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1342 nan 0.1000 0.0714
2 1.0162 nan 0.1000 0.0592
3 0.9188 nan 0.1000 0.0460
4 0.8413 nan 0.1000 0.0370
5 0.7696 nan 0.1000 0.0350
6 0.7082 nan 0.1000 0.0279
7 0.6599 nan 0.1000 0.0223
8 0.6125 nan 0.1000 0.0225
9 0.5724 nan 0.1000 0.0190
10 0.5379 nan 0.1000 0.0143
20 0.3182 nan 0.1000 0.0055
40 0.1896 nan 0.1000 -0.0005
60 0.1514 nan 0.1000 -0.0008
80 0.1264 nan 0.1000 -0.0013
100 0.1106 nan 0.1000 -0.0003
120 0.0989 nan 0.1000 -0.0006
140 0.0863 nan 0.1000 -0.0004
160 0.0755 nan 0.1000 -0.0004
180 0.0664 nan 0.1000 -0.0003
200 0.0601 nan 0.1000 -0.0004
220 0.0549 nan 0.1000 -0.0003
240 0.0501 nan 0.1000 -0.0004
260 0.0459 nan 0.1000 -0.0005
280 0.0414 nan 0.1000 -0.0003
300 0.0383 nan 0.1000 -0.0004
320 0.0353 nan 0.1000 -0.0003
340 0.0323 nan 0.1000 -0.0002
360 0.0296 nan 0.1000 -0.0002
380 0.0279 nan 0.1000 -0.0002
400 0.0265 nan 0.1000 -0.0002
420 0.0241 nan 0.1000 -0.0001
440 0.0222 nan 0.1000 -0.0001
460 0.0201 nan 0.1000 -0.0002
480 0.0185 nan 0.1000 -0.0002
500 0.0164 nan 0.1000 -0.0001
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1377 nan 0.1000 0.0749
2 1.0149 nan 0.1000 0.0601
3 0.9131 nan 0.1000 0.0486
4 0.8280 nan 0.1000 0.0357
5 0.7546 nan 0.1000 0.0372
6 0.6949 nan 0.1000 0.0270
7 0.6405 nan 0.1000 0.0255
8 0.5907 nan 0.1000 0.0226
9 0.5499 nan 0.1000 0.0184
10 0.5107 nan 0.1000 0.0177
20 0.2912 nan 0.1000 0.0062
40 0.1619 nan 0.1000 -0.0012
60 0.1097 nan 0.1000 0.0009
80 0.0842 nan 0.1000 -0.0006
100 0.0684 nan 0.1000 -0.0010
120 0.0564 nan 0.1000 -0.0008
140 0.0471 nan 0.1000 -0.0003
160 0.0407 nan 0.1000 -0.0004
180 0.0319 nan 0.1000 -0.0001
200 0.0251 nan 0.1000 -0.0001
220 0.0206 nan 0.1000 -0.0001
240 0.0177 nan 0.1000 -0.0002
260 0.0156 nan 0.1000 -0.0001
280 0.0134 nan 0.1000 -0.0001
300 0.0115 nan 0.1000 -0.0001
320 0.0101 nan 0.1000 -0.0001
340 0.0088 nan 0.1000 -0.0001
360 0.0076 nan 0.1000 -0.0001
380 0.0065 nan 0.1000 -0.0001
400 0.0056 nan 0.1000 -0.0000
420 0.0048 nan 0.1000 -0.0000
440 0.0042 nan 0.1000 -0.0000
460 0.0037 nan 0.1000 -0.0000
480 0.0032 nan 0.1000 -0.0000
500 0.0028 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1268 nan 0.1000 0.0769
2 1.0068 nan 0.1000 0.0570
3 0.9047 nan 0.1000 0.0476
4 0.8201 nan 0.1000 0.0398
5 0.7439 nan 0.1000 0.0367
6 0.6816 nan 0.1000 0.0286
7 0.6296 nan 0.1000 0.0219
8 0.5793 nan 0.1000 0.0221
9 0.5404 nan 0.1000 0.0165
10 0.5041 nan 0.1000 0.0171
20 0.2851 nan 0.1000 0.0053
40 0.1446 nan 0.1000 0.0006
60 0.0930 nan 0.1000 0.0005
80 0.0681 nan 0.1000 -0.0009
100 0.0512 nan 0.1000 -0.0005
120 0.0398 nan 0.1000 -0.0002
140 0.0323 nan 0.1000 -0.0002
160 0.0259 nan 0.1000 -0.0000
180 0.0222 nan 0.1000 -0.0004
200 0.0176 nan 0.1000 -0.0001
220 0.0151 nan 0.1000 -0.0002
240 0.0125 nan 0.1000 -0.0000
260 0.0107 nan 0.1000 -0.0001
280 0.0081 nan 0.1000 -0.0000
300 0.0063 nan 0.1000 -0.0000
320 0.0052 nan 0.1000 -0.0001
340 0.0042 nan 0.1000 -0.0000
360 0.0035 nan 0.1000 -0.0000
380 0.0029 nan 0.1000 -0.0000
400 0.0025 nan 0.1000 -0.0001
420 0.0020 nan 0.1000 -0.0000
440 0.0016 nan 0.1000 -0.0000
460 0.0013 nan 0.1000 -0.0000
480 0.0011 nan 0.1000 -0.0000
500 0.0009 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1352 nan 0.1000 0.0716
2 1.0122 nan 0.1000 0.0583
3 0.9094 nan 0.1000 0.0471
4 0.8230 nan 0.1000 0.0373
5 0.7545 nan 0.1000 0.0325
6 0.6911 nan 0.1000 0.0294
7 0.6316 nan 0.1000 0.0252
8 0.5823 nan 0.1000 0.0214
9 0.5387 nan 0.1000 0.0202
10 0.5056 nan 0.1000 0.0130
20 0.2824 nan 0.1000 0.0043
40 0.1389 nan 0.1000 -0.0007
60 0.0873 nan 0.1000 -0.0008
80 0.0552 nan 0.1000 -0.0003
100 0.0419 nan 0.1000 -0.0006
120 0.0289 nan 0.1000 -0.0005
140 0.0218 nan 0.1000 -0.0002
160 0.0169 nan 0.1000 -0.0001
180 0.0129 nan 0.1000 -0.0002
200 0.0104 nan 0.1000 -0.0002
220 0.0080 nan 0.1000 -0.0001
240 0.0059 nan 0.1000 -0.0001
260 0.0045 nan 0.1000 -0.0001
280 0.0036 nan 0.1000 0.0000
300 0.0029 nan 0.1000 -0.0000
320 0.0020 nan 0.1000 -0.0000
340 0.0016 nan 0.1000 -0.0000
360 0.0013 nan 0.1000 -0.0000
380 0.0010 nan 0.1000 -0.0000
400 0.0008 nan 0.1000 0.0000
420 0.0006 nan 0.1000 -0.0000
440 0.0005 nan 0.1000 -0.0000
460 0.0004 nan 0.1000 -0.0000
480 0.0003 nan 0.1000 0.0000
500 0.0003 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1333 nan 0.1000 0.0751
2 1.0092 nan 0.1000 0.0595
3 0.9123 nan 0.1000 0.0440
4 0.8287 nan 0.1000 0.0417
5 0.7558 nan 0.1000 0.0347
6 0.6900 nan 0.1000 0.0303
7 0.6355 nan 0.1000 0.0240
8 0.5864 nan 0.1000 0.0234
9 0.5414 nan 0.1000 0.0224
10 0.5033 nan 0.1000 0.0157
20 0.2783 nan 0.1000 0.0038
40 0.1241 nan 0.1000 0.0006
60 0.0721 nan 0.1000 -0.0009
80 0.0503 nan 0.1000 -0.0013
100 0.0384 nan 0.1000 -0.0006
120 0.0283 nan 0.1000 -0.0006
140 0.0206 nan 0.1000 -0.0002
160 0.0140 nan 0.1000 -0.0002
180 0.0105 nan 0.1000 -0.0002
200 0.0077 nan 0.1000 -0.0001
220 0.0060 nan 0.1000 -0.0002
240 0.0044 nan 0.1000 -0.0001
260 0.0032 nan 0.1000 -0.0000
280 0.0026 nan 0.1000 -0.0000
300 0.0018 nan 0.1000 -0.0000
320 0.0015 nan 0.1000 -0.0000
340 0.0011 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1362 nan 0.1000 0.0723
2 1.0098 nan 0.1000 0.0587
3 0.9041 nan 0.1000 0.0466
4 0.8182 nan 0.1000 0.0382
5 0.7424 nan 0.1000 0.0329
6 0.6810 nan 0.1000 0.0280
7 0.6232 nan 0.1000 0.0262
8 0.5743 nan 0.1000 0.0235
9 0.5331 nan 0.1000 0.0177
10 0.4973 nan 0.1000 0.0146
20 0.2697 nan 0.1000 0.0046
40 0.1188 nan 0.1000 -0.0001
60 0.0684 nan 0.1000 -0.0007
80 0.0465 nan 0.1000 -0.0004
100 0.0302 nan 0.1000 -0.0011
120 0.0219 nan 0.1000 -0.0004
140 0.0152 nan 0.1000 -0.0000
160 0.0094 nan 0.1000 -0.0000
180 0.0073 nan 0.1000 -0.0001
200 0.0050 nan 0.1000 -0.0001
220 0.0039 nan 0.1000 -0.0001
240 0.0030 nan 0.1000 -0.0001
260 0.0023 nan 0.1000 -0.0000
280 0.0019 nan 0.1000 -0.0001
300 0.0013 nan 0.1000 -0.0000
320 0.0010 nan 0.1000 0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1354 nan 0.1000 0.0743
2 1.0105 nan 0.1000 0.0568
3 0.9111 nan 0.1000 0.0497
4 0.8276 nan 0.1000 0.0390
5 0.7478 nan 0.1000 0.0355
6 0.6883 nan 0.1000 0.0266
7 0.6312 nan 0.1000 0.0261
8 0.5831 nan 0.1000 0.0207
9 0.5384 nan 0.1000 0.0191
10 0.4979 nan 0.1000 0.0182
20 0.2739 nan 0.1000 0.0052
40 0.1147 nan 0.1000 -0.0002
60 0.0631 nan 0.1000 -0.0010
80 0.0405 nan 0.1000 -0.0006
100 0.0281 nan 0.1000 -0.0005
120 0.0199 nan 0.1000 -0.0002
140 0.0140 nan 0.1000 -0.0001
160 0.0092 nan 0.1000 -0.0002
180 0.0076 nan 0.1000 -0.0001
200 0.0055 nan 0.1000 -0.0001
220 0.0037 nan 0.1000 -0.0001
240 0.0031 nan 0.1000 -0.0001
260 0.0022 nan 0.1000 -0.0001
280 0.0015 nan 0.1000 -0.0000
300 0.0011 nan 0.1000 -0.0000
320 0.0008 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1390 nan 0.1000 0.0741
2 1.0180 nan 0.1000 0.0554
3 0.9125 nan 0.1000 0.0502
4 0.8252 nan 0.1000 0.0412
5 0.7554 nan 0.1000 0.0328
6 0.6921 nan 0.1000 0.0283
7 0.6320 nan 0.1000 0.0265
8 0.5874 nan 0.1000 0.0192
9 0.5427 nan 0.1000 0.0184
10 0.4980 nan 0.1000 0.0204
20 0.2686 nan 0.1000 0.0047
40 0.1162 nan 0.1000 0.0004
60 0.0619 nan 0.1000 -0.0006
80 0.0402 nan 0.1000 -0.0001
100 0.0251 nan 0.1000 -0.0005
120 0.0171 nan 0.1000 -0.0002
140 0.0119 nan 0.1000 -0.0001
160 0.0089 nan 0.1000 -0.0002
180 0.0062 nan 0.1000 -0.0001
200 0.0047 nan 0.1000 -0.0001
220 0.0039 nan 0.1000 -0.0001
240 0.0030 nan 0.1000 -0.0000
260 0.0022 nan 0.1000 -0.0000
280 0.0016 nan 0.1000 -0.0000
300 0.0011 nan 0.1000 -0.0000
320 0.0008 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1418 nan 0.1000 0.0772
2 1.0167 nan 0.1000 0.0577
3 0.9129 nan 0.1000 0.0548
4 0.8249 nan 0.1000 0.0420
5 0.7461 nan 0.1000 0.0346
6 0.6801 nan 0.1000 0.0313
7 0.6274 nan 0.1000 0.0240
8 0.5797 nan 0.1000 0.0199
9 0.5373 nan 0.1000 0.0196
10 0.4990 nan 0.1000 0.0156
20 0.2792 nan 0.1000 0.0049
40 0.1227 nan 0.1000 -0.0002
60 0.0641 nan 0.1000 -0.0010
80 0.0406 nan 0.1000 -0.0008
100 0.0251 nan 0.1000 -0.0006
120 0.0175 nan 0.1000 -0.0001
140 0.0115 nan 0.1000 -0.0001
160 0.0081 nan 0.1000 -0.0001
180 0.0068 nan 0.1000 -0.0002
200 0.0052 nan 0.1000 -0.0002
220 0.0039 nan 0.1000 -0.0000
240 0.0031 nan 0.1000 -0.0000
260 0.0021 nan 0.1000 -0.0000
280 0.0015 nan 0.1000 -0.0000
300 0.0012 nan 0.1000 -0.0000
320 0.0009 nan 0.1000 -0.0000
340 0.0009 nan 0.1000 -0.0000
360 0.0009 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1613 nan 0.1000 0.0631
2 1.0553 nan 0.1000 0.0510
3 0.9610 nan 0.1000 0.0491
4 0.8846 nan 0.1000 0.0379
5 0.8118 nan 0.1000 0.0345
6 0.7602 nan 0.1000 0.0245
7 0.7073 nan 0.1000 0.0259
8 0.6631 nan 0.1000 0.0225
9 0.6241 nan 0.1000 0.0169
10 0.5903 nan 0.1000 0.0164
20 0.3859 nan 0.1000 0.0043
40 0.2497 nan 0.1000 0.0005
60 0.2098 nan 0.1000 -0.0008
80 0.1830 nan 0.1000 -0.0007
100 0.1686 nan 0.1000 0.0005
120 0.1521 nan 0.1000 -0.0010
140 0.1449 nan 0.1000 -0.0005
160 0.1401 nan 0.1000 -0.0011
180 0.1351 nan 0.1000 -0.0015
200 0.1268 nan 0.1000 -0.0006
220 0.1199 nan 0.1000 -0.0009
240 0.1158 nan 0.1000 -0.0002
260 0.1141 nan 0.1000 -0.0010
280 0.1134 nan 0.1000 -0.0014
300 0.1097 nan 0.1000 -0.0008
320 0.1084 nan 0.1000 -0.0013
340 0.1054 nan 0.1000 -0.0008
360 0.1041 nan 0.1000 -0.0016
380 0.1037 nan 0.1000 -0.0007
400 0.0989 nan 0.1000 -0.0002
420 0.0985 nan 0.1000 -0.0004
440 0.0962 nan 0.1000 -0.0007
460 0.0965 nan 0.1000 -0.0014
480 0.0955 nan 0.1000 -0.0010
500 0.0944 nan 0.1000 -0.0005
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1341 nan 0.1000 0.0735
2 1.0147 nan 0.1000 0.0594
3 0.9153 nan 0.1000 0.0488
4 0.8325 nan 0.1000 0.0409
5 0.7605 nan 0.1000 0.0346
6 0.6995 nan 0.1000 0.0300
7 0.6503 nan 0.1000 0.0233
8 0.6041 nan 0.1000 0.0213
9 0.5624 nan 0.1000 0.0186
10 0.5236 nan 0.1000 0.0182
20 0.3140 nan 0.1000 0.0027
40 0.1830 nan 0.1000 -0.0013
60 0.1434 nan 0.1000 0.0002
80 0.1211 nan 0.1000 -0.0006
100 0.1018 nan 0.1000 -0.0007
120 0.0883 nan 0.1000 -0.0012
140 0.0757 nan 0.1000 -0.0011
160 0.0666 nan 0.1000 -0.0011
180 0.0591 nan 0.1000 -0.0001
200 0.0528 nan 0.1000 -0.0004
220 0.0466 nan 0.1000 -0.0004
240 0.0430 nan 0.1000 -0.0002
260 0.0387 nan 0.1000 0.0000
280 0.0352 nan 0.1000 -0.0005
300 0.0315 nan 0.1000 -0.0003
320 0.0282 nan 0.1000 -0.0004
340 0.0253 nan 0.1000 -0.0001
360 0.0235 nan 0.1000 -0.0002
380 0.0212 nan 0.1000 -0.0001
400 0.0190 nan 0.1000 -0.0000
420 0.0171 nan 0.1000 -0.0001
440 0.0162 nan 0.1000 -0.0001
460 0.0152 nan 0.1000 -0.0002
480 0.0143 nan 0.1000 -0.0002
500 0.0133 nan 0.1000 -0.0001
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1365 nan 0.1000 0.0771
2 1.0120 nan 0.1000 0.0640
3 0.9064 nan 0.1000 0.0484
4 0.8233 nan 0.1000 0.0408
5 0.7522 nan 0.1000 0.0344
6 0.6931 nan 0.1000 0.0284
7 0.6385 nan 0.1000 0.0260
8 0.5885 nan 0.1000 0.0225
9 0.5415 nan 0.1000 0.0205
10 0.5010 nan 0.1000 0.0176
20 0.2841 nan 0.1000 0.0055
40 0.1578 nan 0.1000 -0.0015
60 0.1146 nan 0.1000 -0.0010
80 0.0890 nan 0.1000 0.0001
100 0.0688 nan 0.1000 -0.0005
120 0.0555 nan 0.1000 -0.0006
140 0.0441 nan 0.1000 0.0000
160 0.0351 nan 0.1000 -0.0003
180 0.0279 nan 0.1000 -0.0002
200 0.0236 nan 0.1000 -0.0002
220 0.0208 nan 0.1000 -0.0002
240 0.0175 nan 0.1000 -0.0001
260 0.0150 nan 0.1000 -0.0002
280 0.0130 nan 0.1000 -0.0002
300 0.0107 nan 0.1000 -0.0001
320 0.0093 nan 0.1000 -0.0002
340 0.0081 nan 0.1000 -0.0000
360 0.0071 nan 0.1000 -0.0000
380 0.0061 nan 0.1000 -0.0001
400 0.0051 nan 0.1000 -0.0001
420 0.0044 nan 0.1000 -0.0000
440 0.0037 nan 0.1000 -0.0001
460 0.0032 nan 0.1000 -0.0001
480 0.0028 nan 0.1000 -0.0000
500 0.0023 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1443 nan 0.1000 0.0686
2 1.0194 nan 0.1000 0.0593
3 0.9210 nan 0.1000 0.0477
4 0.8396 nan 0.1000 0.0388
5 0.7628 nan 0.1000 0.0363
6 0.6994 nan 0.1000 0.0309
7 0.6436 nan 0.1000 0.0246
8 0.5874 nan 0.1000 0.0243
9 0.5467 nan 0.1000 0.0193
10 0.5093 nan 0.1000 0.0171
20 0.2871 nan 0.1000 0.0045
40 0.1422 nan 0.1000 0.0013
60 0.0895 nan 0.1000 -0.0008
80 0.0663 nan 0.1000 -0.0008
100 0.0490 nan 0.1000 -0.0002
120 0.0392 nan 0.1000 -0.0004
140 0.0319 nan 0.1000 -0.0004
160 0.0252 nan 0.1000 -0.0005
180 0.0190 nan 0.1000 -0.0003
200 0.0148 nan 0.1000 -0.0002
220 0.0117 nan 0.1000 -0.0001
240 0.0097 nan 0.1000 -0.0002
260 0.0074 nan 0.1000 -0.0001
280 0.0060 nan 0.1000 -0.0001
300 0.0048 nan 0.1000 -0.0000
320 0.0041 nan 0.1000 -0.0000
340 0.0032 nan 0.1000 -0.0000
360 0.0026 nan 0.1000 -0.0000
380 0.0022 nan 0.1000 -0.0000
400 0.0019 nan 0.1000 -0.0000
420 0.0015 nan 0.1000 -0.0000
440 0.0013 nan 0.1000 -0.0000
460 0.0011 nan 0.1000 -0.0000
480 0.0009 nan 0.1000 -0.0000
500 0.0007 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1342 nan 0.1000 0.0735
2 1.0106 nan 0.1000 0.0598
3 0.9114 nan 0.1000 0.0471
4 0.8277 nan 0.1000 0.0376
5 0.7512 nan 0.1000 0.0386
6 0.6900 nan 0.1000 0.0307
7 0.6335 nan 0.1000 0.0289
8 0.5867 nan 0.1000 0.0204
9 0.5398 nan 0.1000 0.0214
10 0.5033 nan 0.1000 0.0162
20 0.2769 nan 0.1000 0.0036
40 0.1218 nan 0.1000 -0.0012
60 0.0752 nan 0.1000 -0.0007
80 0.0535 nan 0.1000 -0.0010
100 0.0382 nan 0.1000 -0.0008
120 0.0290 nan 0.1000 -0.0004
140 0.0206 nan 0.1000 -0.0002
160 0.0168 nan 0.1000 -0.0003
180 0.0120 nan 0.1000 -0.0001
200 0.0086 nan 0.1000 -0.0000
220 0.0067 nan 0.1000 -0.0001
240 0.0050 nan 0.1000 -0.0000
260 0.0040 nan 0.1000 -0.0000
280 0.0031 nan 0.1000 -0.0001
300 0.0023 nan 0.1000 -0.0000
320 0.0017 nan 0.1000 -0.0000
340 0.0013 nan 0.1000 -0.0000
360 0.0010 nan 0.1000 -0.0000
380 0.0007 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0005 nan 0.1000 -0.0000
440 0.0004 nan 0.1000 -0.0000
460 0.0003 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0002 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1350 nan 0.1000 0.0694
2 1.0050 nan 0.1000 0.0577
3 0.8994 nan 0.1000 0.0495
4 0.8119 nan 0.1000 0.0426
5 0.7381 nan 0.1000 0.0333
6 0.6721 nan 0.1000 0.0318
7 0.6145 nan 0.1000 0.0228
8 0.5624 nan 0.1000 0.0232
9 0.5172 nan 0.1000 0.0190
10 0.4836 nan 0.1000 0.0127
20 0.2649 nan 0.1000 0.0023
40 0.1197 nan 0.1000 -0.0006
60 0.0728 nan 0.1000 -0.0016
80 0.0462 nan 0.1000 -0.0005
100 0.0326 nan 0.1000 -0.0001
120 0.0211 nan 0.1000 -0.0001
140 0.0155 nan 0.1000 -0.0001
160 0.0109 nan 0.1000 -0.0001
180 0.0082 nan 0.1000 -0.0001
200 0.0060 nan 0.1000 -0.0001
220 0.0044 nan 0.1000 -0.0000
240 0.0033 nan 0.1000 -0.0000
260 0.0024 nan 0.1000 -0.0000
280 0.0018 nan 0.1000 -0.0001
300 0.0014 nan 0.1000 -0.0000
320 0.0010 nan 0.1000 -0.0000
340 0.0008 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0005 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0003 nan 0.1000 -0.0000
480 0.0003 nan 0.1000 -0.0000
500 0.0002 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1292 nan 0.1000 0.0731
2 1.0025 nan 0.1000 0.0615
3 0.9037 nan 0.1000 0.0462
4 0.8177 nan 0.1000 0.0404
5 0.7445 nan 0.1000 0.0367
6 0.6832 nan 0.1000 0.0286
7 0.6326 nan 0.1000 0.0208
8 0.5858 nan 0.1000 0.0205
9 0.5412 nan 0.1000 0.0186
10 0.5031 nan 0.1000 0.0146
20 0.2703 nan 0.1000 0.0038
40 0.1143 nan 0.1000 -0.0011
60 0.0623 nan 0.1000 -0.0003
80 0.0390 nan 0.1000 -0.0003
100 0.0260 nan 0.1000 -0.0004
120 0.0172 nan 0.1000 -0.0003
140 0.0120 nan 0.1000 -0.0000
160 0.0086 nan 0.1000 -0.0001
180 0.0055 nan 0.1000 -0.0000
200 0.0043 nan 0.1000 -0.0001
220 0.0031 nan 0.1000 -0.0000
240 0.0025 nan 0.1000 -0.0001
260 0.0015 nan 0.1000 -0.0000
280 0.0012 nan 0.1000 -0.0000
300 0.0009 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0006 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1288 nan 0.1000 0.0711
2 1.0064 nan 0.1000 0.0595
3 0.9036 nan 0.1000 0.0466
4 0.8216 nan 0.1000 0.0395
5 0.7497 nan 0.1000 0.0338
6 0.6853 nan 0.1000 0.0298
7 0.6338 nan 0.1000 0.0237
8 0.5863 nan 0.1000 0.0214
9 0.5397 nan 0.1000 0.0209
10 0.5025 nan 0.1000 0.0160
20 0.2693 nan 0.1000 0.0062
40 0.1143 nan 0.1000 0.0008
60 0.0619 nan 0.1000 -0.0014
80 0.0368 nan 0.1000 -0.0005
100 0.0252 nan 0.1000 -0.0001
120 0.0171 nan 0.1000 -0.0003
140 0.0129 nan 0.1000 -0.0004
160 0.0089 nan 0.1000 0.0000
180 0.0064 nan 0.1000 -0.0001
200 0.0049 nan 0.1000 -0.0001
220 0.0039 nan 0.1000 0.0000
240 0.0027 nan 0.1000 -0.0000
260 0.0021 nan 0.1000 -0.0000
280 0.0017 nan 0.1000 -0.0001
300 0.0012 nan 0.1000 -0.0000
320 0.0008 nan 0.1000 0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1356 nan 0.1000 0.0751
2 1.0164 nan 0.1000 0.0600
3 0.9132 nan 0.1000 0.0486
4 0.8236 nan 0.1000 0.0396
5 0.7520 nan 0.1000 0.0330
6 0.6900 nan 0.1000 0.0283
7 0.6317 nan 0.1000 0.0274
8 0.5851 nan 0.1000 0.0220
9 0.5400 nan 0.1000 0.0186
10 0.4985 nan 0.1000 0.0174
20 0.2714 nan 0.1000 0.0051
40 0.1090 nan 0.1000 -0.0012
60 0.0593 nan 0.1000 -0.0007
80 0.0396 nan 0.1000 -0.0001
100 0.0256 nan 0.1000 0.0001
120 0.0192 nan 0.1000 -0.0003
140 0.0143 nan 0.1000 -0.0002
160 0.0103 nan 0.1000 -0.0002
180 0.0074 nan 0.1000 -0.0003
200 0.0045 nan 0.1000 -0.0001
220 0.0029 nan 0.1000 -0.0001
240 0.0022 nan 0.1000 -0.0000
260 0.0014 nan 0.1000 -0.0000
280 0.0010 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0006 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1309 nan 0.1000 0.0724
2 1.0101 nan 0.1000 0.0554
3 0.9109 nan 0.1000 0.0475
4 0.8252 nan 0.1000 0.0398
5 0.7524 nan 0.1000 0.0328
6 0.6917 nan 0.1000 0.0266
7 0.6385 nan 0.1000 0.0237
8 0.5857 nan 0.1000 0.0225
9 0.5416 nan 0.1000 0.0199
10 0.5062 nan 0.1000 0.0146
20 0.2705 nan 0.1000 0.0051
40 0.1151 nan 0.1000 0.0000
60 0.0600 nan 0.1000 -0.0006
80 0.0352 nan 0.1000 -0.0003
100 0.0216 nan 0.1000 -0.0003
120 0.0140 nan 0.1000 -0.0001
140 0.0099 nan 0.1000 -0.0002
160 0.0067 nan 0.1000 -0.0000
180 0.0058 nan 0.1000 -0.0002
200 0.0035 nan 0.1000 -0.0000
220 0.0025 nan 0.1000 -0.0000
240 0.0021 nan 0.1000 -0.0001
260 0.0015 nan 0.1000 0.0000
280 0.0013 nan 0.1000 -0.0000
300 0.0011 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 0.0000
380 0.0003 nan 0.1000 0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1529 nan 0.1000 0.0627
2 1.0422 nan 0.1000 0.0535
3 0.9454 nan 0.1000 0.0485
4 0.8647 nan 0.1000 0.0404
5 0.7992 nan 0.1000 0.0333
6 0.7419 nan 0.1000 0.0277
7 0.6939 nan 0.1000 0.0234
8 0.6501 nan 0.1000 0.0194
9 0.6073 nan 0.1000 0.0196
10 0.5718 nan 0.1000 0.0159
20 0.3737 nan 0.1000 0.0059
40 0.2386 nan 0.1000 -0.0002
60 0.1871 nan 0.1000 0.0011
80 0.1652 nan 0.1000 -0.0006
100 0.1487 nan 0.1000 -0.0011
120 0.1391 nan 0.1000 -0.0023
140 0.1302 nan 0.1000 -0.0004
160 0.1236 nan 0.1000 0.0001
180 0.1141 nan 0.1000 -0.0001
200 0.1080 nan 0.1000 -0.0010
220 0.1053 nan 0.1000 -0.0009
240 0.0989 nan 0.1000 -0.0007
260 0.0958 nan 0.1000 -0.0008
280 0.0936 nan 0.1000 -0.0005
300 0.0901 nan 0.1000 -0.0005
320 0.0882 nan 0.1000 -0.0003
340 0.0873 nan 0.1000 -0.0010
360 0.0851 nan 0.1000 -0.0006
380 0.0834 nan 0.1000 -0.0007
400 0.0812 nan 0.1000 -0.0009
420 0.0776 nan 0.1000 -0.0010
440 0.0764 nan 0.1000 -0.0011
460 0.0758 nan 0.1000 -0.0003
480 0.0747 nan 0.1000 -0.0005
500 0.0737 nan 0.1000 -0.0002
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1456 nan 0.1000 0.0697
2 1.0206 nan 0.1000 0.0677
3 0.9204 nan 0.1000 0.0506
4 0.8354 nan 0.1000 0.0393
5 0.7583 nan 0.1000 0.0349
6 0.6986 nan 0.1000 0.0282
7 0.6396 nan 0.1000 0.0269
8 0.5938 nan 0.1000 0.0225
9 0.5520 nan 0.1000 0.0188
10 0.5137 nan 0.1000 0.0158
20 0.2921 nan 0.1000 0.0064
40 0.1787 nan 0.1000 -0.0003
60 0.1342 nan 0.1000 -0.0001
80 0.1113 nan 0.1000 -0.0004
100 0.0922 nan 0.1000 -0.0005
120 0.0789 nan 0.1000 -0.0002
140 0.0658 nan 0.1000 -0.0010
160 0.0585 nan 0.1000 -0.0008
180 0.0517 nan 0.1000 -0.0008
200 0.0460 nan 0.1000 -0.0005
220 0.0401 nan 0.1000 -0.0002
240 0.0358 nan 0.1000 -0.0001
260 0.0340 nan 0.1000 -0.0007
280 0.0308 nan 0.1000 -0.0001
300 0.0277 nan 0.1000 -0.0003
320 0.0253 nan 0.1000 -0.0003
340 0.0230 nan 0.1000 -0.0001
360 0.0211 nan 0.1000 -0.0001
380 0.0193 nan 0.1000 -0.0003
400 0.0178 nan 0.1000 -0.0002
420 0.0159 nan 0.1000 -0.0003
440 0.0147 nan 0.1000 -0.0000
460 0.0133 nan 0.1000 -0.0001
480 0.0123 nan 0.1000 -0.0001
500 0.0109 nan 0.1000 -0.0001
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1408 nan 0.1000 0.0641
2 1.0134 nan 0.1000 0.0611
3 0.9112 nan 0.1000 0.0464
4 0.8236 nan 0.1000 0.0396
5 0.7548 nan 0.1000 0.0329
6 0.6971 nan 0.1000 0.0272
7 0.6442 nan 0.1000 0.0242
8 0.5944 nan 0.1000 0.0237
9 0.5523 nan 0.1000 0.0188
10 0.5140 nan 0.1000 0.0177
20 0.2908 nan 0.1000 0.0052
40 0.1461 nan 0.1000 0.0002
60 0.1002 nan 0.1000 -0.0002
80 0.0739 nan 0.1000 -0.0006
100 0.0591 nan 0.1000 -0.0002
120 0.0462 nan 0.1000 -0.0001
140 0.0369 nan 0.1000 -0.0004
160 0.0310 nan 0.1000 -0.0003
180 0.0269 nan 0.1000 -0.0005
200 0.0222 nan 0.1000 -0.0001
220 0.0182 nan 0.1000 0.0001
240 0.0154 nan 0.1000 -0.0001
260 0.0131 nan 0.1000 -0.0001
280 0.0113 nan 0.1000 -0.0000
300 0.0099 nan 0.1000 -0.0001
320 0.0084 nan 0.1000 -0.0001
340 0.0072 nan 0.1000 -0.0000
360 0.0061 nan 0.1000 -0.0000
380 0.0055 nan 0.1000 -0.0000
400 0.0047 nan 0.1000 -0.0000
420 0.0038 nan 0.1000 -0.0000
440 0.0033 nan 0.1000 -0.0000
460 0.0028 nan 0.1000 -0.0000
480 0.0024 nan 0.1000 0.0000
500 0.0021 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1331 nan 0.1000 0.0773
2 1.0115 nan 0.1000 0.0621
3 0.9083 nan 0.1000 0.0462
4 0.8296 nan 0.1000 0.0383
5 0.7528 nan 0.1000 0.0336
6 0.6912 nan 0.1000 0.0256
7 0.6359 nan 0.1000 0.0249
8 0.5833 nan 0.1000 0.0243
9 0.5405 nan 0.1000 0.0164
10 0.5022 nan 0.1000 0.0183
20 0.2835 nan 0.1000 0.0053
40 0.1367 nan 0.1000 0.0016
60 0.0891 nan 0.1000 0.0002
80 0.0588 nan 0.1000 -0.0009
100 0.0448 nan 0.1000 -0.0006
120 0.0343 nan 0.1000 -0.0002
140 0.0282 nan 0.1000 -0.0004
160 0.0208 nan 0.1000 -0.0002
180 0.0172 nan 0.1000 -0.0002
200 0.0130 nan 0.1000 -0.0001
220 0.0101 nan 0.1000 -0.0001
240 0.0078 nan 0.1000 -0.0001
260 0.0065 nan 0.1000 -0.0001
280 0.0051 nan 0.1000 -0.0000
300 0.0042 nan 0.1000 -0.0000
320 0.0033 nan 0.1000 -0.0000
340 0.0026 nan 0.1000 0.0000
360 0.0021 nan 0.1000 -0.0000
380 0.0016 nan 0.1000 -0.0000
400 0.0014 nan 0.1000 -0.0000
420 0.0011 nan 0.1000 -0.0000
440 0.0009 nan 0.1000 -0.0000
460 0.0007 nan 0.1000 -0.0000
480 0.0006 nan 0.1000 -0.0000
500 0.0005 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1344 nan 0.1000 0.0733
2 1.0068 nan 0.1000 0.0618
3 0.9022 nan 0.1000 0.0507
4 0.8127 nan 0.1000 0.0401
5 0.7384 nan 0.1000 0.0358
6 0.6715 nan 0.1000 0.0304
7 0.6126 nan 0.1000 0.0273
8 0.5668 nan 0.1000 0.0195
9 0.5265 nan 0.1000 0.0174
10 0.4867 nan 0.1000 0.0187
20 0.2667 nan 0.1000 0.0062
40 0.1212 nan 0.1000 0.0011
60 0.0713 nan 0.1000 -0.0002
80 0.0442 nan 0.1000 0.0000
100 0.0327 nan 0.1000 -0.0005
120 0.0237 nan 0.1000 -0.0004
140 0.0175 nan 0.1000 -0.0001
160 0.0124 nan 0.1000 0.0000
180 0.0089 nan 0.1000 -0.0001
200 0.0066 nan 0.1000 -0.0001
220 0.0047 nan 0.1000 -0.0001
240 0.0037 nan 0.1000 -0.0001
260 0.0028 nan 0.1000 -0.0001
280 0.0022 nan 0.1000 -0.0000
300 0.0017 nan 0.1000 -0.0000
320 0.0013 nan 0.1000 -0.0000
340 0.0010 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1342 nan 0.1000 0.0749
2 1.0105 nan 0.1000 0.0593
3 0.9092 nan 0.1000 0.0485
4 0.8211 nan 0.1000 0.0415
5 0.7409 nan 0.1000 0.0383
6 0.6751 nan 0.1000 0.0316
7 0.6194 nan 0.1000 0.0272
8 0.5717 nan 0.1000 0.0219
9 0.5267 nan 0.1000 0.0203
10 0.4913 nan 0.1000 0.0170
20 0.2586 nan 0.1000 0.0041
40 0.1213 nan 0.1000 -0.0010
60 0.0718 nan 0.1000 -0.0004
80 0.0437 nan 0.1000 -0.0002
100 0.0291 nan 0.1000 -0.0004
120 0.0209 nan 0.1000 -0.0006
140 0.0146 nan 0.1000 -0.0002
160 0.0101 nan 0.1000 -0.0003
180 0.0074 nan 0.1000 -0.0001
200 0.0054 nan 0.1000 -0.0001
220 0.0042 nan 0.1000 -0.0001
240 0.0032 nan 0.1000 -0.0000
260 0.0025 nan 0.1000 -0.0000
280 0.0018 nan 0.1000 -0.0000
300 0.0013 nan 0.1000 -0.0000
320 0.0010 nan 0.1000 -0.0000
340 0.0009 nan 0.1000 -0.0000
360 0.0007 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1331 nan 0.1000 0.0715
2 1.0134 nan 0.1000 0.0589
3 0.9158 nan 0.1000 0.0442
4 0.8229 nan 0.1000 0.0400
5 0.7494 nan 0.1000 0.0337
6 0.6853 nan 0.1000 0.0291
7 0.6320 nan 0.1000 0.0247
8 0.5837 nan 0.1000 0.0216
9 0.5405 nan 0.1000 0.0201
10 0.4996 nan 0.1000 0.0179
20 0.2679 nan 0.1000 0.0055
40 0.1203 nan 0.1000 0.0003
60 0.0684 nan 0.1000 -0.0005
80 0.0435 nan 0.1000 -0.0002
100 0.0299 nan 0.1000 -0.0007
120 0.0185 nan 0.1000 -0.0003
140 0.0138 nan 0.1000 -0.0005
160 0.0090 nan 0.1000 -0.0001
180 0.0068 nan 0.1000 -0.0002
200 0.0048 nan 0.1000 -0.0001
220 0.0038 nan 0.1000 0.0000
240 0.0029 nan 0.1000 0.0000
260 0.0021 nan 0.1000 -0.0000
280 0.0015 nan 0.1000 -0.0000
300 0.0012 nan 0.1000 -0.0000
320 0.0011 nan 0.1000 -0.0000
340 0.0008 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 0.0000
380 0.0005 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0005 nan 0.1000 -0.0000
440 0.0004 nan 0.1000 -0.0000
460 0.0003 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1238 nan 0.1000 0.0828
2 0.9996 nan 0.1000 0.0619
3 0.8962 nan 0.1000 0.0476
4 0.8064 nan 0.1000 0.0428
5 0.7354 nan 0.1000 0.0337
6 0.6674 nan 0.1000 0.0312
7 0.6102 nan 0.1000 0.0249
8 0.5662 nan 0.1000 0.0185
9 0.5241 nan 0.1000 0.0197
10 0.4847 nan 0.1000 0.0179
20 0.2566 nan 0.1000 0.0053
40 0.0967 nan 0.1000 0.0008
60 0.0487 nan 0.1000 -0.0002
80 0.0289 nan 0.1000 -0.0000
100 0.0194 nan 0.1000 -0.0001
120 0.0137 nan 0.1000 -0.0003
140 0.0093 nan 0.1000 -0.0001
160 0.0072 nan 0.1000 -0.0002
180 0.0047 nan 0.1000 -0.0001
200 0.0033 nan 0.1000 -0.0000
220 0.0024 nan 0.1000 -0.0000
240 0.0020 nan 0.1000 -0.0001
260 0.0013 nan 0.1000 -0.0000
280 0.0009 nan 0.1000 -0.0000
300 0.0007 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1292 nan 0.1000 0.0808
2 1.0022 nan 0.1000 0.0586
3 0.9062 nan 0.1000 0.0442
4 0.8209 nan 0.1000 0.0405
5 0.7437 nan 0.1000 0.0388
6 0.6800 nan 0.1000 0.0301
7 0.6251 nan 0.1000 0.0254
8 0.5803 nan 0.1000 0.0193
9 0.5320 nan 0.1000 0.0234
10 0.4928 nan 0.1000 0.0161
20 0.2588 nan 0.1000 0.0049
40 0.0983 nan 0.1000 0.0008
60 0.0515 nan 0.1000 -0.0008
80 0.0317 nan 0.1000 -0.0004
100 0.0227 nan 0.1000 -0.0002
120 0.0144 nan 0.1000 -0.0001
140 0.0129 nan 0.1000 -0.0005
160 0.0086 nan 0.1000 0.0000
180 0.0061 nan 0.1000 0.0000
200 0.0044 nan 0.1000 -0.0001
220 0.0034 nan 0.1000 -0.0001
240 0.0024 nan 0.1000 -0.0000
260 0.0016 nan 0.1000 -0.0000
280 0.0013 nan 0.1000 -0.0001
300 0.0011 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1386 nan 0.1000 0.0720
2 1.0105 nan 0.1000 0.0640
3 0.9081 nan 0.1000 0.0487
4 0.8222 nan 0.1000 0.0398
5 0.7427 nan 0.1000 0.0354
6 0.6802 nan 0.1000 0.0302
7 0.6262 nan 0.1000 0.0249
8 0.5728 nan 0.1000 0.0250
9 0.5276 nan 0.1000 0.0188
10 0.4918 nan 0.1000 0.0158
20 0.2586 nan 0.1000 0.0053
40 0.1033 nan 0.1000 -0.0004
60 0.0529 nan 0.1000 -0.0008
80 0.0337 nan 0.1000 -0.0003
100 0.0212 nan 0.1000 -0.0004
120 0.0146 nan 0.1000 -0.0004
140 0.0097 nan 0.1000 -0.0003
160 0.0075 nan 0.1000 -0.0002
180 0.0056 nan 0.1000 -0.0000
200 0.0039 nan 0.1000 -0.0001
220 0.0026 nan 0.1000 -0.0001
240 0.0021 nan 0.1000 -0.0001
260 0.0017 nan 0.1000 -0.0001
280 0.0013 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1522 nan 0.1000 0.0678
2 1.0417 nan 0.1000 0.0503
3 0.9464 nan 0.1000 0.0455
4 0.8631 nan 0.1000 0.0418
5 0.7950 nan 0.1000 0.0337
6 0.7381 nan 0.1000 0.0270
7 0.6856 nan 0.1000 0.0255
8 0.6401 nan 0.1000 0.0194
9 0.6049 nan 0.1000 0.0181
10 0.5647 nan 0.1000 0.0180
20 0.3614 nan 0.1000 0.0060
40 0.2264 nan 0.1000 0.0012
60 0.1873 nan 0.1000 -0.0009
80 0.1633 nan 0.1000 -0.0006
100 0.1459 nan 0.1000 -0.0008
120 0.1334 nan 0.1000 0.0001
140 0.1258 nan 0.1000 -0.0010
160 0.1195 nan 0.1000 -0.0008
180 0.1112 nan 0.1000 0.0000
200 0.1075 nan 0.1000 -0.0007
220 0.1029 nan 0.1000 -0.0007
240 0.0995 nan 0.1000 -0.0005
260 0.0976 nan 0.1000 -0.0003
280 0.0956 nan 0.1000 -0.0001
300 0.0930 nan 0.1000 -0.0007
320 0.0899 nan 0.1000 -0.0009
340 0.0885 nan 0.1000 -0.0006
360 0.0869 nan 0.1000 -0.0007
380 0.0855 nan 0.1000 -0.0007
400 0.0840 nan 0.1000 -0.0007
420 0.0822 nan 0.1000 -0.0007
440 0.0819 nan 0.1000 -0.0003
460 0.0801 nan 0.1000 -0.0005
480 0.0786 nan 0.1000 -0.0004
500 0.0783 nan 0.1000 -0.0011
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1353 nan 0.1000 0.0730
2 1.0175 nan 0.1000 0.0620
3 0.9192 nan 0.1000 0.0481
4 0.8336 nan 0.1000 0.0414
5 0.7661 nan 0.1000 0.0334
6 0.7064 nan 0.1000 0.0265
7 0.6485 nan 0.1000 0.0275
8 0.5948 nan 0.1000 0.0240
9 0.5526 nan 0.1000 0.0195
10 0.5175 nan 0.1000 0.0160
20 0.2955 nan 0.1000 0.0056
40 0.1693 nan 0.1000 -0.0016
60 0.1321 nan 0.1000 -0.0006
80 0.1114 nan 0.1000 -0.0005
100 0.0958 nan 0.1000 -0.0009
120 0.0828 nan 0.1000 -0.0003
140 0.0743 nan 0.1000 -0.0006
160 0.0657 nan 0.1000 -0.0007
180 0.0613 nan 0.1000 -0.0009
200 0.0539 nan 0.1000 -0.0006
220 0.0472 nan 0.1000 -0.0001
240 0.0436 nan 0.1000 -0.0002
260 0.0402 nan 0.1000 -0.0004
280 0.0352 nan 0.1000 -0.0002
300 0.0326 nan 0.1000 -0.0002
320 0.0297 nan 0.1000 -0.0003
340 0.0272 nan 0.1000 -0.0001
360 0.0248 nan 0.1000 -0.0001
380 0.0222 nan 0.1000 -0.0002
400 0.0199 nan 0.1000 -0.0002
420 0.0183 nan 0.1000 -0.0001
440 0.0169 nan 0.1000 -0.0001
460 0.0152 nan 0.1000 -0.0001
480 0.0135 nan 0.1000 -0.0001
500 0.0122 nan 0.1000 -0.0001
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1341 nan 0.1000 0.0725
2 1.0093 nan 0.1000 0.0586
3 0.9047 nan 0.1000 0.0492
4 0.8189 nan 0.1000 0.0427
5 0.7440 nan 0.1000 0.0363
6 0.6800 nan 0.1000 0.0322
7 0.6238 nan 0.1000 0.0265
8 0.5767 nan 0.1000 0.0221
9 0.5332 nan 0.1000 0.0190
10 0.4980 nan 0.1000 0.0158
20 0.2783 nan 0.1000 0.0054
40 0.1453 nan 0.1000 -0.0000
60 0.1008 nan 0.1000 0.0000
80 0.0788 nan 0.1000 -0.0006
100 0.0632 nan 0.1000 -0.0007
120 0.0501 nan 0.1000 -0.0009
140 0.0399 nan 0.1000 -0.0003
160 0.0311 nan 0.1000 -0.0004
180 0.0270 nan 0.1000 -0.0001
200 0.0227 nan 0.1000 -0.0002
220 0.0180 nan 0.1000 -0.0002
240 0.0147 nan 0.1000 -0.0001
260 0.0127 nan 0.1000 -0.0002
280 0.0109 nan 0.1000 -0.0001
300 0.0091 nan 0.1000 -0.0001
320 0.0079 nan 0.1000 -0.0001
340 0.0069 nan 0.1000 -0.0000
360 0.0060 nan 0.1000 -0.0000
380 0.0050 nan 0.1000 -0.0001
400 0.0044 nan 0.1000 -0.0000
420 0.0037 nan 0.1000 -0.0000
440 0.0032 nan 0.1000 -0.0000
460 0.0029 nan 0.1000 -0.0000
480 0.0024 nan 0.1000 -0.0000
500 0.0021 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1324 nan 0.1000 0.0750
2 1.0074 nan 0.1000 0.0583
3 0.9005 nan 0.1000 0.0509
4 0.8152 nan 0.1000 0.0404
5 0.7381 nan 0.1000 0.0361
6 0.6761 nan 0.1000 0.0296
7 0.6224 nan 0.1000 0.0244
8 0.5741 nan 0.1000 0.0213
9 0.5342 nan 0.1000 0.0161
10 0.4949 nan 0.1000 0.0177
20 0.2735 nan 0.1000 0.0043
40 0.1340 nan 0.1000 0.0008
60 0.0867 nan 0.1000 -0.0010
80 0.0622 nan 0.1000 -0.0007
100 0.0479 nan 0.1000 -0.0005
120 0.0372 nan 0.1000 -0.0001
140 0.0283 nan 0.1000 -0.0003
160 0.0229 nan 0.1000 -0.0005
180 0.0184 nan 0.1000 -0.0003
200 0.0144 nan 0.1000 -0.0002
220 0.0113 nan 0.1000 -0.0000
240 0.0095 nan 0.1000 -0.0001
260 0.0077 nan 0.1000 -0.0001
280 0.0063 nan 0.1000 -0.0001
300 0.0052 nan 0.1000 -0.0000
320 0.0041 nan 0.1000 -0.0001
340 0.0032 nan 0.1000 -0.0001
360 0.0027 nan 0.1000 -0.0000
380 0.0022 nan 0.1000 -0.0001
400 0.0018 nan 0.1000 -0.0000
420 0.0014 nan 0.1000 -0.0000
440 0.0011 nan 0.1000 -0.0000
460 0.0009 nan 0.1000 -0.0000
480 0.0008 nan 0.1000 -0.0000
500 0.0006 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1259 nan 0.1000 0.0782
2 1.0025 nan 0.1000 0.0620
3 0.9038 nan 0.1000 0.0460
4 0.8143 nan 0.1000 0.0411
5 0.7391 nan 0.1000 0.0344
6 0.6779 nan 0.1000 0.0279
7 0.6214 nan 0.1000 0.0275
8 0.5703 nan 0.1000 0.0241
9 0.5216 nan 0.1000 0.0215
10 0.4854 nan 0.1000 0.0177
20 0.2617 nan 0.1000 0.0048
40 0.1137 nan 0.1000 -0.0002
60 0.0720 nan 0.1000 -0.0001
80 0.0473 nan 0.1000 -0.0006
100 0.0343 nan 0.1000 -0.0001
120 0.0248 nan 0.1000 0.0000
140 0.0187 nan 0.1000 -0.0004
160 0.0129 nan 0.1000 -0.0001
180 0.0104 nan 0.1000 -0.0003
200 0.0075 nan 0.1000 -0.0000
220 0.0060 nan 0.1000 -0.0001
240 0.0042 nan 0.1000 -0.0000
260 0.0031 nan 0.1000 -0.0000
280 0.0023 nan 0.1000 -0.0000
300 0.0018 nan 0.1000 -0.0000
320 0.0014 nan 0.1000 -0.0000
340 0.0011 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1276 nan 0.1000 0.0761
2 1.0000 nan 0.1000 0.0627
3 0.8967 nan 0.1000 0.0480
4 0.8121 nan 0.1000 0.0376
5 0.7377 nan 0.1000 0.0352
6 0.6741 nan 0.1000 0.0317
7 0.6170 nan 0.1000 0.0264
8 0.5700 nan 0.1000 0.0211
9 0.5272 nan 0.1000 0.0200
10 0.4892 nan 0.1000 0.0171
20 0.2586 nan 0.1000 0.0059
40 0.1141 nan 0.1000 -0.0008
60 0.0654 nan 0.1000 -0.0001
80 0.0413 nan 0.1000 -0.0005
100 0.0271 nan 0.1000 -0.0003
120 0.0205 nan 0.1000 -0.0001
140 0.0151 nan 0.1000 -0.0000
160 0.0102 nan 0.1000 -0.0001
180 0.0073 nan 0.1000 0.0000
200 0.0055 nan 0.1000 -0.0001
220 0.0041 nan 0.1000 -0.0000
240 0.0030 nan 0.1000 -0.0000
260 0.0023 nan 0.1000 -0.0000
280 0.0018 nan 0.1000 -0.0000
300 0.0015 nan 0.1000 0.0000
320 0.0013 nan 0.1000 -0.0000
340 0.0009 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1260 nan 0.1000 0.0743
2 1.0022 nan 0.1000 0.0590
3 0.9013 nan 0.1000 0.0481
4 0.8139 nan 0.1000 0.0436
5 0.7402 nan 0.1000 0.0350
6 0.6766 nan 0.1000 0.0292
7 0.6185 nan 0.1000 0.0263
8 0.5691 nan 0.1000 0.0197
9 0.5295 nan 0.1000 0.0153
10 0.4912 nan 0.1000 0.0148
20 0.2662 nan 0.1000 0.0053
40 0.1215 nan 0.1000 0.0013
60 0.0663 nan 0.1000 -0.0015
80 0.0424 nan 0.1000 -0.0001
100 0.0271 nan 0.1000 -0.0005
120 0.0185 nan 0.1000 -0.0004
140 0.0123 nan 0.1000 -0.0001
160 0.0091 nan 0.1000 -0.0001
180 0.0069 nan 0.1000 -0.0001
200 0.0053 nan 0.1000 -0.0001
220 0.0043 nan 0.1000 -0.0001
240 0.0033 nan 0.1000 -0.0000
260 0.0026 nan 0.1000 -0.0000
280 0.0023 nan 0.1000 -0.0000
300 0.0016 nan 0.1000 -0.0000
320 0.0011 nan 0.1000 -0.0000
340 0.0010 nan 0.1000 0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0005 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0005 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1304 nan 0.1000 0.0793
2 1.0040 nan 0.1000 0.0661
3 0.8991 nan 0.1000 0.0527
4 0.8117 nan 0.1000 0.0407
5 0.7399 nan 0.1000 0.0341
6 0.6753 nan 0.1000 0.0288
7 0.6158 nan 0.1000 0.0276
8 0.5681 nan 0.1000 0.0199
9 0.5283 nan 0.1000 0.0182
10 0.4889 nan 0.1000 0.0193
20 0.2578 nan 0.1000 0.0055
40 0.1026 nan 0.1000 -0.0002
60 0.0584 nan 0.1000 -0.0008
80 0.0358 nan 0.1000 -0.0006
100 0.0233 nan 0.1000 -0.0002
120 0.0155 nan 0.1000 -0.0002
140 0.0104 nan 0.1000 -0.0003
160 0.0077 nan 0.1000 -0.0002
180 0.0057 nan 0.1000 -0.0001
200 0.0043 nan 0.1000 -0.0002
220 0.0033 nan 0.1000 -0.0000
240 0.0021 nan 0.1000 0.0000
260 0.0015 nan 0.1000 -0.0000
280 0.0011 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0001 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1289 nan 0.1000 0.0775
2 1.0045 nan 0.1000 0.0552
3 0.9004 nan 0.1000 0.0528
4 0.8131 nan 0.1000 0.0418
5 0.7422 nan 0.1000 0.0344
6 0.6785 nan 0.1000 0.0289
7 0.6192 nan 0.1000 0.0294
8 0.5730 nan 0.1000 0.0206
9 0.5294 nan 0.1000 0.0202
10 0.4906 nan 0.1000 0.0161
20 0.2536 nan 0.1000 0.0053
40 0.1083 nan 0.1000 -0.0015
60 0.0616 nan 0.1000 -0.0004
80 0.0422 nan 0.1000 -0.0006
100 0.0263 nan 0.1000 0.0002
120 0.0192 nan 0.1000 0.0002
140 0.0136 nan 0.1000 -0.0002
160 0.0106 nan 0.1000 -0.0004
180 0.0076 nan 0.1000 -0.0002
200 0.0051 nan 0.1000 -0.0001
220 0.0038 nan 0.1000 0.0000
240 0.0028 nan 0.1000 -0.0001
260 0.0023 nan 0.1000 -0.0001
280 0.0015 nan 0.1000 -0.0000
300 0.0014 nan 0.1000 -0.0000
320 0.0012 nan 0.1000 -0.0000
340 0.0011 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0005 nan 0.1000 0.0000
460 0.0003 nan 0.1000 0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1356 nan 0.1000 0.0688
2 1.0110 nan 0.1000 0.0564
3 0.9145 nan 0.1000 0.0428
4 0.8241 nan 0.1000 0.0403
5 0.7496 nan 0.1000 0.0358
6 0.6851 nan 0.1000 0.0302
7 0.6268 nan 0.1000 0.0273
8 0.5772 nan 0.1000 0.0225
9 0.5356 nan 0.1000 0.0173
10 0.4986 nan 0.1000 0.0163
20 0.2640 nan 0.1000 0.0062
40 0.1149 nan 0.1000 0.0002
60 0.0587 nan 0.1000 -0.0001
80 0.0399 nan 0.1000 -0.0008
100 0.0267 nan 0.1000 -0.0005
120 0.0190 nan 0.1000 -0.0001
140 0.0137 nan 0.1000 -0.0003
160 0.0095 nan 0.1000 -0.0000
180 0.0078 nan 0.1000 0.0000
200 0.0056 nan 0.1000 -0.0001
220 0.0041 nan 0.1000 -0.0000
240 0.0032 nan 0.1000 -0.0001
260 0.0025 nan 0.1000 -0.0001
280 0.0021 nan 0.1000 -0.0001
300 0.0014 nan 0.1000 -0.0000
320 0.0009 nan 0.1000 -0.0000
340 0.0008 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1439 nan 0.1000 0.0628
2 1.0362 nan 0.1000 0.0525
3 0.9414 nan 0.1000 0.0446
4 0.8649 nan 0.1000 0.0371
5 0.8013 nan 0.1000 0.0324
6 0.7436 nan 0.1000 0.0276
7 0.6945 nan 0.1000 0.0243
8 0.6499 nan 0.1000 0.0201
9 0.6103 nan 0.1000 0.0187
10 0.5753 nan 0.1000 0.0170
20 0.3754 nan 0.1000 0.0059
40 0.2446 nan 0.1000 0.0015
60 0.2035 nan 0.1000 0.0006
80 0.1753 nan 0.1000 -0.0002
100 0.1601 nan 0.1000 -0.0008
120 0.1478 nan 0.1000 -0.0002
140 0.1379 nan 0.1000 -0.0015
160 0.1278 nan 0.1000 -0.0008
180 0.1220 nan 0.1000 -0.0006
200 0.1184 nan 0.1000 -0.0010
220 0.1142 nan 0.1000 -0.0007
240 0.1094 nan 0.1000 -0.0005
260 0.1068 nan 0.1000 -0.0003
280 0.1045 nan 0.1000 -0.0010
300 0.1012 nan 0.1000 -0.0006
320 0.1000 nan 0.1000 -0.0012
340 0.0976 nan 0.1000 -0.0003
360 0.0962 nan 0.1000 -0.0007
380 0.0936 nan 0.1000 -0.0012
400 0.0925 nan 0.1000 -0.0006
420 0.0908 nan 0.1000 -0.0009
440 0.0887 nan 0.1000 -0.0011
460 0.0868 nan 0.1000 -0.0004
480 0.0872 nan 0.1000 -0.0002
500 0.0860 nan 0.1000 -0.0008
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1368 nan 0.1000 0.0781
2 1.0196 nan 0.1000 0.0561
3 0.9214 nan 0.1000 0.0442
4 0.8391 nan 0.1000 0.0420
5 0.7680 nan 0.1000 0.0325
6 0.7099 nan 0.1000 0.0291
7 0.6545 nan 0.1000 0.0233
8 0.6047 nan 0.1000 0.0238
9 0.5672 nan 0.1000 0.0174
10 0.5289 nan 0.1000 0.0168
20 0.3168 nan 0.1000 0.0059
40 0.1817 nan 0.1000 0.0001
60 0.1395 nan 0.1000 -0.0013
80 0.1077 nan 0.1000 0.0003
100 0.0910 nan 0.1000 -0.0006
120 0.0808 nan 0.1000 -0.0002
140 0.0728 nan 0.1000 -0.0004
160 0.0634 nan 0.1000 -0.0009
180 0.0563 nan 0.1000 -0.0006
200 0.0489 nan 0.1000 -0.0004
220 0.0447 nan 0.1000 -0.0004
240 0.0407 nan 0.1000 -0.0003
260 0.0367 nan 0.1000 -0.0003
280 0.0339 nan 0.1000 -0.0007
300 0.0312 nan 0.1000 -0.0000
320 0.0281 nan 0.1000 -0.0001
340 0.0261 nan 0.1000 -0.0002
360 0.0236 nan 0.1000 -0.0002
380 0.0215 nan 0.1000 -0.0003
400 0.0201 nan 0.1000 -0.0002
420 0.0184 nan 0.1000 -0.0004
440 0.0170 nan 0.1000 -0.0001
460 0.0161 nan 0.1000 -0.0001
480 0.0148 nan 0.1000 -0.0002
500 0.0136 nan 0.1000 -0.0002
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1336 nan 0.1000 0.0742
2 1.0140 nan 0.1000 0.0601
3 0.9102 nan 0.1000 0.0511
4 0.8274 nan 0.1000 0.0371
5 0.7544 nan 0.1000 0.0369
6 0.6897 nan 0.1000 0.0314
7 0.6370 nan 0.1000 0.0251
8 0.5884 nan 0.1000 0.0234
9 0.5453 nan 0.1000 0.0215
10 0.5089 nan 0.1000 0.0153
20 0.2903 nan 0.1000 0.0059
40 0.1557 nan 0.1000 0.0001
60 0.1081 nan 0.1000 -0.0001
80 0.0854 nan 0.1000 -0.0004
100 0.0676 nan 0.1000 -0.0008
120 0.0549 nan 0.1000 -0.0004
140 0.0464 nan 0.1000 -0.0003
160 0.0369 nan 0.1000 -0.0003
180 0.0317 nan 0.1000 -0.0003
200 0.0268 nan 0.1000 -0.0004
220 0.0226 nan 0.1000 -0.0005
240 0.0187 nan 0.1000 -0.0001
260 0.0158 nan 0.1000 -0.0000
280 0.0141 nan 0.1000 -0.0002
300 0.0122 nan 0.1000 -0.0002
320 0.0104 nan 0.1000 -0.0003
340 0.0089 nan 0.1000 -0.0001
360 0.0077 nan 0.1000 -0.0000
380 0.0068 nan 0.1000 -0.0000
400 0.0060 nan 0.1000 -0.0001
420 0.0051 nan 0.1000 -0.0001
440 0.0042 nan 0.1000 -0.0000
460 0.0037 nan 0.1000 -0.0000
480 0.0031 nan 0.1000 -0.0000
500 0.0026 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1381 nan 0.1000 0.0713
2 1.0158 nan 0.1000 0.0565
3 0.9123 nan 0.1000 0.0473
4 0.8283 nan 0.1000 0.0401
5 0.7559 nan 0.1000 0.0323
6 0.6946 nan 0.1000 0.0296
7 0.6408 nan 0.1000 0.0248
8 0.5922 nan 0.1000 0.0211
9 0.5514 nan 0.1000 0.0165
10 0.5135 nan 0.1000 0.0166
20 0.2820 nan 0.1000 0.0049
40 0.1424 nan 0.1000 0.0009
60 0.0906 nan 0.1000 0.0001
80 0.0651 nan 0.1000 -0.0002
100 0.0470 nan 0.1000 -0.0007
120 0.0371 nan 0.1000 -0.0005
140 0.0285 nan 0.1000 -0.0002
160 0.0226 nan 0.1000 -0.0002
180 0.0183 nan 0.1000 -0.0001
200 0.0147 nan 0.1000 -0.0001
220 0.0125 nan 0.1000 -0.0002
240 0.0100 nan 0.1000 0.0000
260 0.0081 nan 0.1000 -0.0001
280 0.0067 nan 0.1000 -0.0000
300 0.0055 nan 0.1000 -0.0001
320 0.0045 nan 0.1000 -0.0001
340 0.0036 nan 0.1000 -0.0000
360 0.0029 nan 0.1000 -0.0000
380 0.0023 nan 0.1000 -0.0000
400 0.0019 nan 0.1000 -0.0000
420 0.0014 nan 0.1000 -0.0000
440 0.0012 nan 0.1000 -0.0000
460 0.0009 nan 0.1000 -0.0000
480 0.0008 nan 0.1000 -0.0000
500 0.0006 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1299 nan 0.1000 0.0747
2 1.0100 nan 0.1000 0.0587
3 0.9072 nan 0.1000 0.0512
4 0.8194 nan 0.1000 0.0396
5 0.7458 nan 0.1000 0.0340
6 0.6783 nan 0.1000 0.0320
7 0.6198 nan 0.1000 0.0267
8 0.5760 nan 0.1000 0.0177
9 0.5306 nan 0.1000 0.0228
10 0.4917 nan 0.1000 0.0173
20 0.2727 nan 0.1000 0.0040
40 0.1270 nan 0.1000 -0.0003
60 0.0778 nan 0.1000 -0.0005
80 0.0526 nan 0.1000 -0.0016
100 0.0381 nan 0.1000 -0.0005
120 0.0295 nan 0.1000 -0.0004
140 0.0222 nan 0.1000 -0.0004
160 0.0172 nan 0.1000 -0.0003
180 0.0136 nan 0.1000 -0.0002
200 0.0105 nan 0.1000 -0.0000
220 0.0080 nan 0.1000 -0.0001
240 0.0066 nan 0.1000 -0.0000
260 0.0050 nan 0.1000 -0.0001
280 0.0041 nan 0.1000 -0.0001
300 0.0030 nan 0.1000 -0.0000
320 0.0023 nan 0.1000 -0.0000
340 0.0018 nan 0.1000 -0.0000
360 0.0014 nan 0.1000 -0.0000
380 0.0011 nan 0.1000 -0.0000
400 0.0008 nan 0.1000 -0.0000
420 0.0007 nan 0.1000 -0.0000
440 0.0005 nan 0.1000 -0.0000
460 0.0004 nan 0.1000 -0.0000
480 0.0003 nan 0.1000 -0.0000
500 0.0002 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1304 nan 0.1000 0.0740
2 1.0050 nan 0.1000 0.0534
3 0.9058 nan 0.1000 0.0466
4 0.8176 nan 0.1000 0.0410
5 0.7434 nan 0.1000 0.0367
6 0.6801 nan 0.1000 0.0268
7 0.6200 nan 0.1000 0.0255
8 0.5742 nan 0.1000 0.0209
9 0.5338 nan 0.1000 0.0168
10 0.4955 nan 0.1000 0.0161
20 0.2681 nan 0.1000 0.0050
40 0.1229 nan 0.1000 0.0005
60 0.0730 nan 0.1000 0.0002
80 0.0468 nan 0.1000 -0.0003
100 0.0317 nan 0.1000 -0.0004
120 0.0224 nan 0.1000 -0.0005
140 0.0163 nan 0.1000 -0.0001
160 0.0116 nan 0.1000 -0.0001
180 0.0091 nan 0.1000 -0.0002
200 0.0070 nan 0.1000 -0.0002
220 0.0059 nan 0.1000 -0.0000
240 0.0044 nan 0.1000 -0.0001
260 0.0030 nan 0.1000 0.0000
280 0.0022 nan 0.1000 -0.0000
300 0.0017 nan 0.1000 -0.0000
320 0.0012 nan 0.1000 -0.0000
340 0.0010 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1298 nan 0.1000 0.0723
2 1.0059 nan 0.1000 0.0560
3 0.9021 nan 0.1000 0.0467
4 0.8181 nan 0.1000 0.0404
5 0.7409 nan 0.1000 0.0361
6 0.6805 nan 0.1000 0.0302
7 0.6287 nan 0.1000 0.0248
8 0.5798 nan 0.1000 0.0220
9 0.5379 nan 0.1000 0.0196
10 0.5015 nan 0.1000 0.0146
20 0.2709 nan 0.1000 0.0050
40 0.1164 nan 0.1000 -0.0013
60 0.0680 nan 0.1000 -0.0005
80 0.0423 nan 0.1000 -0.0007
100 0.0308 nan 0.1000 -0.0007
120 0.0203 nan 0.1000 -0.0002
140 0.0148 nan 0.1000 -0.0001
160 0.0116 nan 0.1000 -0.0003
180 0.0075 nan 0.1000 -0.0000
200 0.0062 nan 0.1000 -0.0002
220 0.0052 nan 0.1000 -0.0000
240 0.0037 nan 0.1000 -0.0000
260 0.0026 nan 0.1000 -0.0001
280 0.0021 nan 0.1000 -0.0000
300 0.0015 nan 0.1000 -0.0000
320 0.0011 nan 0.1000 -0.0000
340 0.0008 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1336 nan 0.1000 0.0754
2 1.0054 nan 0.1000 0.0583
3 0.9056 nan 0.1000 0.0482
4 0.8212 nan 0.1000 0.0364
5 0.7457 nan 0.1000 0.0358
6 0.6833 nan 0.1000 0.0270
7 0.6284 nan 0.1000 0.0252
8 0.5812 nan 0.1000 0.0228
9 0.5378 nan 0.1000 0.0191
10 0.4989 nan 0.1000 0.0163
20 0.2683 nan 0.1000 0.0066
40 0.1162 nan 0.1000 -0.0001
60 0.0628 nan 0.1000 -0.0009
80 0.0370 nan 0.1000 -0.0005
100 0.0257 nan 0.1000 -0.0005
120 0.0179 nan 0.1000 -0.0004
140 0.0138 nan 0.1000 -0.0004
160 0.0095 nan 0.1000 -0.0002
180 0.0077 nan 0.1000 -0.0002
200 0.0062 nan 0.1000 0.0001
220 0.0047 nan 0.1000 -0.0001
240 0.0042 nan 0.1000 -0.0002
260 0.0030 nan 0.1000 -0.0000
280 0.0028 nan 0.1000 -0.0001
300 0.0026 nan 0.1000 -0.0002
320 0.0017 nan 0.1000 -0.0000
340 0.0014 nan 0.1000 -0.0001
360 0.0010 nan 0.1000 -0.0000
380 0.0007 nan 0.1000 -0.0000
400 0.0008 nan 0.1000 -0.0000
420 0.0008 nan 0.1000 -0.0000
440 0.0005 nan 0.1000 -0.0000
460 0.0004 nan 0.1000 -0.0000
480 0.0005 nan 0.1000 -0.0000
500 0.0004 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1365 nan 0.1000 0.0680
2 1.0047 nan 0.1000 0.0644
3 0.9008 nan 0.1000 0.0501
4 0.8158 nan 0.1000 0.0429
5 0.7441 nan 0.1000 0.0355
6 0.6784 nan 0.1000 0.0295
7 0.6200 nan 0.1000 0.0282
8 0.5706 nan 0.1000 0.0216
9 0.5297 nan 0.1000 0.0183
10 0.4921 nan 0.1000 0.0168
20 0.2710 nan 0.1000 0.0040
40 0.1081 nan 0.1000 -0.0007
60 0.0632 nan 0.1000 -0.0005
80 0.0391 nan 0.1000 -0.0007
100 0.0273 nan 0.1000 -0.0002
120 0.0191 nan 0.1000 -0.0002
140 0.0123 nan 0.1000 -0.0002
160 0.0090 nan 0.1000 -0.0001
180 0.0072 nan 0.1000 0.0000
200 0.0049 nan 0.1000 -0.0001
220 0.0033 nan 0.1000 -0.0000
240 0.0024 nan 0.1000 -0.0001
260 0.0019 nan 0.1000 -0.0001
280 0.0014 nan 0.1000 -0.0000
300 0.0011 nan 0.1000 -0.0000
320 0.0008 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1341 nan 0.1000 0.0790
2 1.0139 nan 0.1000 0.0541
3 0.9144 nan 0.1000 0.0455
4 0.8284 nan 0.1000 0.0395
5 0.7531 nan 0.1000 0.0327
6 0.6886 nan 0.1000 0.0293
7 0.6337 nan 0.1000 0.0246
8 0.5828 nan 0.1000 0.0231
9 0.5426 nan 0.1000 0.0178
10 0.5051 nan 0.1000 0.0153
20 0.2743 nan 0.1000 0.0057
40 0.1147 nan 0.1000 -0.0004
60 0.0622 nan 0.1000 -0.0008
80 0.0377 nan 0.1000 -0.0005
100 0.0271 nan 0.1000 -0.0005
120 0.0172 nan 0.1000 -0.0003
140 0.0106 nan 0.1000 0.0000
160 0.0072 nan 0.1000 -0.0000
180 0.0053 nan 0.1000 -0.0001
200 0.0038 nan 0.1000 -0.0000
220 0.0026 nan 0.1000 -0.0000
240 0.0017 nan 0.1000 -0.0000
260 0.0015 nan 0.1000 -0.0000
280 0.0011 nan 0.1000 -0.0000
300 0.0009 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1541 nan 0.1000 0.0701
2 1.0366 nan 0.1000 0.0537
3 0.9429 nan 0.1000 0.0447
4 0.8686 nan 0.1000 0.0360
5 0.8031 nan 0.1000 0.0308
6 0.7428 nan 0.1000 0.0276
7 0.6927 nan 0.1000 0.0250
8 0.6555 nan 0.1000 0.0159
9 0.6149 nan 0.1000 0.0175
10 0.5782 nan 0.1000 0.0173
20 0.3729 nan 0.1000 0.0042
40 0.2388 nan 0.1000 0.0012
60 0.1871 nan 0.1000 0.0007
80 0.1567 nan 0.1000 -0.0003
100 0.1379 nan 0.1000 0.0000
120 0.1253 nan 0.1000 -0.0007
140 0.1141 nan 0.1000 -0.0005
160 0.1083 nan 0.1000 -0.0003
180 0.1024 nan 0.1000 -0.0009
200 0.0952 nan 0.1000 -0.0005
220 0.0904 nan 0.1000 -0.0002
240 0.0883 nan 0.1000 -0.0004
260 0.0875 nan 0.1000 -0.0006
280 0.0851 nan 0.1000 -0.0008
300 0.0802 nan 0.1000 -0.0012
320 0.0769 nan 0.1000 -0.0008
340 0.0754 nan 0.1000 -0.0015
360 0.0724 nan 0.1000 -0.0000
380 0.0701 nan 0.1000 -0.0004
400 0.0684 nan 0.1000 -0.0004
420 0.0665 nan 0.1000 -0.0004
440 0.0658 nan 0.1000 -0.0002
460 0.0639 nan 0.1000 -0.0005
480 0.0620 nan 0.1000 -0.0001
500 0.0606 nan 0.1000 -0.0007
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1377 nan 0.1000 0.0701
2 1.0237 nan 0.1000 0.0559
3 0.9250 nan 0.1000 0.0485
4 0.8476 nan 0.1000 0.0372
5 0.7783 nan 0.1000 0.0346
6 0.7205 nan 0.1000 0.0265
7 0.6672 nan 0.1000 0.0261
8 0.6152 nan 0.1000 0.0238
9 0.5692 nan 0.1000 0.0209
10 0.5321 nan 0.1000 0.0170
20 0.3164 nan 0.1000 0.0027
40 0.1731 nan 0.1000 0.0003
60 0.1319 nan 0.1000 -0.0006
80 0.1063 nan 0.1000 -0.0003
100 0.0872 nan 0.1000 0.0001
120 0.0742 nan 0.1000 -0.0004
140 0.0671 nan 0.1000 -0.0001
160 0.0592 nan 0.1000 -0.0004
180 0.0519 nan 0.1000 -0.0004
200 0.0450 nan 0.1000 -0.0002
220 0.0391 nan 0.1000 -0.0008
240 0.0347 nan 0.1000 -0.0002
260 0.0306 nan 0.1000 -0.0003
280 0.0268 nan 0.1000 -0.0002
300 0.0243 nan 0.1000 -0.0003
320 0.0218 nan 0.1000 -0.0002
340 0.0200 nan 0.1000 -0.0003
360 0.0184 nan 0.1000 -0.0001
380 0.0161 nan 0.1000 -0.0003
400 0.0148 nan 0.1000 -0.0002
420 0.0141 nan 0.1000 -0.0003
440 0.0129 nan 0.1000 -0.0001
460 0.0116 nan 0.1000 -0.0001
480 0.0104 nan 0.1000 -0.0001
500 0.0095 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1367 nan 0.1000 0.0773
2 1.0090 nan 0.1000 0.0615
3 0.9080 nan 0.1000 0.0490
4 0.8262 nan 0.1000 0.0391
5 0.7475 nan 0.1000 0.0355
6 0.6878 nan 0.1000 0.0277
7 0.6326 nan 0.1000 0.0255
8 0.5823 nan 0.1000 0.0229
9 0.5410 nan 0.1000 0.0190
10 0.5049 nan 0.1000 0.0148
20 0.2832 nan 0.1000 0.0076
40 0.1466 nan 0.1000 0.0001
60 0.0996 nan 0.1000 -0.0008
80 0.0749 nan 0.1000 -0.0002
100 0.0580 nan 0.1000 -0.0001
120 0.0473 nan 0.1000 -0.0006
140 0.0392 nan 0.1000 -0.0002
160 0.0323 nan 0.1000 -0.0001
180 0.0264 nan 0.1000 -0.0003
200 0.0217 nan 0.1000 -0.0002
220 0.0177 nan 0.1000 -0.0001
240 0.0147 nan 0.1000 -0.0002
260 0.0121 nan 0.1000 -0.0001
280 0.0102 nan 0.1000 -0.0001
300 0.0088 nan 0.1000 -0.0001
320 0.0079 nan 0.1000 -0.0001
340 0.0068 nan 0.1000 -0.0001
360 0.0059 nan 0.1000 -0.0001
380 0.0052 nan 0.1000 -0.0001
400 0.0044 nan 0.1000 -0.0000
420 0.0036 nan 0.1000 -0.0000
440 0.0029 nan 0.1000 -0.0000
460 0.0025 nan 0.1000 -0.0000
480 0.0021 nan 0.1000 -0.0000
500 0.0017 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1415 nan 0.1000 0.0712
2 1.0170 nan 0.1000 0.0584
3 0.9133 nan 0.1000 0.0477
4 0.8278 nan 0.1000 0.0435
5 0.7548 nan 0.1000 0.0353
6 0.6904 nan 0.1000 0.0281
7 0.6323 nan 0.1000 0.0262
8 0.5851 nan 0.1000 0.0234
9 0.5439 nan 0.1000 0.0186
10 0.5059 nan 0.1000 0.0164
20 0.2762 nan 0.1000 0.0045
40 0.1312 nan 0.1000 0.0003
60 0.0857 nan 0.1000 -0.0008
80 0.0619 nan 0.1000 -0.0006
100 0.0445 nan 0.1000 -0.0004
120 0.0321 nan 0.1000 -0.0000
140 0.0241 nan 0.1000 -0.0002
160 0.0191 nan 0.1000 -0.0002
180 0.0154 nan 0.1000 -0.0002
200 0.0126 nan 0.1000 -0.0002
220 0.0099 nan 0.1000 -0.0002
240 0.0078 nan 0.1000 -0.0001
260 0.0063 nan 0.1000 -0.0001
280 0.0052 nan 0.1000 -0.0001
300 0.0044 nan 0.1000 -0.0001
320 0.0035 nan 0.1000 -0.0000
340 0.0029 nan 0.1000 -0.0000
360 0.0023 nan 0.1000 -0.0000
380 0.0018 nan 0.1000 -0.0000
400 0.0015 nan 0.1000 -0.0000
420 0.0012 nan 0.1000 -0.0000
440 0.0009 nan 0.1000 -0.0000
460 0.0008 nan 0.1000 -0.0000
480 0.0006 nan 0.1000 -0.0000
500 0.0005 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1311 nan 0.1000 0.0731
2 1.0055 nan 0.1000 0.0606
3 0.8984 nan 0.1000 0.0490
4 0.8133 nan 0.1000 0.0394
5 0.7441 nan 0.1000 0.0330
6 0.6847 nan 0.1000 0.0275
7 0.6309 nan 0.1000 0.0259
8 0.5832 nan 0.1000 0.0223
9 0.5365 nan 0.1000 0.0211
10 0.4969 nan 0.1000 0.0171
20 0.2702 nan 0.1000 0.0053
40 0.1174 nan 0.1000 -0.0006
60 0.0693 nan 0.1000 -0.0001
80 0.0443 nan 0.1000 0.0001
100 0.0318 nan 0.1000 -0.0003
120 0.0230 nan 0.1000 -0.0002
140 0.0172 nan 0.1000 -0.0001
160 0.0123 nan 0.1000 -0.0001
180 0.0099 nan 0.1000 -0.0003
200 0.0073 nan 0.1000 -0.0001
220 0.0053 nan 0.1000 -0.0001
240 0.0040 nan 0.1000 -0.0000
260 0.0032 nan 0.1000 -0.0000
280 0.0025 nan 0.1000 -0.0000
300 0.0020 nan 0.1000 -0.0000
320 0.0014 nan 0.1000 -0.0000
340 0.0010 nan 0.1000 -0.0000
360 0.0008 nan 0.1000 -0.0000
380 0.0006 nan 0.1000 -0.0000
400 0.0005 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1415 nan 0.1000 0.0616
2 1.0165 nan 0.1000 0.0583
3 0.9169 nan 0.1000 0.0478
4 0.8295 nan 0.1000 0.0430
5 0.7522 nan 0.1000 0.0355
6 0.6870 nan 0.1000 0.0312
7 0.6320 nan 0.1000 0.0238
8 0.5833 nan 0.1000 0.0227
9 0.5402 nan 0.1000 0.0203
10 0.4976 nan 0.1000 0.0181
20 0.2580 nan 0.1000 0.0054
40 0.1060 nan 0.1000 0.0005
60 0.0593 nan 0.1000 -0.0008
80 0.0379 nan 0.1000 -0.0002
100 0.0258 nan 0.1000 -0.0003
120 0.0179 nan 0.1000 -0.0001
140 0.0119 nan 0.1000 -0.0001
160 0.0083 nan 0.1000 -0.0001
180 0.0056 nan 0.1000 -0.0000
200 0.0044 nan 0.1000 -0.0001
220 0.0032 nan 0.1000 -0.0001
240 0.0024 nan 0.1000 -0.0000
260 0.0017 nan 0.1000 0.0000
280 0.0013 nan 0.1000 -0.0000
300 0.0010 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0006 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1389 nan 0.1000 0.0709
2 1.0121 nan 0.1000 0.0599
3 0.9074 nan 0.1000 0.0502
4 0.8216 nan 0.1000 0.0402
5 0.7502 nan 0.1000 0.0318
6 0.6878 nan 0.1000 0.0294
7 0.6314 nan 0.1000 0.0247
8 0.5847 nan 0.1000 0.0201
9 0.5357 nan 0.1000 0.0213
10 0.4972 nan 0.1000 0.0175
20 0.2655 nan 0.1000 0.0057
40 0.1135 nan 0.1000 0.0002
60 0.0609 nan 0.1000 -0.0003
80 0.0368 nan 0.1000 0.0000
100 0.0250 nan 0.1000 0.0000
120 0.0165 nan 0.1000 -0.0002
140 0.0108 nan 0.1000 -0.0002
160 0.0083 nan 0.1000 -0.0001
180 0.0059 nan 0.1000 -0.0001
200 0.0042 nan 0.1000 0.0000
220 0.0029 nan 0.1000 -0.0001
240 0.0021 nan 0.1000 -0.0000
260 0.0017 nan 0.1000 -0.0000
280 0.0014 nan 0.1000 -0.0000
300 0.0010 nan 0.1000 0.0000
320 0.0008 nan 0.1000 -0.0000
340 0.0006 nan 0.1000 0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 0.0000
480 0.0001 nan 0.1000 0.0000
500 0.0001 nan 0.1000 0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1272 nan 0.1000 0.0776
2 1.0056 nan 0.1000 0.0574
3 0.9048 nan 0.1000 0.0446
4 0.8199 nan 0.1000 0.0422
5 0.7485 nan 0.1000 0.0347
6 0.6868 nan 0.1000 0.0265
7 0.6313 nan 0.1000 0.0260
8 0.5848 nan 0.1000 0.0201
9 0.5417 nan 0.1000 0.0193
10 0.5048 nan 0.1000 0.0146
20 0.2634 nan 0.1000 0.0058
40 0.1083 nan 0.1000 -0.0002
60 0.0532 nan 0.1000 -0.0001
80 0.0332 nan 0.1000 -0.0006
100 0.0237 nan 0.1000 -0.0005
120 0.0165 nan 0.1000 -0.0000
140 0.0121 nan 0.1000 -0.0001
160 0.0085 nan 0.1000 -0.0003
180 0.0061 nan 0.1000 -0.0001
200 0.0042 nan 0.1000 -0.0001
220 0.0031 nan 0.1000 -0.0000
240 0.0019 nan 0.1000 -0.0000
260 0.0014 nan 0.1000 -0.0000
280 0.0009 nan 0.1000 -0.0000
300 0.0007 nan 0.1000 -0.0000
320 0.0005 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0001 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0000 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1307 nan 0.1000 0.0759
2 1.0111 nan 0.1000 0.0623
3 0.9089 nan 0.1000 0.0512
4 0.8201 nan 0.1000 0.0438
5 0.7432 nan 0.1000 0.0354
6 0.6809 nan 0.1000 0.0281
7 0.6275 nan 0.1000 0.0218
8 0.5808 nan 0.1000 0.0210
9 0.5356 nan 0.1000 0.0211
10 0.4953 nan 0.1000 0.0188
20 0.2613 nan 0.1000 0.0032
40 0.1067 nan 0.1000 0.0003
60 0.0561 nan 0.1000 0.0008
80 0.0322 nan 0.1000 0.0003
100 0.0226 nan 0.1000 -0.0004
120 0.0150 nan 0.1000 -0.0003
140 0.0096 nan 0.1000 -0.0002
160 0.0068 nan 0.1000 -0.0000
180 0.0046 nan 0.1000 -0.0001
200 0.0032 nan 0.1000 -0.0001
220 0.0024 nan 0.1000 -0.0001
240 0.0019 nan 0.1000 -0.0001
260 0.0013 nan 0.1000 -0.0000
280 0.0011 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1387 nan 0.1000 0.0768
2 1.0158 nan 0.1000 0.0572
3 0.9154 nan 0.1000 0.0463
4 0.8269 nan 0.1000 0.0429
5 0.7474 nan 0.1000 0.0373
6 0.6857 nan 0.1000 0.0266
7 0.6315 nan 0.1000 0.0233
8 0.5814 nan 0.1000 0.0214
9 0.5406 nan 0.1000 0.0184
10 0.5029 nan 0.1000 0.0158
20 0.2681 nan 0.1000 0.0035
40 0.1060 nan 0.1000 -0.0004
60 0.0518 nan 0.1000 -0.0003
80 0.0290 nan 0.1000 -0.0004
100 0.0200 nan 0.1000 -0.0003
120 0.0136 nan 0.1000 -0.0003
140 0.0106 nan 0.1000 -0.0002
160 0.0067 nan 0.1000 -0.0000
180 0.0043 nan 0.1000 -0.0001
200 0.0032 nan 0.1000 -0.0001
220 0.0021 nan 0.1000 -0.0000
240 0.0015 nan 0.1000 -0.0000
260 0.0011 nan 0.1000 -0.0000
280 0.0008 nan 0.1000 -0.0000
300 0.0006 nan 0.1000 -0.0000
320 0.0004 nan 0.1000 -0.0000
340 0.0003 nan 0.1000 0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1465 nan 0.1000 0.0655
2 1.0386 nan 0.1000 0.0534
3 0.9470 nan 0.1000 0.0428
4 0.8707 nan 0.1000 0.0344
5 0.8022 nan 0.1000 0.0328
6 0.7459 nan 0.1000 0.0293
7 0.6991 nan 0.1000 0.0204
8 0.6544 nan 0.1000 0.0202
9 0.6085 nan 0.1000 0.0193
10 0.5731 nan 0.1000 0.0168
20 0.3679 nan 0.1000 0.0059
40 0.2498 nan 0.1000 -0.0004
60 0.1936 nan 0.1000 -0.0014
80 0.1705 nan 0.1000 0.0002
100 0.1564 nan 0.1000 -0.0016
120 0.1442 nan 0.1000 -0.0016
140 0.1354 nan 0.1000 -0.0010
160 0.1302 nan 0.1000 -0.0017
180 0.1284 nan 0.1000 -0.0009
200 0.1220 nan 0.1000 -0.0009
220 0.1185 nan 0.1000 -0.0017
240 0.1156 nan 0.1000 -0.0012
260 0.1135 nan 0.1000 -0.0009
280 0.1122 nan 0.1000 -0.0007
300 0.1071 nan 0.1000 -0.0003
320 0.1058 nan 0.1000 -0.0011
340 0.1047 nan 0.1000 -0.0004
360 0.1024 nan 0.1000 -0.0005
380 0.1001 nan 0.1000 -0.0007
400 0.0997 nan 0.1000 -0.0008
420 0.0969 nan 0.1000 -0.0006
440 0.0958 nan 0.1000 -0.0003
460 0.0955 nan 0.1000 -0.0008
480 0.0953 nan 0.1000 -0.0005
500 0.0941 nan 0.1000 -0.0012
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1383 nan 0.1000 0.0719
2 1.0233 nan 0.1000 0.0560
3 0.9258 nan 0.1000 0.0472
4 0.8407 nan 0.1000 0.0403
5 0.7715 nan 0.1000 0.0339
6 0.7066 nan 0.1000 0.0301
7 0.6541 nan 0.1000 0.0259
8 0.6047 nan 0.1000 0.0231
9 0.5600 nan 0.1000 0.0209
10 0.5232 nan 0.1000 0.0175
20 0.3084 nan 0.1000 0.0056
40 0.1879 nan 0.1000 0.0002
60 0.1427 nan 0.1000 0.0001
80 0.1177 nan 0.1000 -0.0005
100 0.1042 nan 0.1000 -0.0013
120 0.0935 nan 0.1000 -0.0012
140 0.0813 nan 0.1000 -0.0013
160 0.0736 nan 0.1000 0.0002
180 0.0633 nan 0.1000 -0.0002
200 0.0549 nan 0.1000 -0.0007
220 0.0498 nan 0.1000 -0.0006
240 0.0460 nan 0.1000 -0.0004
260 0.0425 nan 0.1000 -0.0005
280 0.0385 nan 0.1000 -0.0002
300 0.0346 nan 0.1000 -0.0006
320 0.0304 nan 0.1000 -0.0002
340 0.0280 nan 0.1000 -0.0002
360 0.0258 nan 0.1000 -0.0002
380 0.0230 nan 0.1000 -0.0001
400 0.0221 nan 0.1000 -0.0001
420 0.0203 nan 0.1000 -0.0002
440 0.0188 nan 0.1000 -0.0001
460 0.0176 nan 0.1000 -0.0001
480 0.0166 nan 0.1000 -0.0002
500 0.0154 nan 0.1000 -0.0003
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1330 nan 0.1000 0.0730
2 1.0129 nan 0.1000 0.0563
3 0.9141 nan 0.1000 0.0455
4 0.8308 nan 0.1000 0.0389
5 0.7536 nan 0.1000 0.0381
6 0.6910 nan 0.1000 0.0285
7 0.6324 nan 0.1000 0.0279
8 0.5817 nan 0.1000 0.0247
9 0.5357 nan 0.1000 0.0205
10 0.5025 nan 0.1000 0.0148
20 0.2862 nan 0.1000 0.0049
40 0.1491 nan 0.1000 -0.0001
60 0.1035 nan 0.1000 -0.0004
80 0.0809 nan 0.1000 -0.0007
100 0.0679 nan 0.1000 -0.0009
120 0.0513 nan 0.1000 -0.0004
140 0.0426 nan 0.1000 -0.0007
160 0.0352 nan 0.1000 -0.0003
180 0.0286 nan 0.1000 -0.0003
200 0.0237 nan 0.1000 -0.0002
220 0.0203 nan 0.1000 -0.0002
240 0.0173 nan 0.1000 -0.0000
260 0.0159 nan 0.1000 0.0000
280 0.0136 nan 0.1000 -0.0001
300 0.0114 nan 0.1000 -0.0001
320 0.0100 nan 0.1000 -0.0001
340 0.0086 nan 0.1000 -0.0001
360 0.0074 nan 0.1000 -0.0001
380 0.0063 nan 0.1000 -0.0001
400 0.0056 nan 0.1000 -0.0001
420 0.0048 nan 0.1000 -0.0001
440 0.0041 nan 0.1000 -0.0000
460 0.0036 nan 0.1000 -0.0000
480 0.0031 nan 0.1000 -0.0000
500 0.0028 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1427 nan 0.1000 0.0749
2 1.0182 nan 0.1000 0.0586
3 0.9185 nan 0.1000 0.0493
4 0.8359 nan 0.1000 0.0375
5 0.7577 nan 0.1000 0.0371
6 0.6942 nan 0.1000 0.0309
7 0.6411 nan 0.1000 0.0227
8 0.5923 nan 0.1000 0.0220
9 0.5520 nan 0.1000 0.0169
10 0.5097 nan 0.1000 0.0177
20 0.2837 nan 0.1000 0.0051
40 0.1424 nan 0.1000 0.0004
60 0.0915 nan 0.1000 -0.0005
80 0.0665 nan 0.1000 -0.0008
100 0.0495 nan 0.1000 -0.0004
120 0.0384 nan 0.1000 -0.0004
140 0.0281 nan 0.1000 -0.0001
160 0.0218 nan 0.1000 -0.0002
180 0.0170 nan 0.1000 -0.0001
200 0.0134 nan 0.1000 -0.0002
220 0.0113 nan 0.1000 -0.0002
240 0.0088 nan 0.1000 -0.0002
260 0.0071 nan 0.1000 -0.0001
280 0.0057 nan 0.1000 -0.0001
300 0.0045 nan 0.1000 -0.0000
320 0.0037 nan 0.1000 -0.0000
340 0.0029 nan 0.1000 -0.0000
360 0.0024 nan 0.1000 -0.0000
380 0.0019 nan 0.1000 -0.0000
400 0.0017 nan 0.1000 -0.0000
420 0.0013 nan 0.1000 -0.0000
440 0.0011 nan 0.1000 -0.0000
460 0.0008 nan 0.1000 -0.0000
480 0.0007 nan 0.1000 -0.0000
500 0.0006 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1334 nan 0.1000 0.0794
2 1.0111 nan 0.1000 0.0582
3 0.9049 nan 0.1000 0.0492
4 0.8172 nan 0.1000 0.0394
5 0.7420 nan 0.1000 0.0365
6 0.6830 nan 0.1000 0.0260
7 0.6211 nan 0.1000 0.0280
8 0.5729 nan 0.1000 0.0223
9 0.5292 nan 0.1000 0.0191
10 0.4908 nan 0.1000 0.0164
20 0.2612 nan 0.1000 0.0064
40 0.1273 nan 0.1000 -0.0008
60 0.0740 nan 0.1000 0.0002
80 0.0498 nan 0.1000 -0.0008
100 0.0339 nan 0.1000 -0.0002
120 0.0225 nan 0.1000 0.0000
140 0.0165 nan 0.1000 -0.0001
160 0.0124 nan 0.1000 -0.0002
180 0.0091 nan 0.1000 -0.0002
200 0.0070 nan 0.1000 -0.0001
220 0.0052 nan 0.1000 -0.0001
240 0.0043 nan 0.1000 -0.0001
260 0.0033 nan 0.1000 -0.0001
280 0.0026 nan 0.1000 -0.0001
300 0.0021 nan 0.1000 -0.0000
320 0.0015 nan 0.1000 -0.0000
340 0.0011 nan 0.1000 -0.0000
360 0.0009 nan 0.1000 0.0000
380 0.0008 nan 0.1000 0.0000
400 0.0006 nan 0.1000 -0.0000
420 0.0004 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0003 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0002 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1248 nan 0.1000 0.0775
2 1.0020 nan 0.1000 0.0586
3 0.8990 nan 0.1000 0.0477
4 0.8099 nan 0.1000 0.0435
5 0.7371 nan 0.1000 0.0343
6 0.6758 nan 0.1000 0.0305
7 0.6179 nan 0.1000 0.0268
8 0.5670 nan 0.1000 0.0235
9 0.5227 nan 0.1000 0.0184
10 0.4856 nan 0.1000 0.0161
20 0.2601 nan 0.1000 0.0042
40 0.1128 nan 0.1000 -0.0004
60 0.0642 nan 0.1000 -0.0002
80 0.0417 nan 0.1000 -0.0002
100 0.0286 nan 0.1000 -0.0002
120 0.0196 nan 0.1000 -0.0003
140 0.0142 nan 0.1000 -0.0003
160 0.0098 nan 0.1000 -0.0000
180 0.0074 nan 0.1000 0.0000
200 0.0055 nan 0.1000 -0.0001
220 0.0041 nan 0.1000 -0.0001
240 0.0030 nan 0.1000 0.0000
260 0.0022 nan 0.1000 -0.0001
280 0.0017 nan 0.1000 0.0000
300 0.0012 nan 0.1000 -0.0000
320 0.0011 nan 0.1000 -0.0000
340 0.0007 nan 0.1000 0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1349 nan 0.1000 0.0731
2 1.0166 nan 0.1000 0.0585
3 0.9210 nan 0.1000 0.0474
4 0.8317 nan 0.1000 0.0455
5 0.7565 nan 0.1000 0.0332
6 0.6954 nan 0.1000 0.0273
7 0.6350 nan 0.1000 0.0277
8 0.5868 nan 0.1000 0.0226
9 0.5399 nan 0.1000 0.0213
10 0.5002 nan 0.1000 0.0180
20 0.2673 nan 0.1000 0.0049
40 0.1078 nan 0.1000 0.0000
60 0.0588 nan 0.1000 -0.0007
80 0.0389 nan 0.1000 -0.0005
100 0.0264 nan 0.1000 -0.0001
120 0.0166 nan 0.1000 0.0001
140 0.0119 nan 0.1000 -0.0001
160 0.0078 nan 0.1000 -0.0000
180 0.0056 nan 0.1000 -0.0001
200 0.0043 nan 0.1000 -0.0000
220 0.0033 nan 0.1000 -0.0000
240 0.0024 nan 0.1000 -0.0001
260 0.0020 nan 0.1000 -0.0000
280 0.0016 nan 0.1000 -0.0000
300 0.0011 nan 0.1000 -0.0000
320 0.0009 nan 0.1000 -0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0005 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0002 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1362 nan 0.1000 0.0673
2 1.0171 nan 0.1000 0.0571
3 0.9143 nan 0.1000 0.0484
4 0.8277 nan 0.1000 0.0392
5 0.7516 nan 0.1000 0.0350
6 0.6888 nan 0.1000 0.0268
7 0.6286 nan 0.1000 0.0273
8 0.5822 nan 0.1000 0.0206
9 0.5341 nan 0.1000 0.0223
10 0.4953 nan 0.1000 0.0157
20 0.2710 nan 0.1000 0.0059
40 0.1102 nan 0.1000 0.0001
60 0.0576 nan 0.1000 -0.0005
80 0.0407 nan 0.1000 -0.0015
100 0.0272 nan 0.1000 -0.0005
120 0.0193 nan 0.1000 -0.0003
140 0.0133 nan 0.1000 -0.0004
160 0.0092 nan 0.1000 -0.0001
180 0.0061 nan 0.1000 -0.0001
200 0.0045 nan 0.1000 -0.0001
220 0.0032 nan 0.1000 -0.0000
240 0.0023 nan 0.1000 -0.0001
260 0.0017 nan 0.1000 -0.0001
280 0.0014 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1367 nan 0.1000 0.0799
2 1.0113 nan 0.1000 0.0588
3 0.9111 nan 0.1000 0.0477
4 0.8227 nan 0.1000 0.0429
5 0.7496 nan 0.1000 0.0316
6 0.6814 nan 0.1000 0.0321
7 0.6282 nan 0.1000 0.0239
8 0.5802 nan 0.1000 0.0230
9 0.5375 nan 0.1000 0.0189
10 0.5015 nan 0.1000 0.0153
20 0.2652 nan 0.1000 0.0052
40 0.1092 nan 0.1000 -0.0000
60 0.0577 nan 0.1000 -0.0006
80 0.0335 nan 0.1000 0.0000
100 0.0228 nan 0.1000 0.0001
120 0.0156 nan 0.1000 -0.0001
140 0.0105 nan 0.1000 -0.0001
160 0.0066 nan 0.1000 -0.0001
180 0.0048 nan 0.1000 -0.0001
200 0.0041 nan 0.1000 -0.0001
220 0.0030 nan 0.1000 -0.0001
240 0.0023 nan 0.1000 -0.0001
260 0.0017 nan 0.1000 -0.0000
280 0.0012 nan 0.1000 -0.0000
300 0.0009 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1337 nan 0.1000 0.0727
2 1.0061 nan 0.1000 0.0577
3 0.9016 nan 0.1000 0.0474
4 0.8122 nan 0.1000 0.0436
5 0.7370 nan 0.1000 0.0352
6 0.6708 nan 0.1000 0.0299
7 0.6119 nan 0.1000 0.0245
8 0.5633 nan 0.1000 0.0215
9 0.5190 nan 0.1000 0.0180
10 0.4801 nan 0.1000 0.0183
20 0.2560 nan 0.1000 0.0046
40 0.1069 nan 0.1000 -0.0007
60 0.0629 nan 0.1000 -0.0002
80 0.0389 nan 0.1000 -0.0007
100 0.0265 nan 0.1000 -0.0008
120 0.0177 nan 0.1000 -0.0005
140 0.0122 nan 0.1000 -0.0001
160 0.0080 nan 0.1000 -0.0001
180 0.0051 nan 0.1000 -0.0001
200 0.0037 nan 0.1000 -0.0000
220 0.0031 nan 0.1000 -0.0001
240 0.0024 nan 0.1000 -0.0001
260 0.0018 nan 0.1000 0.0000
280 0.0012 nan 0.1000 0.0000
300 0.0009 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 0.0000
360 0.0004 nan 0.1000 0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1471 nan 0.1000 0.0669
2 1.0395 nan 0.1000 0.0476
3 0.9467 nan 0.1000 0.0480
4 0.8642 nan 0.1000 0.0397
5 0.7925 nan 0.1000 0.0339
6 0.7360 nan 0.1000 0.0312
7 0.6830 nan 0.1000 0.0254
8 0.6341 nan 0.1000 0.0209
9 0.5967 nan 0.1000 0.0193
10 0.5578 nan 0.1000 0.0167
20 0.3582 nan 0.1000 0.0047
40 0.2303 nan 0.1000 0.0005
60 0.1875 nan 0.1000 0.0003
80 0.1639 nan 0.1000 -0.0007
100 0.1466 nan 0.1000 -0.0007
120 0.1341 nan 0.1000 -0.0008
140 0.1249 nan 0.1000 -0.0003
160 0.1199 nan 0.1000 -0.0005
180 0.1150 nan 0.1000 -0.0010
200 0.1091 nan 0.1000 -0.0008
220 0.1049 nan 0.1000 -0.0011
240 0.1010 nan 0.1000 -0.0008
260 0.1004 nan 0.1000 -0.0004
280 0.0988 nan 0.1000 -0.0011
300 0.0939 nan 0.1000 -0.0003
320 0.0912 nan 0.1000 -0.0009
340 0.0888 nan 0.1000 -0.0006
360 0.0868 nan 0.1000 -0.0009
380 0.0869 nan 0.1000 -0.0010
400 0.0843 nan 0.1000 -0.0006
420 0.0850 nan 0.1000 -0.0011
440 0.0841 nan 0.1000 -0.0003
460 0.0827 nan 0.1000 -0.0003
480 0.0812 nan 0.1000 -0.0012
500 0.0807 nan 0.1000 -0.0004
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1395 nan 0.1000 0.0737
2 1.0166 nan 0.1000 0.0593
3 0.9162 nan 0.1000 0.0493
4 0.8289 nan 0.1000 0.0428
5 0.7546 nan 0.1000 0.0350
6 0.6894 nan 0.1000 0.0320
7 0.6372 nan 0.1000 0.0238
8 0.5902 nan 0.1000 0.0213
9 0.5541 nan 0.1000 0.0165
10 0.5144 nan 0.1000 0.0178
20 0.3018 nan 0.1000 0.0016
40 0.1804 nan 0.1000 -0.0006
60 0.1397 nan 0.1000 -0.0016
80 0.1143 nan 0.1000 -0.0006
100 0.0981 nan 0.1000 -0.0016
120 0.0838 nan 0.1000 -0.0003
140 0.0732 nan 0.1000 -0.0011
160 0.0629 nan 0.1000 -0.0008
180 0.0565 nan 0.1000 -0.0005
200 0.0511 nan 0.1000 -0.0004
220 0.0456 nan 0.1000 -0.0006
240 0.0416 nan 0.1000 -0.0004
260 0.0374 nan 0.1000 -0.0004
280 0.0338 nan 0.1000 -0.0005
300 0.0309 nan 0.1000 -0.0000
320 0.0290 nan 0.1000 -0.0003
340 0.0269 nan 0.1000 -0.0001
360 0.0244 nan 0.1000 -0.0003
380 0.0232 nan 0.1000 -0.0002
400 0.0216 nan 0.1000 -0.0001
420 0.0200 nan 0.1000 0.0001
440 0.0181 nan 0.1000 -0.0002
460 0.0167 nan 0.1000 -0.0001
480 0.0157 nan 0.1000 -0.0001
500 0.0145 nan 0.1000 -0.0001
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1362 nan 0.1000 0.0736
2 1.0107 nan 0.1000 0.0609
3 0.9082 nan 0.1000 0.0485
4 0.8214 nan 0.1000 0.0417
5 0.7484 nan 0.1000 0.0362
6 0.6836 nan 0.1000 0.0311
7 0.6293 nan 0.1000 0.0275
8 0.5800 nan 0.1000 0.0227
9 0.5361 nan 0.1000 0.0213
10 0.4959 nan 0.1000 0.0176
20 0.2752 nan 0.1000 0.0051
40 0.1543 nan 0.1000 -0.0014
60 0.1074 nan 0.1000 -0.0006
80 0.0817 nan 0.1000 0.0001
100 0.0646 nan 0.1000 -0.0006
120 0.0532 nan 0.1000 -0.0001
140 0.0453 nan 0.1000 -0.0004
160 0.0369 nan 0.1000 -0.0002
180 0.0302 nan 0.1000 -0.0004
200 0.0255 nan 0.1000 -0.0002
220 0.0216 nan 0.1000 -0.0003
240 0.0184 nan 0.1000 -0.0003
260 0.0150 nan 0.1000 -0.0002
280 0.0126 nan 0.1000 -0.0001
300 0.0109 nan 0.1000 -0.0001
320 0.0094 nan 0.1000 -0.0001
340 0.0080 nan 0.1000 -0.0001
360 0.0068 nan 0.1000 -0.0001
380 0.0060 nan 0.1000 -0.0000
400 0.0053 nan 0.1000 -0.0000
420 0.0044 nan 0.1000 -0.0000
440 0.0038 nan 0.1000 -0.0000
460 0.0034 nan 0.1000 -0.0000
480 0.0029 nan 0.1000 -0.0000
500 0.0025 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1332 nan 0.1000 0.0718
2 1.0043 nan 0.1000 0.0626
3 0.9047 nan 0.1000 0.0474
4 0.8178 nan 0.1000 0.0411
5 0.7488 nan 0.1000 0.0337
6 0.6847 nan 0.1000 0.0306
7 0.6287 nan 0.1000 0.0268
8 0.5792 nan 0.1000 0.0215
9 0.5335 nan 0.1000 0.0220
10 0.4962 nan 0.1000 0.0150
20 0.2747 nan 0.1000 0.0062
40 0.1322 nan 0.1000 -0.0001
60 0.0865 nan 0.1000 -0.0011
80 0.0620 nan 0.1000 -0.0007
100 0.0446 nan 0.1000 -0.0004
120 0.0349 nan 0.1000 -0.0004
140 0.0266 nan 0.1000 -0.0003
160 0.0213 nan 0.1000 -0.0002
180 0.0173 nan 0.1000 -0.0002
200 0.0130 nan 0.1000 -0.0003
220 0.0110 nan 0.1000 -0.0001
240 0.0090 nan 0.1000 -0.0001
260 0.0074 nan 0.1000 -0.0002
280 0.0061 nan 0.1000 -0.0001
300 0.0046 nan 0.1000 -0.0001
320 0.0039 nan 0.1000 -0.0001
340 0.0031 nan 0.1000 -0.0000
360 0.0026 nan 0.1000 -0.0000
380 0.0021 nan 0.1000 -0.0000
400 0.0017 nan 0.1000 -0.0000
420 0.0014 nan 0.1000 -0.0000
440 0.0011 nan 0.1000 -0.0000
460 0.0009 nan 0.1000 -0.0000
480 0.0008 nan 0.1000 -0.0000
500 0.0006 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1326 nan 0.1000 0.0707
2 1.0062 nan 0.1000 0.0624
3 0.9061 nan 0.1000 0.0471
4 0.8203 nan 0.1000 0.0399
5 0.7461 nan 0.1000 0.0339
6 0.6796 nan 0.1000 0.0333
7 0.6202 nan 0.1000 0.0298
8 0.5715 nan 0.1000 0.0222
9 0.5285 nan 0.1000 0.0181
10 0.4913 nan 0.1000 0.0169
20 0.2669 nan 0.1000 0.0052
40 0.1270 nan 0.1000 -0.0008
60 0.0764 nan 0.1000 -0.0008
80 0.0498 nan 0.1000 -0.0003
100 0.0355 nan 0.1000 -0.0001
120 0.0248 nan 0.1000 -0.0002
140 0.0187 nan 0.1000 -0.0002
160 0.0151 nan 0.1000 -0.0002
180 0.0105 nan 0.1000 -0.0001
200 0.0083 nan 0.1000 -0.0001
220 0.0066 nan 0.1000 -0.0001
240 0.0054 nan 0.1000 -0.0001
260 0.0042 nan 0.1000 -0.0000
280 0.0031 nan 0.1000 -0.0000
300 0.0025 nan 0.1000 0.0000
320 0.0021 nan 0.1000 -0.0000
340 0.0016 nan 0.1000 -0.0000
360 0.0013 nan 0.1000 -0.0000
380 0.0010 nan 0.1000 -0.0000
400 0.0009 nan 0.1000 -0.0000
420 0.0007 nan 0.1000 0.0000
440 0.0006 nan 0.1000 -0.0000
460 0.0005 nan 0.1000 -0.0000
480 0.0005 nan 0.1000 -0.0000
500 0.0003 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1308 nan 0.1000 0.0746
2 1.0121 nan 0.1000 0.0532
3 0.9105 nan 0.1000 0.0506
4 0.8204 nan 0.1000 0.0444
5 0.7404 nan 0.1000 0.0397
6 0.6794 nan 0.1000 0.0267
7 0.6229 nan 0.1000 0.0229
8 0.5729 nan 0.1000 0.0216
9 0.5323 nan 0.1000 0.0187
10 0.4943 nan 0.1000 0.0150
20 0.2627 nan 0.1000 0.0059
40 0.1123 nan 0.1000 0.0013
60 0.0746 nan 0.1000 -0.0015
80 0.0492 nan 0.1000 -0.0002
100 0.0319 nan 0.1000 -0.0005
120 0.0250 nan 0.1000 -0.0004
140 0.0171 nan 0.1000 -0.0002
160 0.0126 nan 0.1000 -0.0002
180 0.0084 nan 0.1000 -0.0001
200 0.0062 nan 0.1000 -0.0000
220 0.0047 nan 0.1000 -0.0001
240 0.0035 nan 0.1000 -0.0000
260 0.0028 nan 0.1000 -0.0000
280 0.0021 nan 0.1000 -0.0000
300 0.0016 nan 0.1000 -0.0000
320 0.0013 nan 0.1000 -0.0000
340 0.0009 nan 0.1000 -0.0000
360 0.0007 nan 0.1000 0.0000
380 0.0007 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1283 nan 0.1000 0.0768
2 1.0007 nan 0.1000 0.0611
3 0.9009 nan 0.1000 0.0493
4 0.8158 nan 0.1000 0.0419
5 0.7425 nan 0.1000 0.0344
6 0.6781 nan 0.1000 0.0315
7 0.6212 nan 0.1000 0.0268
8 0.5726 nan 0.1000 0.0226
9 0.5323 nan 0.1000 0.0173
10 0.4942 nan 0.1000 0.0152
20 0.2605 nan 0.1000 0.0042
40 0.1140 nan 0.1000 0.0004
60 0.0620 nan 0.1000 -0.0010
80 0.0379 nan 0.1000 -0.0004
100 0.0253 nan 0.1000 -0.0002
120 0.0171 nan 0.1000 -0.0003
140 0.0121 nan 0.1000 -0.0004
160 0.0084 nan 0.1000 -0.0001
180 0.0059 nan 0.1000 -0.0001
200 0.0043 nan 0.1000 -0.0001
220 0.0038 nan 0.1000 -0.0001
240 0.0031 nan 0.1000 0.0000
260 0.0020 nan 0.1000 -0.0001
280 0.0013 nan 0.1000 0.0000
300 0.0011 nan 0.1000 -0.0000
320 0.0008 nan 0.1000 -0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0005 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1352 nan 0.1000 0.0764
2 1.0084 nan 0.1000 0.0584
3 0.9052 nan 0.1000 0.0501
4 0.8148 nan 0.1000 0.0448
5 0.7366 nan 0.1000 0.0350
6 0.6703 nan 0.1000 0.0299
7 0.6150 nan 0.1000 0.0229
8 0.5660 nan 0.1000 0.0225
9 0.5247 nan 0.1000 0.0174
10 0.4907 nan 0.1000 0.0152
20 0.2597 nan 0.1000 0.0039
40 0.1129 nan 0.1000 -0.0004
60 0.0620 nan 0.1000 -0.0005
80 0.0398 nan 0.1000 -0.0002
100 0.0262 nan 0.1000 -0.0002
120 0.0177 nan 0.1000 -0.0002
140 0.0122 nan 0.1000 -0.0004
160 0.0079 nan 0.1000 -0.0003
180 0.0059 nan 0.1000 -0.0000
200 0.0041 nan 0.1000 -0.0001
220 0.0028 nan 0.1000 -0.0000
240 0.0019 nan 0.1000 -0.0000
260 0.0015 nan 0.1000 -0.0000
280 0.0011 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0000 nan 0.1000 -0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1319 nan 0.1000 0.0768
2 1.0097 nan 0.1000 0.0575
3 0.9010 nan 0.1000 0.0520
4 0.8182 nan 0.1000 0.0368
5 0.7401 nan 0.1000 0.0372
6 0.6727 nan 0.1000 0.0305
7 0.6163 nan 0.1000 0.0268
8 0.5656 nan 0.1000 0.0237
9 0.5228 nan 0.1000 0.0191
10 0.4806 nan 0.1000 0.0177
20 0.2537 nan 0.1000 0.0066
40 0.1029 nan 0.1000 -0.0006
60 0.0591 nan 0.1000 -0.0008
80 0.0360 nan 0.1000 -0.0005
100 0.0251 nan 0.1000 -0.0004
120 0.0177 nan 0.1000 -0.0001
140 0.0131 nan 0.1000 -0.0004
160 0.0095 nan 0.1000 -0.0000
180 0.0068 nan 0.1000 -0.0002
200 0.0054 nan 0.1000 -0.0000
220 0.0045 nan 0.1000 -0.0000
240 0.0031 nan 0.1000 -0.0000
260 0.0024 nan 0.1000 -0.0000
280 0.0016 nan 0.1000 -0.0000
300 0.0012 nan 0.1000 -0.0000
320 0.0009 nan 0.1000 -0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1361 nan 0.1000 0.0664
2 1.0081 nan 0.1000 0.0651
3 0.9089 nan 0.1000 0.0453
4 0.8242 nan 0.1000 0.0398
5 0.7505 nan 0.1000 0.0360
6 0.6855 nan 0.1000 0.0304
7 0.6271 nan 0.1000 0.0261
8 0.5758 nan 0.1000 0.0234
9 0.5285 nan 0.1000 0.0218
10 0.4886 nan 0.1000 0.0177
20 0.2665 nan 0.1000 0.0035
40 0.1112 nan 0.1000 0.0006
60 0.0654 nan 0.1000 -0.0002
80 0.0423 nan 0.1000 -0.0003
100 0.0278 nan 0.1000 -0.0008
120 0.0200 nan 0.1000 -0.0002
140 0.0135 nan 0.1000 -0.0004
160 0.0098 nan 0.1000 -0.0003
180 0.0081 nan 0.1000 -0.0002
200 0.0055 nan 0.1000 -0.0001
220 0.0039 nan 0.1000 -0.0001
240 0.0028 nan 0.1000 -0.0001
260 0.0022 nan 0.1000 -0.0001
280 0.0015 nan 0.1000 -0.0000
300 0.0012 nan 0.1000 -0.0000
320 0.0009 nan 0.1000 -0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1486 nan 0.1000 0.0693
2 1.0371 nan 0.1000 0.0560
3 0.9406 nan 0.1000 0.0438
4 0.8594 nan 0.1000 0.0364
5 0.7884 nan 0.1000 0.0352
6 0.7323 nan 0.1000 0.0272
7 0.6816 nan 0.1000 0.0256
8 0.6412 nan 0.1000 0.0166
9 0.6004 nan 0.1000 0.0184
10 0.5655 nan 0.1000 0.0174
20 0.3605 nan 0.1000 0.0051
40 0.2243 nan 0.1000 -0.0003
60 0.1808 nan 0.1000 -0.0010
80 0.1563 nan 0.1000 -0.0013
100 0.1349 nan 0.1000 -0.0002
120 0.1245 nan 0.1000 -0.0007
140 0.1107 nan 0.1000 0.0003
160 0.1032 nan 0.1000 -0.0005
180 0.0980 nan 0.1000 -0.0017
200 0.0934 nan 0.1000 -0.0008
220 0.0906 nan 0.1000 0.0001
240 0.0872 nan 0.1000 -0.0005
260 0.0837 nan 0.1000 -0.0011
280 0.0809 nan 0.1000 -0.0003
300 0.0775 nan 0.1000 0.0002
320 0.0746 nan 0.1000 -0.0002
340 0.0730 nan 0.1000 -0.0007
360 0.0715 nan 0.1000 -0.0005
380 0.0686 nan 0.1000 -0.0005
400 0.0677 nan 0.1000 -0.0004
420 0.0662 nan 0.1000 -0.0005
440 0.0656 nan 0.1000 -0.0004
460 0.0640 nan 0.1000 -0.0008
480 0.0640 nan 0.1000 -0.0005
500 0.0624 nan 0.1000 -0.0006
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1349 nan 0.1000 0.0777
2 1.0132 nan 0.1000 0.0609
3 0.9132 nan 0.1000 0.0517
4 0.8321 nan 0.1000 0.0363
5 0.7590 nan 0.1000 0.0348
6 0.6956 nan 0.1000 0.0303
7 0.6448 nan 0.1000 0.0252
8 0.5950 nan 0.1000 0.0228
9 0.5534 nan 0.1000 0.0191
10 0.5143 nan 0.1000 0.0172
20 0.2924 nan 0.1000 0.0059
40 0.1708 nan 0.1000 -0.0005
60 0.1252 nan 0.1000 -0.0010
80 0.0978 nan 0.1000 -0.0009
100 0.0808 nan 0.1000 -0.0002
120 0.0700 nan 0.1000 -0.0004
140 0.0605 nan 0.1000 -0.0001
160 0.0513 nan 0.1000 -0.0005
180 0.0461 nan 0.1000 -0.0005
200 0.0397 nan 0.1000 -0.0002
220 0.0359 nan 0.1000 -0.0004
240 0.0316 nan 0.1000 -0.0001
260 0.0287 nan 0.1000 -0.0002
280 0.0260 nan 0.1000 -0.0002
300 0.0235 nan 0.1000 -0.0001
320 0.0221 nan 0.1000 -0.0002
340 0.0203 nan 0.1000 -0.0004
360 0.0177 nan 0.1000 -0.0001
380 0.0161 nan 0.1000 -0.0002
400 0.0142 nan 0.1000 -0.0001
420 0.0128 nan 0.1000 -0.0002
440 0.0118 nan 0.1000 -0.0000
460 0.0105 nan 0.1000 -0.0001
480 0.0093 nan 0.1000 -0.0001
500 0.0087 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1351 nan 0.1000 0.0762
2 1.0134 nan 0.1000 0.0566
3 0.9129 nan 0.1000 0.0478
4 0.8316 nan 0.1000 0.0415
5 0.7607 nan 0.1000 0.0349
6 0.6939 nan 0.1000 0.0330
7 0.6405 nan 0.1000 0.0233
8 0.5900 nan 0.1000 0.0245
9 0.5457 nan 0.1000 0.0215
10 0.5070 nan 0.1000 0.0177
20 0.2781 nan 0.1000 0.0050
40 0.1420 nan 0.1000 0.0008
60 0.0982 nan 0.1000 -0.0009
80 0.0759 nan 0.1000 -0.0006
100 0.0561 nan 0.1000 -0.0002
120 0.0433 nan 0.1000 -0.0003
140 0.0334 nan 0.1000 -0.0006
160 0.0274 nan 0.1000 -0.0001
180 0.0228 nan 0.1000 -0.0001
200 0.0188 nan 0.1000 -0.0000
220 0.0159 nan 0.1000 -0.0002
240 0.0130 nan 0.1000 0.0000
260 0.0110 nan 0.1000 -0.0001
280 0.0095 nan 0.1000 -0.0002
300 0.0080 nan 0.1000 -0.0001
320 0.0070 nan 0.1000 -0.0001
340 0.0061 nan 0.1000 -0.0001
360 0.0051 nan 0.1000 0.0000
380 0.0042 nan 0.1000 -0.0000
400 0.0035 nan 0.1000 -0.0001
420 0.0029 nan 0.1000 -0.0000
440 0.0024 nan 0.1000 -0.0000
460 0.0020 nan 0.1000 -0.0000
480 0.0016 nan 0.1000 -0.0000
500 0.0014 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1318 nan 0.1000 0.0754
2 1.0063 nan 0.1000 0.0592
3 0.9009 nan 0.1000 0.0511
4 0.8209 nan 0.1000 0.0364
5 0.7440 nan 0.1000 0.0350
6 0.6802 nan 0.1000 0.0292
7 0.6276 nan 0.1000 0.0232
8 0.5778 nan 0.1000 0.0223
9 0.5346 nan 0.1000 0.0211
10 0.4963 nan 0.1000 0.0193
20 0.2649 nan 0.1000 0.0055
40 0.1214 nan 0.1000 -0.0004
60 0.0760 nan 0.1000 -0.0006
80 0.0575 nan 0.1000 -0.0008
100 0.0426 nan 0.1000 -0.0007
120 0.0300 nan 0.1000 -0.0004
140 0.0227 nan 0.1000 -0.0003
160 0.0177 nan 0.1000 -0.0001
180 0.0132 nan 0.1000 -0.0001
200 0.0103 nan 0.1000 -0.0001
220 0.0080 nan 0.1000 -0.0000
240 0.0065 nan 0.1000 -0.0001
260 0.0052 nan 0.1000 -0.0001
280 0.0042 nan 0.1000 -0.0001
300 0.0033 nan 0.1000 -0.0000
320 0.0024 nan 0.1000 -0.0000
340 0.0021 nan 0.1000 -0.0000
360 0.0018 nan 0.1000 -0.0000
380 0.0016 nan 0.1000 -0.0001
400 0.0013 nan 0.1000 -0.0000
420 0.0010 nan 0.1000 0.0000
440 0.0008 nan 0.1000 -0.0000
460 0.0006 nan 0.1000 -0.0000
480 0.0005 nan 0.1000 -0.0000
500 0.0004 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1364 nan 0.1000 0.0751
2 1.0112 nan 0.1000 0.0606
3 0.9097 nan 0.1000 0.0465
4 0.8215 nan 0.1000 0.0454
5 0.7483 nan 0.1000 0.0345
6 0.6827 nan 0.1000 0.0306
7 0.6260 nan 0.1000 0.0263
8 0.5801 nan 0.1000 0.0206
9 0.5385 nan 0.1000 0.0189
10 0.4989 nan 0.1000 0.0172
20 0.2594 nan 0.1000 0.0059
40 0.1147 nan 0.1000 0.0008
60 0.0709 nan 0.1000 -0.0014
80 0.0460 nan 0.1000 -0.0006
100 0.0315 nan 0.1000 -0.0003
120 0.0214 nan 0.1000 -0.0002
140 0.0160 nan 0.1000 -0.0003
160 0.0114 nan 0.1000 -0.0003
180 0.0091 nan 0.1000 -0.0002
200 0.0067 nan 0.1000 0.0000
220 0.0049 nan 0.1000 -0.0001
240 0.0037 nan 0.1000 -0.0001
260 0.0028 nan 0.1000 -0.0001
280 0.0020 nan 0.1000 -0.0000
300 0.0014 nan 0.1000 -0.0000
320 0.0011 nan 0.1000 -0.0000
340 0.0009 nan 0.1000 -0.0000
360 0.0007 nan 0.1000 -0.0000
380 0.0005 nan 0.1000 -0.0000
400 0.0004 nan 0.1000 -0.0000
420 0.0003 nan 0.1000 -0.0000
440 0.0003 nan 0.1000 -0.0000
460 0.0002 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1330 nan 0.1000 0.0734
2 1.0096 nan 0.1000 0.0591
3 0.9074 nan 0.1000 0.0469
4 0.8220 nan 0.1000 0.0383
5 0.7435 nan 0.1000 0.0371
6 0.6788 nan 0.1000 0.0277
7 0.6284 nan 0.1000 0.0221
8 0.5749 nan 0.1000 0.0237
9 0.5341 nan 0.1000 0.0186
10 0.4962 nan 0.1000 0.0163
20 0.2647 nan 0.1000 0.0054
40 0.1160 nan 0.1000 0.0003
60 0.0678 nan 0.1000 -0.0004
80 0.0453 nan 0.1000 -0.0005
100 0.0302 nan 0.1000 -0.0003
120 0.0192 nan 0.1000 -0.0004
140 0.0121 nan 0.1000 -0.0002
160 0.0081 nan 0.1000 -0.0000
180 0.0058 nan 0.1000 -0.0001
200 0.0041 nan 0.1000 -0.0000
220 0.0028 nan 0.1000 -0.0001
240 0.0021 nan 0.1000 -0.0001
260 0.0015 nan 0.1000 -0.0000
280 0.0011 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0004 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1286 nan 0.1000 0.0762
2 1.0057 nan 0.1000 0.0573
3 0.9064 nan 0.1000 0.0462
4 0.8172 nan 0.1000 0.0419
5 0.7359 nan 0.1000 0.0343
6 0.6717 nan 0.1000 0.0288
7 0.6160 nan 0.1000 0.0274
8 0.5663 nan 0.1000 0.0222
9 0.5248 nan 0.1000 0.0180
10 0.4818 nan 0.1000 0.0205
20 0.2629 nan 0.1000 0.0039
40 0.1285 nan 0.1000 -0.0005
60 0.0668 nan 0.1000 -0.0005
80 0.0357 nan 0.1000 -0.0002
100 0.0240 nan 0.1000 -0.0004
120 0.0180 nan 0.1000 -0.0003
140 0.0120 nan 0.1000 -0.0002
160 0.0087 nan 0.1000 -0.0002
180 0.0054 nan 0.1000 0.0000
200 0.0037 nan 0.1000 -0.0000
220 0.0023 nan 0.1000 -0.0000
240 0.0017 nan 0.1000 -0.0000
260 0.0013 nan 0.1000 -0.0000
280 0.0011 nan 0.1000 -0.0000
300 0.0008 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0004 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0002 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0000 nan 0.1000 0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1362 nan 0.1000 0.0808
2 1.0087 nan 0.1000 0.0627
3 0.9071 nan 0.1000 0.0496
4 0.8222 nan 0.1000 0.0395
5 0.7433 nan 0.1000 0.0381
6 0.6755 nan 0.1000 0.0337
7 0.6213 nan 0.1000 0.0239
8 0.5722 nan 0.1000 0.0228
9 0.5321 nan 0.1000 0.0158
10 0.4904 nan 0.1000 0.0189
20 0.2564 nan 0.1000 0.0045
40 0.0965 nan 0.1000 -0.0001
60 0.0489 nan 0.1000 -0.0003
80 0.0306 nan 0.1000 -0.0005
100 0.0174 nan 0.1000 -0.0001
120 0.0129 nan 0.1000 -0.0003
140 0.0081 nan 0.1000 -0.0001
160 0.0053 nan 0.1000 -0.0001
180 0.0037 nan 0.1000 -0.0001
200 0.0027 nan 0.1000 -0.0001
220 0.0022 nan 0.1000 -0.0001
240 0.0019 nan 0.1000 -0.0001
260 0.0014 nan 0.1000 -0.0000
280 0.0009 nan 0.1000 -0.0000
300 0.0006 nan 0.1000 -0.0000
320 0.0006 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1323 nan 0.1000 0.0686
2 1.0041 nan 0.1000 0.0634
3 0.9018 nan 0.1000 0.0488
4 0.8174 nan 0.1000 0.0390
5 0.7427 nan 0.1000 0.0371
6 0.6804 nan 0.1000 0.0292
7 0.6201 nan 0.1000 0.0280
8 0.5749 nan 0.1000 0.0218
9 0.5313 nan 0.1000 0.0205
10 0.4973 nan 0.1000 0.0148
20 0.2640 nan 0.1000 0.0067
40 0.1056 nan 0.1000 0.0006
60 0.0576 nan 0.1000 -0.0006
80 0.0341 nan 0.1000 -0.0007
100 0.0230 nan 0.1000 -0.0004
120 0.0153 nan 0.1000 -0.0003
140 0.0107 nan 0.1000 -0.0003
160 0.0075 nan 0.1000 -0.0002
180 0.0065 nan 0.1000 -0.0002
200 0.0044 nan 0.1000 -0.0001
220 0.0036 nan 0.1000 -0.0002
240 0.0022 nan 0.1000 -0.0000
260 0.0017 nan 0.1000 -0.0001
280 0.0015 nan 0.1000 -0.0001
300 0.0009 nan 0.1000 -0.0000
320 0.0008 nan 0.1000 -0.0000
340 0.0006 nan 0.1000 -0.0000
360 0.0003 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0003 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1306 nan 0.1000 0.0777
2 1.0014 nan 0.1000 0.0570
3 0.9031 nan 0.1000 0.0472
4 0.8138 nan 0.1000 0.0431
5 0.7376 nan 0.1000 0.0373
6 0.6702 nan 0.1000 0.0295
7 0.6145 nan 0.1000 0.0252
8 0.5651 nan 0.1000 0.0225
9 0.5198 nan 0.1000 0.0198
10 0.4833 nan 0.1000 0.0155
20 0.2539 nan 0.1000 0.0061
40 0.1016 nan 0.1000 -0.0001
60 0.0497 nan 0.1000 -0.0006
80 0.0287 nan 0.1000 -0.0000
100 0.0167 nan 0.1000 -0.0003
120 0.0102 nan 0.1000 -0.0002
140 0.0076 nan 0.1000 -0.0003
160 0.0046 nan 0.1000 0.0000
180 0.0036 nan 0.1000 -0.0001
200 0.0024 nan 0.1000 -0.0000
220 0.0021 nan 0.1000 -0.0001
240 0.0018 nan 0.1000 -0.0000
260 0.0014 nan 0.1000 -0.0000
280 0.0012 nan 0.1000 -0.0001
300 0.0008 nan 0.1000 -0.0000
320 0.0007 nan 0.1000 -0.0000
340 0.0005 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0003 nan 0.1000 -0.0000
400 0.0002 nan 0.1000 -0.0000
420 0.0001 nan 0.1000 -0.0000
440 0.0001 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 0.0000
500 0.0000 nan 0.1000 -0.0000
Iter TrainDeviance ValidDeviance StepSize Improve
1 1.1327 nan 0.1000 0.0755
2 1.0042 nan 0.1000 0.0633
3 0.9032 nan 0.1000 0.0484
4 0.8145 nan 0.1000 0.0408
5 0.7413 nan 0.1000 0.0335
6 0.6778 nan 0.1000 0.0302
7 0.6235 nan 0.1000 0.0242
8 0.5730 nan 0.1000 0.0226
9 0.5296 nan 0.1000 0.0196
10 0.4911 nan 0.1000 0.0161
20 0.2733 nan 0.1000 0.0062
40 0.1181 nan 0.1000 0.0005
60 0.0695 nan 0.1000 -0.0006
80 0.0437 nan 0.1000 -0.0004
100 0.0306 nan 0.1000 -0.0004
120 0.0222 nan 0.1000 -0.0004
140 0.0152 nan 0.1000 -0.0003
160 0.0111 nan 0.1000 -0.0001
180 0.0080 nan 0.1000 -0.0001
200 0.0056 nan 0.1000 -0.0001
220 0.0040 nan 0.1000 -0.0000
240 0.0028 nan 0.1000 -0.0000
260 0.0021 nan 0.1000 -0.0000
280 0.0016 nan 0.1000 -0.0000
300 0.0012 nan 0.1000 -0.0000
320 0.0009 nan 0.1000 -0.0000
340 0.0007 nan 0.1000 -0.0000
360 0.0006 nan 0.1000 -0.0000
380 0.0004 nan 0.1000 -0.0000
400 0.0003 nan 0.1000 -0.0000
420 0.0002 nan 0.1000 -0.0000
440 0.0002 nan 0.1000 -0.0000
460 0.0001 nan 0.1000 -0.0000
480 0.0001 nan 0.1000 -0.0000
500 0.0001 nan 0.1000 -0.0000
Setting levels: control = benign, case = malignant
Setting direction: controls < cases
roc_gbm
Call:
roc.default(response = BreastCancer$Class, predictor = as.numeric(pred_gbm))
Data: as.numeric(pred_gbm) in 458 controls (BreastCancer$Class benign) < 241 cases (BreastCancer$Class malignant).
Area under the curve: 0.975
10.3.2.2 Light Gradient Boosting Machine
Light gbm is built by Microsoft as an improvement on GBM. It uses a histogram based method to partition the data row wise and column wise. Light GBM performs analysis using gradient based one-sided sampling whereby instances with large gradients are retained and random sampling of instances with small gradients are performed. Light GBM uses a leaf-wise tree growth strategy. Leaf-wise growth is built using global best split and can lead to increase depth of trees and asymmetric trees. This method also uses feature bundling strategy by combining several features together. Overfitting is reduced by employing regularisation. It can tolerate missing values and handle categorical data without need for one hot encoding. The data is prepared using the function sparse.model.matrix. The target variable needs to be factor of 0 and 1.
library(parsnip)
Attaching package: 'parsnip'
The following object is masked from 'package:randomForestSRC':
tune
The following object is masked from 'package:party':
fit
The following object is masked from 'package:modeltools':
fit
library(recipes)data("BreastCancer",package ="mlbench") #convert Class to numericBreastCancer$Class<-as.character(BreastCancer$Class)BreastCancer$Class[BreastCancer$Class=="benign"]<-0BreastCancer$Class[BreastCancer$Class=="malignant"]<-1BreastCancer$Class<-as.factor(BreastCancer$Class)BreastCancer2<-BreastCancer[,-1]BC_split <- rsample::initial_split(BreastCancer2,prop =0.75)preprocessing_recipe <-recipes::recipe(Class ~ ., data =training(BC_split)) %>%# combine low frequency factor levelsrecipes::step_other(all_nominal(), threshold =0.01) %>%# remove no variance predictors which provide no predictive information recipes::step_nzv(all_nominal()) %>%# prep the recipe so it can be used on other dataprep()
LightGBM_model<-parsnip::boost_tree(mode ="classification",trees =1000,min_n =tune(),learn_rate =tune(),tree_depth =tune()) %>%set_engine("lightgbm")LightGBM_params <-dials::parameters(min_n(), # min data in leaftree_depth(range =c(2,10)),learn_rate())
library(bonsai)
Warning: package 'bonsai' was built under R version 4.3.3
LGBM_grid <-dials::grid_max_entropy(LightGBM_params,size =20# set this to a higher number to get better results# I don't want to run this all night, so I set it to 30)head(LGBM_grid)
# emptyLightGBM_modelBoosted Tree Model Specification (classification)Main Arguments:trees =1000min_n =tune()tree_depth =tune()learn_rate =tune()Engine-Specific Arguments:loss_function = accuracyComputational engine: lightgbm# filled inLGBM_model_finalBoosted Tree Model Specification (regression)Main Arguments:trees =1000min_n =5tree_depth =4learn_rate =0.0438633239970453Engine-Specific Arguments:loss_function = squarederrorComputational engine: catboost
10.3.2.3 Extreme gradient boost machine
XGBoost uses a level-wise growth strategy compared to leaf wise strategy of light GBM. Overfitting is reduced by employing regularisation. It requires treatment missing values separately. In the examples above, the outcome variable is treated as a factor. Extreme gradient boost machine xgboost requires conversion to numeric variable.
library(xgboost)
Attaching package: 'xgboost'
The following object is masked from 'package:dplyr':
slice
The following object is masked from 'package:rattle':
xgboost
library(caret)data("BreastCancer",package ="mlbench")#predict breast cancerBreastCancer$Class<-as.character(BreastCancer$Class)BreastCancer$Class[BreastCancer$Class=="benign"]<-0BreastCancer$Class[BreastCancer$Class=="malignant"]<-1BreastCancer$Class<-as.numeric(BreastCancer$Class)#remove ID column#remaining 9 columns#convert multiple columns to numeric as Breast Cancer data has many columns containing factors#lapply output a listBreastCancer2<-lapply(BreastCancer[,-c(1,7)], as.numeric)BreastCancer2<-as.data.frame(BreastCancer2)set.seed(1234)parts =createDataPartition(BreastCancer2$Class, p =0.75, list=F)train = BreastCancer2[parts, ]test = BreastCancer2[-parts, ]X_train =data.matrix(train[,-9]) # independent variables for trainy_train = train[,9] # dependent variables for trainX_test =data.matrix(test[,-9]) # independent variables for testy_test = test[,9] # dependent variables for test# convert the train and test data into xgboost matrix type.xgboost_train =xgb.DMatrix(data=X_train, label=as.matrix(y_train))xgboost_test =xgb.DMatrix(data=X_test, label=as.matrix(y_test))# train a model using our training data# nthread is the number of CPU threads we use# nrounds is the number of passes on the data#the function xgboost exist in xgboost and rattlemodel <- xgboost::xgboost(data = xgboost_train, max.depth =2, eta =1, nthread =2, nrounds =2, objective ="binary:logistic", verbose =2)
Attaching package: 'mlr3pipelines'
The following object is masked from 'package:bitops':
%>>%
library(mlr3hyperband)
Loading required package: mlr3tuning
Loading required package: paradox
Attaching package: 'mlr3tuning'
The following object is masked from 'package:tune':
tune
The following object is masked from 'package:parsnip':
tune
The following object is masked from 'package:randomForestSRC':
tune
library(BBmisc)
Attaching package: 'BBmisc'
The following objects are masked from 'package:dplyr':
coalesce, collapse, symdiff
The following object is masked from 'package:grid':
explode
The following object is masked from 'package:base':
isFALSE
data(Melanoma, package ="MASS")N <-length(Melanoma$status)#table(Melanoma$ph.karno, cancer$pat.karno)## if physician's KPS unavailable, then use the patient's#h <- which(is.na(cancer$ph.karno))#cancer$ph.karno[h] <- cancer$pat.karno[h]times <- Melanoma$timetimes <-ceiling(times/7) ## weeks#1 died from melanoma, 2 alive, 3 dead from other causes.##delta: 0=censored, 1=deaddelta=ifelse(Melanoma$status==2,0,1)## matrix of observed covariatesx.train <-cbind(Melanoma$sex, Melanoma$age, Melanoma$thickness)#provide column namesdimnames(x.train)[[2]] <-c('M(1):F(0)','age', 'thickness')
10.3.2.5 CatBoost
Catboost, or category boosting, is a machine learning method used in search engine (Yandex), recommender system, self driving cars among others. It shares the features of other tree-based boosting algorithm in that it sequentially builds simple decision tree models and tries improve by adding new terms to the model to correct the errors of the preceding models. CatBoost is a boosted method designed to handle categorical data without further need for preprocessing of the categorical data. It automatically performs one hot encoding of categorical data. Unlike other machine learning methods which requires scaling prior to analysis, CatBoost internally scale the data. It uses symmetric weighted quantile sketch(SWQS) to handle missing values.
Catboost employs several strategies to handle collinearity issue and prevent overfitting such as penalty term and random permutation of the data during training and separately from during the gradient descent calculation to decrease reliance on any particular weight. This step is different from the boosting strategy for XGboost.
Registered S3 method overwritten by 'treesnip':
method from
multi_predict._lgb.Booster bonsai
Attaching package: 'treesnip'
The following objects are masked from 'package:bonsai':
predict_lightgbm_classification_class,
predict_lightgbm_classification_prob,
predict_lightgbm_classification_raw,
predict_lightgbm_regression_numeric, train_lightgbm
#remotes::install_github("curso-r/treesnip@catboost")library(catboost)data("BreastCancer",package ="mlbench") #convert Class to numericBreastCancer$Class<-as.character(BreastCancer$Class)BreastCancer$Class[BreastCancer$Class=="benign"]<-0BreastCancer$Class[BreastCancer$Class=="malignant"]<-1BreastCancer$Class<-as.factor(BreastCancer$Class)BreastCancer2<-BreastCancer[,-1]BC_split <- rsample::initial_split(BreastCancer2,prop =0.75)preprocessing_recipe <-recipes::recipe(Class ~ ., data =training(BC_split)) %>%# combine low frequency factor levelsrecipes::step_other(all_nominal(), threshold =0.01) %>%# remove no variance predictors which provide no predictive information recipes::step_nzv(all_nominal()) %>%# prep the recipe so it can be used on other dataprep()
CatBoost_params <-dials::parameters(min_n(), # min data in leaftree_depth(range =c(4,10)), # depth# In most cases, the optimal depth ranges from 4 to 10. # Values in the range from 6 to 10 are recommended. learn_rate() # learning rate)
cbst_grid <-dials::grid_max_entropy(CatBoost_params,size =20# set this to a higher number to get better results# I don't want to run this all night, so I set it to 30)head(cbst_grid)
# create train settrain_processed <-bake(preprocessing_recipe, new_data =training(BC_split))# fit model on entire trainsettrained_model_all_data <- cbst_model_final %>%# fit the model on all the training datafit(formula = Class ~ .,data = train_processed)train_prediction <-trained_model_all_data %>%predict(new_data = train_processed) %>%bind_cols(training(BC_split))
library(catboost)#partition data parts =createDataPartition(BreastCancer$Class, p =0.75, list=F)train = BreastCancer[parts, ]test = BreastCancer[-parts, ]set.seed(1234)#prepare data using load pool function#target variable is labelfeatures<-train[,-c(1,9)]labels<-as.numeric(train[,c(9)])train_pool<-catboost.load_pool(data=features,label=labels)features<-test[,-c(1,9)]labels=as.numeric(test[,c(9)])test_pool<-catboost.load_pool(data=features,label =labels )# train a model using our training data#test data is set to NULLmodel <-catboost.train(data = train_pool, NULL,params =list(loss_function ='Logloss',iterations =100, metric_period=10)) summary(model)
BART or Bayesian additive regression trees is a non-parametric method that uses a sum of Bayesian trees to estimate an unknown function. Every tree acts as a weak learner in this ensemble method. It can also be used in causal inference. It uses tuning parameters derived from Bayesian priors. Each predicted value has a posterior distribution. BART uses a regularization prior that forces each tree to be able to explain only a limited subset of the relationships between the covariates and the predictor variable. In some instances, BART outperforms xgboost.
library(BART)
Loading required package: nlme
Attaching package: 'nlme'
The following object is masked from 'package:BBmisc':
collapse
The following object is masked from 'package:dplyr':
collapse
Loading required package: nnet
data(Melanoma, package ="MASS")N <-length(Melanoma$status)#table(Melanoma$ph.karno, cancer$pat.karno)## if physician's KPS unavailable, then use the patient's#h <- which(is.na(cancer$ph.karno))#cancer$ph.karno[h] <- cancer$pat.karno[h]times <- Melanoma$timetimes <-ceiling(times/7) ## weeks#1 died from melanoma, 2 alive, 3 dead from other causes.##delta: 0=censored, 1=deaddelta=ifelse(Melanoma$status==2,0,1)## matrix of observed covariatesx.train <-cbind(Melanoma$sex, Melanoma$age, Melanoma$thickness)#provide column namesdimnames(x.train)[[2]] <-c('M(1):F(0)','age', 'thickness')table(x.train[ , 1])
0 1
126 79
summary(x.train[ , 2])
Min. 1st Qu. Median Mean 3rd Qu. Max.
4.00 42.00 54.00 52.46 65.00 95.00
##test BART with token run to ensure installation worksset.seed(99)post <-surv.bart(x.train=x.train, times=times, delta=delta,nskip=1, ndpost=1, keepevery=1)
post <-surv.bart(x.train=x.train, times=times, delta=delta,seed=99)
*****Calling gbart: type=2
*****Data:
data:n,p,np: 17042, 4, 0
y1,yn: 1.000000, 0.000000
x1,x[n*p]: 2.000000, 2.900000
*****Number of Trees: 50
*****Number of Cut Points: 100 ... 63
*****burn,nd,thin: 250,10000,10
*****Prior:beta,alpha,tau,nu,lambda,offset: 2,0.95,0.212132,3,1,-2.6383
*****Dirichlet:sparse,theta,omega,a,b,rho,augment: 0,0,1,0.5,1,3,0
*****printevery: 100
MCMC
done 0 (out of 10250)
done 100 (out of 10250)
done 200 (out of 10250)
done 300 (out of 10250)
done 400 (out of 10250)
done 500 (out of 10250)
done 600 (out of 10250)
done 700 (out of 10250)
done 800 (out of 10250)
done 900 (out of 10250)
done 1000 (out of 10250)
done 1100 (out of 10250)
done 1200 (out of 10250)
done 1300 (out of 10250)
done 1400 (out of 10250)
done 1500 (out of 10250)
done 1600 (out of 10250)
done 1700 (out of 10250)
done 1800 (out of 10250)
done 1900 (out of 10250)
done 2000 (out of 10250)
done 2100 (out of 10250)
done 2200 (out of 10250)
done 2300 (out of 10250)
done 2400 (out of 10250)
done 2500 (out of 10250)
done 2600 (out of 10250)
done 2700 (out of 10250)
done 2800 (out of 10250)
done 2900 (out of 10250)
done 3000 (out of 10250)
done 3100 (out of 10250)
done 3200 (out of 10250)
done 3300 (out of 10250)
done 3400 (out of 10250)
done 3500 (out of 10250)
done 3600 (out of 10250)
done 3700 (out of 10250)
done 3800 (out of 10250)
done 3900 (out of 10250)
done 4000 (out of 10250)
done 4100 (out of 10250)
done 4200 (out of 10250)
done 4300 (out of 10250)
done 4400 (out of 10250)
done 4500 (out of 10250)
done 4600 (out of 10250)
done 4700 (out of 10250)
done 4800 (out of 10250)
done 4900 (out of 10250)
done 5000 (out of 10250)
done 5100 (out of 10250)
done 5200 (out of 10250)
done 5300 (out of 10250)
done 5400 (out of 10250)
done 5500 (out of 10250)
done 5600 (out of 10250)
done 5700 (out of 10250)
done 5800 (out of 10250)
done 5900 (out of 10250)
done 6000 (out of 10250)
done 6100 (out of 10250)
done 6200 (out of 10250)
done 6300 (out of 10250)
done 6400 (out of 10250)
done 6500 (out of 10250)
done 6600 (out of 10250)
done 6700 (out of 10250)
done 6800 (out of 10250)
done 6900 (out of 10250)
done 7000 (out of 10250)
done 7100 (out of 10250)
done 7200 (out of 10250)
done 7300 (out of 10250)
done 7400 (out of 10250)
done 7500 (out of 10250)
done 7600 (out of 10250)
done 7700 (out of 10250)
done 7800 (out of 10250)
done 7900 (out of 10250)
done 8000 (out of 10250)
done 8100 (out of 10250)
done 8200 (out of 10250)
done 8300 (out of 10250)
done 8400 (out of 10250)
done 8500 (out of 10250)
done 8600 (out of 10250)
done 8700 (out of 10250)
done 8800 (out of 10250)
done 8900 (out of 10250)
done 9000 (out of 10250)
done 9100 (out of 10250)
done 9200 (out of 10250)
done 9300 (out of 10250)
done 9400 (out of 10250)
done 9500 (out of 10250)
done 9600 (out of 10250)
done 9700 (out of 10250)
done 9800 (out of 10250)
done 9900 (out of 10250)
done 10000 (out of 10250)
done 10100 (out of 10250)
done 10200 (out of 10250)
time: 269s
trcnt,tecnt: 1000,0
pre <-surv.pre.bart(times=times, delta=delta, x.train=x.train,x.test=x.train)K <- pre$KM <-nrow(post$yhat.train)pre$tx.test <-rbind(pre$tx.test, pre$tx.test)pre$tx.test[ , 2] <-c(rep(1, N*K), rep(2, N*K))## sex pushed to col 2, since time is always in col 1pred <-predict(post, newdata=pre$tx.test)
*****In main of C++ for bart prediction
tc (threadcount): 1
number of bart draws: 1000
number of trees in bart sum: 50
number of x columns: 4
from x,np,p: 4, 67240
***using serial code
K nearest neighbour (KNN) uses ’feature similarity based on measure of distance between data points to make prediction. The K in KNN refers to the number of neighbours to define the case for similarity. K nearest neighbour is available from the caret library.
#note Class is benign or malignant of class factor#column Bare.nuclei removed due to NABreastCancer<-BreastCancer[,-c(1,7)]#split dataset.seed(123)split = caTools::sample.split(BreastCancer$Class, SplitRatio =0.75)Train =subset(BreastCancer, split ==TRUE)Test =subset(BreastCancer, split ==FALSE)#grid of values to test in cross-validation.knn_Grid <-expand.grid(k =c(1:15))knn_Control <-trainControl(method ="cv",number =10, # repeats = 10, # uncomment for repeatedcv ## Estimate class probabilitiesclassProbs =TRUE,## Evaluate performance using ## the following functionsummaryFunction = twoClassSummary)#scaling data is performed here under preProcessknn <- caret::train(Class ~ ., data = Train, method ="knn",trControl=knn_Control,tuneGrid=knn_Grid,#optimise with roc metricmetric="ROC")summary(knn)
Length Class Mode
learn 2 -none- list
k 1 -none- numeric
theDots 0 -none- list
xNames 71 -none- character
problemType 1 -none- character
tuneValue 1 data.frame list
obsLevels 2 -none- character
param 0 -none- list
Setting levels: control = benign, case = malignant
Setting direction: controls < cases
roc_knn
Call:
roc.default(response = Test$Class, predictor = as.numeric(pred_knn))
Data: as.numeric(pred_knn) in 114 controls (Test$Class benign) < 60 cases (Test$Class malignant).
Area under the curve: 0.7917
#https://plotly.com/r/knn-classification/pdb <-cbind(Test[,-9], Test[,9])pdb <-cbind(pdb, pred_knn)fig <- plotly::plot_ly(data = pdb,x =~as.numeric(Test$Cl.thickness), y =~as.numeric(Test$Epith.c.size), type ='scatter', mode ='markers',color =~pred_knn, colors ='RdBu', symbol =~Test$Class, split =~Test$Class, symbols =c('square-dot','circle-dot'), marker =list(size =12, line =list(color ='black', width =1)))fig
10.5 Support vector machine
In brief, support vector machine regression (SVR) can be seen as a way to enhance data which may not be easily separated in its native space. It manipulates data from low dimension to higher dimension in feature space and which can reveal relationship not discernible in low dimensional space. It does this around the hyperparameter controlling the margin of the data from a fitted line in a way not dissimilar from fitting a regression line based on minimising least squares. The default setting is radial basis function.
library(e1071)
Attaching package: 'e1071'
The following object is masked from 'package:mlr3tuning':
tune
The following object is masked from 'package:mlr3verse':
tune
The following object is masked from 'package:tune':
tune
The following object is masked from 'package:rsample':
permutations
The following object is masked from 'package:parsnip':
tune
The following objects are masked from 'package:randomForestSRC':
impute, tune
library(caret)# The Breast cancer data is used again from knntrctrl <-trainControl(method ="repeatedcv", number =10, repeats =3)#scaling data is performed here under preProcesssvm_Linear <- caret::train(Class ~ ., data = Train, method ="svmLinear",trControl=trctrl,preProcess =c("center", "scale"),tuneLength =10)summary(svm_Linear)
Setting levels: control = benign, case = malignant
Setting direction: controls < cases
roc_svm
Call:
roc.default(response = BreastCancer$Class, predictor = as.numeric(pred))
Data: as.numeric(pred) in 458 controls (BreastCancer$Class benign) < 241 cases (BreastCancer$Class malignant).
Area under the curve: 0.9698
10.6 Non-negative matrix factorisation
Non-negative matrix factorisation is an unsupervised machine learning method, which seeks to explain the observed clinical features using smaller number of basis components (hidden variables). A matrix V of dimension m x n is factorised to 2 matrices W and H. W has dimensions m x k an H has dimensions n x k. For topic modeling in the chapter of text mining, V matrix is the document term matrix. Each row of H is the word embedding and the columns of W represent the weight.
The interpretation of NMF components is similar to, but often more natural than, related methods such as factor analysis and principal component analysis. The non-negativity constraint in NMF leads to a simple “parts-based” interpretation and has been successfully used in facial recognition, metagene pattern discovery, and market research. For a clinical example, the matrix for NMF decomposition consists of rows of hospitals and their service availability.
The example below used the recommended procedure to estimate the factorization rank, based on stability of the cophenetic correlation coefficient and the residual error, prior to performing the NMF analysis. The data were permuted and the factorization rank computed. These data were used as reference for selecting factorization rank to minimize the chance of overfitting.
There are several versions of NMF with each method different in the choice of cost function. The default method is Lee and Seung (Lee 1999). This method uses the square error Frobenius norm as the cost function to minimise WH or determine the size of distance from the matrix V to W. Another popular method for decomposing gene data is the Brunet method which uses the Kullback-Leibler divergence norm as the cost function (Brunet et al. 2004).
The optimal number of rank for this data is likely to be 4. The output of the search is fed into the NMF analysis below.
#Using the data above we can use which argument to find the order#since the starting point is 2 we just need to add 1Rank=which(estim.r$measures$cophenetic==max(estim.r$measures$cophenetic))+1model<-nmf(df_se, Rank,nrun=100)pmodel<-predict(model,prob=TRUE)coefmap(model)
basismap(model)
consensusmap(model)
10.7 Formal concept analysis
This is an unsupervised machine learning method which takes an input matrix of objects and attributes (binary values) and seeks to find the hierarchy of relations. Each concept shares a set of attributes with other objects and each sub-concept shares a smaller set of attributes with a subset of the objects.
10.7.1 Hasse diagram
A Hasse diagram is used to display the hierarchy of relations. First we will illustrate with a simple relationship among fruit. Note in this example there is no close set for apple and pear, as both share the attribute of green color. There is a close set for the tropical fruit mango and and banana. There are several libraries for FCA. Here we will use multiplex. The fcaR library can also handle fuzzy data.
#BiocManager::install("Rgraphviz")library(multiplex) #Algebraic Tools for the Analysis of Multiple Social Networks
Attaching package: 'multiplex'
The following object is masked from 'package:mlr3tuning':
ti
The following objects are masked from 'package:mlr3verse':
flt, ti
library(Rgraphviz) #plot hasse diagram
Loading required package: graph
Attaching package: 'graph'
The following object is masked from 'package:rsample':
complement
The following objects are masked from 'package:dials':
degree, threshold
The following object is masked from 'package:stringr':
boundary
The following object is masked from 'package:party':
nodes
The following object is masked from 'package:strucchange':
boundary
Attaching package: 'Rgraphviz'
The following object is masked from 'package:NMF':
name
fr<-data.frame(Fruit=c("Apple", "Banana","Pear", "Mango"),round=c(1,0,0,0),cylindrical=c(0,1,0,0),yellow=c(0,1,0,1),red=c(1,0,1,1),green=c(1,0,1,0), #color when ripetropical=c(0,1,0,1),large_seed=c(0,0,0,1))df<-fr[,c(2:dim(fr)[2])]row.names(df)<-fr[,1] #bipartite matrix#perform Galois derivations between partially ordered subsets#galois(df_se',labeling = "full")gf <-galois(df, labeling ="reduced")#partial ordering of conceptpo<-partial.order(gf,type="galois")diagram(po, main="Hasse diagram of partial order - Fruit")
#lattice diagram with reduced contextdiagram.levels(po)
Next we illustrate FCA in network of hospitals in South-Eastern Melbourne. The objects are the hospitals and the attributes are the services available in those hospitals.
#library(multiplex) #Algebraic Tools for the Analysis of Multiple Social Networks#library(Rgraphviz) #plot hasse diagram#install BiocManager::install("Rgraphviz")edge<-read.csv("./Data-Use/Hosp_Network_geocoded.csv")df<-edge[,c(2:dim(edge)[2])]row.names(df)<-edge[,1] #bipartite matrix#select columns#remove distance datadf_se<-edge[,c(2:16)]row.names(df_se)<-edge[,1] #bipartite matrix#south eastern hospitals#select rowsdf_se<-df_se[c(1,6,7,11,12,13,14,17,19,20,24,31,33,34,35),]#perform Galois derivations between partially ordered subsets#galois(df_se',labeling = "full")gf <-galois(df_se, labeling ="reduced")#partial ordering of conceptpo<-partial.order(gf,type="galois")diagram(po, main="Hasse diagram of partial order with reduced context")
#lattice diagram with reduced contextdiagram.levels(po)
This Hasse diagram is only a subset of the data for Melbourne. If we plot data for the entire country then it can be hard to visualise. One way is to embed this in Shiny app and use the search term to look at subset of the data https://gntem3.shinyapps.io/epilepsyadmissions/.
10.8 Evolutionary Algorithm
Evolutionary algorithm are search method which take the source of inspiration from nature such as evolution and survival of the fittest. These are seen as heuristic based method. The results from evolutionary algorithm shouldn’t be compared unless all conditions set are the same. In essence the findings are similar under the same conditions.
10.8.1 Simulated Annealing
This method uses idea in metallurgy whereby metal is heated and then cooled to alter its property.
#SA section is set not to run as the analysis takes a long time.# a saved run is provided belowdata("BreastCancer",package ="mlbench")colnames(BreastCancer)#check for duplicatessum(duplicated(BreastCancer))#remove duplicates#keep Id to avoid creation of new duplicatesBreastCancer1<-unique(BreastCancer) #reduce 699 to 691 rows#convert multiple columns to numeric#lapply output a listBreastCancer2<-lapply(BreastCancer1[,-c(7,11)], as.numeric) #listBreastCancer2<-as.data.frame(BreastCancer2)BreastCancer2$Class<-BreastCancer1$Classx=BreastCancer2[,-10]y=BreastCancer2$Classsa_ctrl <-safsControl(functions = rfSA,method ="repeatedcv",repeats =3, #default is 5improve =50)set.seed(10)glm_sa <-safs(x = x, y = y,iters =5, #default is 250safsControl = sa_ctrl, method="glm")#save(glm_sa,file="Logistic_SimulatedAnnealing.Rda")###############################################Simulated Annealing Feature Selection##691 samples#9 predictors#2 classes: 'benign', 'malignant' ##Maximum search iterations: 5 #Restart after 50 iterations without improvement (0 restarts on average)##Internal performance values: Accuracy, Kappa#Subset selection driven to maximize internal Accuracy ##External performance values: Accuracy, Kappa#Best iteration chose by maximizing external Accuracy #External resampling method: Cross-Validated (10 fold, repeated 3 times) #During resampling:# * the top 5 selected variables (out of a possible 9):# Bl.cromatin (56.7%), Id (46.7%), Cl.thickness (43.3%), Epith.c.size (43.3%), #Marg.adhesion (43.3%)# * on average, 3.5 variables were selected (min = 2, max = 5)##In the final search using the entire training set:# * 2 features selected at iteration 5 including:# Cl.thickness, Cell.size # * external performance at this iteration is## Accuracy Kappa # 0.9314 0.8479
load("./Logistic_SimulatedAnnealing.Rda")#plot output of simulated annealingplot(glm_sa)
10.8.2 Genetic Algorithm
Genetic algorithm is a machine learning tool based on ideas from Darwin’s concept of natural selection. It is based on mutation, crossover and selection. Genetic algorithm can be used in any situation. The issue is in finding the fitness function to evaluate the output. Since it does not depend on gradient descent algorithm, it is less likely to be stuck in local minima compared to other machine learning methods. Genetic algorithm is available in R as part of caret and GA libraries. Genetic algorithm can be used to optimise feature selection for regression modelling at the expense of much longer running time.
One potential issue with using cross-validation in genetic algorithm for feature selection is that it would be not right to use it again when feeding this data into another machine learning method. Genetic algorithm takes a long time to run and in the example below, eval is set as false.
#GAlibrary(caret)data("BreastCancer",package ="mlbench")colnames(BreastCancer)#check for duplicatessum(duplicated(BreastCancer))#remove duplicates#keep Id to avoid creation of new duplicatesBreastCancer1<-unique(BreastCancer) #reduce 699 to 691 rows#convert multiple columns to numeric#lapply output a listBreastCancer2<-lapply(BreastCancer1[,-c(7,11)], as.numeric) #listBreastCancer2<-as.data.frame(BreastCancer2)BreastCancer2$Class<-BreastCancer1$Class#check for NAanyNA(BreastCancer2)split = caTools::sample.split(BreastCancer2$Class, SplitRatio =0.7)Train =subset(BreastCancer2, split ==TRUE)Test =subset(BreastCancer2, split ==FALSE)x=Train[,-10]y=Train$Class#cross validation indicates the number of cycle of the procedure from randomly generating new population of chromosome to mutate child chromosome.ga_ctrl <-gafsControl(functions = rfGA,method ="cv",repeats =3, # default is 10genParallel=TRUE, # Use parallel programmingallowParallel =TRUE )## Use the same random number seed as the RFE process## so that the same CV folds are used for the external## resampling. set.seed(10)system.time(glm_ga <-gafs(x = x, y = y,iters =5, #recommended is 200gafsControl = ga_ctrl, method="glm"))#save(glm_ga,file="Logistic_GeneticAlgorithm.Rda")################################################################# The output of glm_ga#Genetic Algorithm Feature Selection#484 samples#9 predictors#2 classes: 'benign', 'malignant' #Maximum generations: 5 #Population per generation: 50 #Crossover probability: 0.8 #Mutation probability: 0.1 #Elitism: 0 ##Internal performance values: Accuracy, Kappa#Subset selection driven to maximize internal Accuracy ##External performance values: Accuracy, Kappa#Best iteration chose by maximizing external Accuracy #External resampling method: Cross-Validated (10 fold) ##During resampling:# * the top 5 selected variables (out of a possible 9):# Cell.shape (100%), Cl.thickness (100%), Epith.c.size (100%), Normal.nucleoli #(100%), Id (90%)# * on average, 6.7 variables were selected (min = 5, max = 8)##In the final search using the entire training set:# * 7 features selected at iteration 2 including:# Cl.thickness, Cell.shape, Marg.adhesion, Epith.c.size, Bl.cromatin ... # * external performance at this iteration is## Accuracy Kappa # 0.9691 0.9328 #
The output from the Genetic Algorithm is plotted as mean fitness by generations. This plot shows the internal and external accuracy estimate from cross validation.
load("./Logistic_GeneticAlgorithm.Rda")#plot output of genetic algorithm plot(glm_ga)
10.9 Manifold learning
Manifold learning has been described as using geometry information in high dimensional space to map data into cluster in lower dimensional space. This is a non-linear reduction technique. Several manifold learning methods are described below but this list is not exhaustive. It is available through maniTools package.
10.9.1 T-Stochastic Neighbourhood Embedding
T-Stochastic Neighbourhood Embedding (TSNE) is a manifold learning method which seeks to transform the complex data into low (2) dimensions while maintaining the distance between neighbouring objects. The distance between data points are can be measured using Euclidean distance or other measures of distance. The transformed data points are conditional probabilities that represents similarities. The original description of TSNE used PCA as a first step to speed up computation and reduce noise.
This method is listed here as it is a form of data reduction method. This non-linear method is different from PCA in that the low dimensional output of TSNE are not intended for machine learning. TSNE is implemented in R as Rtsne. The perplexity parameter allows tuning of the proximity of the data points. The PCA step can be performed within Rtsne by setting the pca argument. The default number of iterations or max_iter is 1000.
library(Rtsne)library(ggplot2)library(mice) #impute missing data
Attaching package: 'mice'
The following objects are masked from 'package:BiocGenerics':
cbind, rbind
The following object is masked from 'package:stats':
filter
The following objects are masked from 'package:base':
cbind, rbind
#check for duplicatessum(duplicated(BreastCancer))
[1] 8
#remove duplicates#keep Id to avoid creation of new duplicatesBreastCancer1<-unique(BreastCancer) #reduce 699 to 691 rows#impute missing data#m is number of multiple imputation, default is 5#output is a listimputed_Data <-mice(BreastCancer1, m=5, maxit =5, method ='pmm', seed =500)
#choose among the 5 imputed datasetcompleteData <-complete(imputed_Data,2)#convert multiple columns to numeric#lapply output a listBreastCancer2<-lapply(completeData[,-c(11)], as.numeric) #listBreastCancer2<-as.data.frame(BreastCancer2)BreastCancer2$Class<-BreastCancer1$ClassBC_unique <-unique(BreastCancer2) # Remove duplicatesset.seed(42) # Sets seed for reproducibilitytsne_out <-Rtsne(as.matrix(BC_unique[,-11]), normalize = T, #normalise datapca=T, dims =3, #pca before analysisperplexity=20, #tuningverbose=FALSE) # Run TSNE#plot(tsne_out$Y,col=BC_unique$Class,asp=1)# Add a new column with colormycolors <-c('red', 'blue')BC_unique$color <- mycolors[ as.numeric(BC_unique$Class) ]#turn off rgl#rgl::plot3d(x=tsne_out$Y[,1], y=tsne_out$Y[,2], z=tsne_out$Y[,3], type = 'p', col=BC_unique$color, size=8)#rgl::legend3d("topright", legend = names(mycolors), pch = 16, col = colors, cex=1, inset=c(0.02))
The example with Breast cancer didn’t turn out as well. Let’s try TSNE with the iris dataset.
#TSNEdata(iris)#5 columnsIris_unique <-unique(iris) # Remove duplicatesset.seed(42) # Sets seed for reproducibilitytsne_out <-Rtsne(as.matrix(Iris_unique[,-5]), dims =2, perplexity=10, verbose=FALSE) # Run TSNEplot(tsne_out$Y,col=Iris_unique$Species,asp=1)
10.9.2 Self organising map
Self organising map (SOM) is an unsupervised machine learning method and is excellent for viewing complex data in low dimensional space i.e. a data reduction method. SOM is available as part of kohonen library. It uses competitive learning to adjust its weight in contrast to other neural network approaches which use backward propagation or gradient descent to update the weight of the features. Each node is evaluated to participate in the neural network. Input vectors that are close to each other in high dimensional space are mapped to be close to each other in low dimensional space. SOM is a competitive neural network and has been considered as a deep learning method.
The codes below are modified from https://rpubs.com/AlgoritmaAcademy/som for use in analysis of iris data. The first illustration is with unsupervised SOM.
library(kohonen)
Attaching package: 'kohonen'
The following object is masked from 'package:purrr':
map
#unsupervised SOM#use iris dataset 150 x 5set.seed(100)#convert to numeric matrixiris.train <-as.matrix(scale(iris[,-5]))# grid should be smaller than dim(iris) 150 x5#xdim =10 and ydim=10 would be < 120iris.grid <-somgrid(xdim =10, ydim =10, topo ="hexagonal")#som modeliris.model <-som(iris.train, iris.grid, rlen =500, radius =2.5, keep.data =TRUE, dist.fcts ="euclidean")plot(iris.model, type ="mapping", pchs =19, shape ="round")
Plot
plot(iris.model, type ="codes", main ="Codes Plot", palette.name = rainbow)
The plot of training shows that the distance between nodes reached a plateau after 300 iterations.
plot(iris.model, type ="changes")
Supervised SOM is now performed with the same iris data.
#SOMset.seed(100)int <-sample(nrow(iris), nrow(iris)*0.8)train <- iris[int,]test <- iris[-int,]# scaling datatrainX <-scale(train[,-5])testX <-scale(test[,-5], center =attr(trainX, "scaled:center"))# make label#iris$species is already of class factortrain.label <- train[,5]test.label <- test[,5]test[,5] <-916testXY <-list(independent = testX, dependent = test.label)# make a train data sets that scaled# convert them to be a numeric matrix iris.train <-as.matrix(scale(train[,-5]))set.seed(100)# grid should be smaller than dim(train) 120 x5#xdim =10 and ydim=10 would be < 120iris.grid <-somgrid(xdim =10, ydim =10, topo ="hexagonal")#som modeliris.model <-som(iris.train, iris.grid, rlen =500, radius =2.5, keep.data =TRUE, dist.fcts ="euclidean")class <-xyf(trainX, classvec2classmat(train.label), iris.grid, rlen =500)plot(class, type ="changes")
pred <-predict(class, newdata = testXY)table(Predict = pred$predictions[[2]], Actual = test.label)
set.seed(100)clust <-kmeans(iris.model$codes[[1]], 6)plot(iris.model, type ="codes", bgcol =rainbow(9)[clust$cluster], main ="Cluster SOM")add.cluster.boundaries(iris.model, clust$cluster)
10.9.3 Multidimensional scaling
MDS is a method of dimensionality reduction which preserves the distance between variables. This method has been used in geography. It is implemented in IsoplotR package and igraph package as layout.mds.
10.10 Deep learning
Deep learning is a neural network with many layers: inner, multiple hidden and outer layer. Deep learning methods can be supervised or unsupervised. It uses gradient descent algorithm in search for the solution. One potential issue that it may be stuck in a local minima rather than the global minima.
There are several R libraries for performing deep learning. It’s worth checking out the installation requirement as some require installing the library in python and uses the reticulate library to perform analysis. The examples used here are R libraries including RSNNS. The instructions for installing Miniconda from reticulate was provided in the earlier chapter on data wrangling Those instruction include installing torch and kerras.
For tabular data, deep learning may not necessarily be better than tree-based machine learning method (Grinsztajn, Oyallon, and Varoquaux 2022) . By contrast, deep learning may be better for unstructured data such as imaging or text. The reasons for this can be due to tabular data having a mixed (continuous and categorical) data structure, sparsity and lack of locality present from pattern in imaging or audio or text. Missing data is poorly handled by deep learning. Newer approach to deep learning such as transformer for tabular data uses approaches such as entity embedding to examine categorical data in vector space after one hot encoding; attentive mechanism to sequentially focus on features. In situations where the data is over 3000 data points transformer method such as TabPFN perform well against tree-based methods (McElfresh et al. 2023)
10.10.0.1 Multiplayer Perceptron
Multilayer perceptron is a type of deep learning. It passes information in one direction from inner to hidden and outer layer and hence is referred to as feed forward artificial neural network. It trains the data using a loss function which adapt to the parameter and optimises according to the specified learning rate. Overfitting is minimised by using an L2 regularisation penalty termed alpha.
library(caret)library(RSNNS)
Loading required package: Rcpp
Attaching package: 'Rcpp'
The following object is masked from 'package:rsample':
populate
Attaching package: 'RSNNS'
The following object is masked from 'package:kohonen':
som
The following object is masked from 'package:parsnip':
mlp
The following objects are masked from 'package:caret':
confusionMatrix, train
#remove ID column#remove column a=with NA #alternative is to imputeBreastCancer<-BreastCancer[,-c(1,7)]#remaining 9 columns#convert multiple columns to numeric#lapply output a listBreastCancer2<-lapply(BreastCancer[,-c(9)], as.numeric)BreastCancer2<-as.data.frame(BreastCancer2)BreastCancer2<-merge(BreastCancer2, BreastCancer$Class)#note Class is benign or malignant of class factor#column Bare.nuclei removed due to NA#split dataset.seed(123)BreastCancer2Values <- BreastCancer2[,c(1:8)]BreastCancer2Targets <-decodeClassLabels(BreastCancer2[,9])#this returns the orginal file as a listBreastCancer2 <-splitForTrainingAndTest(BreastCancer2Values, BreastCancer2Targets, ratio=0.15) #ratio is percentage for test dataBreastCancer2 <-normTrainingAndTestSet(BreastCancer2) #put out a list objectmodel <-mlp(BreastCancer2$inputsTrain, BreastCancer2$targetsTrain, size=5, #number of unit in hidden layer learnFuncParams=c(0.1), maxit=50, #number of iteration to learninputsTest=BreastCancer2$inputsTest, targetsTest=BreastCancer2$targetsTest)summary(model)
There are several libraries available in survivalmodels which interface with pycox from Python. First, we illustrate the use of deepsurv.
library(survivalmodels)
Warning: package 'survivalmodels' was built under R version 4.3.2
data(Melanoma, package ="MASS")times <- Melanoma$timetimes <-ceiling(times/7) ## weeks#1 died from melanoma, 2 alive, 3 dead from other causes.##delta: 0=censored, 1=deaddelta=ifelse(Melanoma$status==2,0,1)## matrix of observed covariatesx.train <-cbind(Melanoma$sex, Melanoma$age, Melanoma$thickness)deepsurv(data = Melanoma, frac =0.3, #Fraction of data to use for validation activation ="relu",num_nodes =c(4L, 8L, 4L, 2L), dropout =0.1, early_stopping =TRUE, epochs =100L, #number of epochs.batch_size =32L)
Using the same Melanoma dataset, we illustrate DeepHit.
DH<-deephit(data = Melanoma, frac =0.3, #Fraction of data to use for validation activation ="relu",num_nodes =c(4L, 8L, 4L, 2L), dropout =0.1, early_stopping =TRUE, epochs =100L, #number of epochs.batch_size =32L)summary(DH)
Warning: package 'mlr3proba' was built under R version 4.3.2
library(mlr3extralearners)library(mlr3pipelines)library(mlr3tuning)## get the `whas` task from mlr3probawhas <-tsk("whas")## create our own task from the Melanoma datasetMelanoma_data <- MASS::MelanomaMelanoma_data$status=ifelse(Melanoma_data$status==2,0,1)## convert characters to factorsMelanoma_data$sex <-factor(Melanoma_data$sex)MelanomaTS <- TaskSurv$new("Melanoma", Melanoma_data, time ="time", event ="status")#1 died from melanoma, 2 alive, 3 dead from other causes.##delta: 0=censored, 1=deaddelta=ifelse(Melanoma_data$status==2,0,1)## matrix of observed covariatesx.train <-cbind(Melanoma_data$sex, Melanoma_data$age, Melanoma_data$thickness)## combine in listtasks <-list(whas, MelanomaTS)library(paradox)search_space <-ps(## p_dbl for numeric valued parametersdropout =p_dbl(lower =0, upper =1),weight_decay =p_dbl(lower =0, upper =0.5),learning_rate =p_dbl(lower =0, upper =1),## p_int for integer valued parametersnodes =p_int(lower =1, upper =32),k =p_int(lower =1, upper =4))search_space$trafo <-function(x, param_set) { x$num_nodes =rep(x$nodes, x$k) x$nodes = x$k =NULLreturn(x)}create_autotuner <-function(learner) { AutoTuner$new(learner = learner,search_space = search_space,resampling =rsmp("holdout"),measure =msr("surv.cindex"),terminator =trm("evals", n_evals =2),tuner =tnr("random_search") )}## load learnerslearners <-lrns(#paste0("surv.", c("deephit", "deepsurv")), #crash when running multiple learnerspaste0("surv.", c( "deepsurv")),frac =0.3, early_stopping =TRUE, epochs =10, optimizer ="adam")# apply our functionlearners <-lapply(learners, create_autotuner)create_pipeops <-function(learner) {po("encode") %>>%po("scale") %>>%po("learner", learner)}## apply our functionlearners <-lapply(learners, create_pipeops)## select holdout as the resampling strategyresampling <-rsmp("cv", folds =2)## add KM and CPHlearners <-c(learners, lrns(c("surv.kaplan", "surv.coxph")))#learners <- c(learners, lrns(c("surv.coxph")))design <-benchmark_grid(tasks, learners, resampling)bm <-benchmark(design)
INFO [13:40:40.224] [mlr3] Running benchmark with 12 resampling iterations
INFO [13:40:40.467] [mlr3] Applying learner 'encode.scale.surv.deepsurv.tuned' on task 'whas' (iter 1/2)
INFO [13:40:40.874] [bbotk] Starting to optimize 5 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=2, k=0]'
INFO [13:40:40.933] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:40.975] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:40.989] [mlr3] Applying learner 'surv.deepsurv' on task 'whas' (iter 1/1)
INFO [13:40:41.269] [mlr3] Finished benchmark
INFO [13:40:41.365] [bbotk] Result of batch 1:
INFO [13:40:41.371] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:41.371] [bbotk] 0.6247406 0.4554209 0.8456182 11 3 0.6826164 0 0
INFO [13:40:41.371] [bbotk] runtime_learners uhash
INFO [13:40:41.371] [bbotk] 0.25 9a7bf727-fbd3-479f-bef5-6a82a27087ba
INFO [13:40:41.386] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:41.424] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:41.437] [mlr3] Applying learner 'surv.deepsurv' on task 'whas' (iter 1/1)
INFO [13:40:41.595] [mlr3] Finished benchmark
INFO [13:40:41.662] [bbotk] Result of batch 2:
INFO [13:40:41.666] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:41.666] [bbotk] 0.1573471 0.4268247 0.6872897 27 1 0.6589113 0 0
INFO [13:40:41.666] [bbotk] runtime_learners uhash
INFO [13:40:41.666] [bbotk] 0.14 b40c6235-db90-4e72-8f8b-f9e03a8311a9
INFO [13:40:41.687] [bbotk] Finished optimizing after 2 evaluation(s)
INFO [13:40:41.688] [bbotk] Result:
INFO [13:40:41.691] [bbotk] dropout weight_decay learning_rate nodes k learner_param_vals x_domain
INFO [13:40:41.691] [bbotk] 0.6247406 0.4554209 0.8456182 11 3 <list[8]> <list[4]>
INFO [13:40:41.691] [bbotk] surv.cindex
INFO [13:40:41.691] [bbotk] 0.6826164
INFO [13:40:42.143] [mlr3] Applying learner 'encode.scale.surv.deepsurv.tuned' on task 'whas' (iter 2/2)
INFO [13:40:42.435] [bbotk] Starting to optimize 5 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=2, k=0]'
INFO [13:40:42.484] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:42.516] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:42.530] [mlr3] Applying learner 'surv.deepsurv' on task 'whas' (iter 1/1)
INFO [13:40:42.809] [mlr3] Finished benchmark
INFO [13:40:42.879] [bbotk] Result of batch 1:
INFO [13:40:42.884] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:42.884] [bbotk] 0.9949267 0.2285935 0.8684118 17 4 0.3665406 0 0
INFO [13:40:42.884] [bbotk] runtime_learners uhash
INFO [13:40:42.884] [bbotk] 0.27 b45d4244-f0a5-437f-aa97-ae2a63e009c1
INFO [13:40:42.901] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:42.931] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:42.943] [mlr3] Applying learner 'surv.deepsurv' on task 'whas' (iter 1/1)
INFO [13:40:43.097] [mlr3] Finished benchmark
INFO [13:40:43.160] [bbotk] Result of batch 2:
INFO [13:40:43.163] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:43.163] [bbotk] 0.8363856 0.2299571 0.9908099 13 2 0.4871795 0 0
INFO [13:40:43.163] [bbotk] runtime_learners uhash
INFO [13:40:43.163] [bbotk] 0.14 3f464183-7e74-41f4-b6c3-a66595df0fca
INFO [13:40:43.183] [bbotk] Finished optimizing after 2 evaluation(s)
INFO [13:40:43.185] [bbotk] Result:
INFO [13:40:43.187] [bbotk] dropout weight_decay learning_rate nodes k learner_param_vals x_domain
INFO [13:40:43.187] [bbotk] 0.8363856 0.2299571 0.9908099 13 2 <list[8]> <list[4]>
INFO [13:40:43.187] [bbotk] surv.cindex
INFO [13:40:43.187] [bbotk] 0.4871795
INFO [13:40:43.614] [mlr3] Applying learner 'surv.kaplan' on task 'whas' (iter 1/2)
INFO [13:40:43.660] [mlr3] Applying learner 'surv.kaplan' on task 'whas' (iter 2/2)
INFO [13:40:43.706] [mlr3] Applying learner 'surv.coxph' on task 'whas' (iter 1/2)
INFO [13:40:43.794] [mlr3] Applying learner 'surv.coxph' on task 'whas' (iter 2/2)
INFO [13:40:43.857] [mlr3] Applying learner 'encode.scale.surv.deepsurv.tuned' on task 'Melanoma' (iter 1/2)
INFO [13:40:44.131] [bbotk] Starting to optimize 5 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=2, k=0]'
INFO [13:40:44.175] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:44.202] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:44.216] [mlr3] Applying learner 'surv.deepsurv' on task 'Melanoma' (iter 1/1)
INFO [13:40:44.394] [mlr3] Finished benchmark
INFO [13:40:44.457] [bbotk] Result of batch 1:
INFO [13:40:44.461] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:44.461] [bbotk] 0.4570555 0.2741379 0.8601232 20 1 0.6296296 0 0
INFO [13:40:44.461] [bbotk] runtime_learners uhash
INFO [13:40:44.461] [bbotk] 0.18 7a2a9150-18fc-41fc-bdf3-c1c7e6d8305a
INFO [13:40:44.476] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:44.508] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:44.521] [mlr3] Applying learner 'surv.deepsurv' on task 'Melanoma' (iter 1/1)
INFO [13:40:44.669] [mlr3] Finished benchmark
INFO [13:40:44.739] [bbotk] Result of batch 2:
INFO [13:40:44.742] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:44.742] [bbotk] 0.7608218 0.479834 0.7939875 20 3 0.6333333 0 0
INFO [13:40:44.742] [bbotk] runtime_learners uhash
INFO [13:40:44.742] [bbotk] 0.12 86016f68-30c1-41cb-aba6-b3d22afa3dda
INFO [13:40:44.763] [bbotk] Finished optimizing after 2 evaluation(s)
INFO [13:40:44.764] [bbotk] Result:
INFO [13:40:44.767] [bbotk] dropout weight_decay learning_rate nodes k learner_param_vals x_domain
INFO [13:40:44.767] [bbotk] 0.7608218 0.479834 0.7939875 20 3 <list[8]> <list[4]>
INFO [13:40:44.767] [bbotk] surv.cindex
INFO [13:40:44.767] [bbotk] 0.6333333
INFO [13:40:45.250] [mlr3] Applying learner 'encode.scale.surv.deepsurv.tuned' on task 'Melanoma' (iter 2/2)
INFO [13:40:45.529] [bbotk] Starting to optimize 5 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=2, k=0]'
INFO [13:40:45.575] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:45.606] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:45.619] [mlr3] Applying learner 'surv.deepsurv' on task 'Melanoma' (iter 1/1)
INFO [13:40:45.753] [mlr3] Finished benchmark
INFO [13:40:45.813] [bbotk] Result of batch 1:
INFO [13:40:45.817] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:45.817] [bbotk] 0.1140623 0.2846605 0.22349 10 4 0.7709677 0 0
INFO [13:40:45.817] [bbotk] runtime_learners uhash
INFO [13:40:45.817] [bbotk] 0.1 ecc4b9f5-28cb-4a63-a822-046c620e1609
INFO [13:40:45.830] [bbotk] Evaluating 1 configuration(s)
INFO [13:40:45.859] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:40:45.872] [mlr3] Applying learner 'surv.deepsurv' on task 'Melanoma' (iter 1/1)
INFO [13:40:46.013] [mlr3] Finished benchmark
INFO [13:40:46.079] [bbotk] Result of batch 2:
INFO [13:40:46.083] [bbotk] dropout weight_decay learning_rate nodes k surv.cindex warnings errors
INFO [13:40:46.083] [bbotk] 0.2337101 0.3132217 0.5829765 8 3 0.683871 0 0
INFO [13:40:46.083] [bbotk] runtime_learners uhash
INFO [13:40:46.083] [bbotk] 0.12 27616ac3-5685-44c0-860b-ba6bf93cab40
INFO [13:40:46.104] [bbotk] Finished optimizing after 2 evaluation(s)
INFO [13:40:46.105] [bbotk] Result:
INFO [13:40:46.107] [bbotk] dropout weight_decay learning_rate nodes k learner_param_vals x_domain
INFO [13:40:46.107] [bbotk] 0.1140623 0.2846605 0.22349 10 4 <list[8]> <list[4]>
INFO [13:40:46.107] [bbotk] surv.cindex
INFO [13:40:46.107] [bbotk] 0.7709677
INFO [13:40:46.560] [mlr3] Applying learner 'surv.kaplan' on task 'Melanoma' (iter 1/2)
INFO [13:40:46.593] [mlr3] Applying learner 'surv.kaplan' on task 'Melanoma' (iter 2/2)
INFO [13:40:46.628] [mlr3] Applying learner 'surv.coxph' on task 'Melanoma' (iter 1/2)
INFO [13:40:46.671] [mlr3] Applying learner 'surv.coxph' on task 'Melanoma' (iter 2/2)
INFO [13:40:46.711] [mlr3] Finished benchmark
## Aggreggate with Harrell's C and Integrated Graf Scoremsrs <-msrs(c("surv.cindex", "surv.graf"))bm$aggregate(msrs)[, c(3, 4, 7, 8)]
Warning: 'as.BenchmarkAggr' is deprecated.
Use 'as_benchmark_aggr' instead.
See help("Deprecated")
## run global Friedman testbma$friedman_test()
Friedman rank sum test
data: cindex and learner_id and task_id
Friedman chi-squared = 4, df = 2, p-value = 0.1353
10.10.1 Transformer
Transformer is a deep neural network architecture that uses entity embedding (vectorisation of categorical data) and attentive mechanism (differential weighting of tokens by their importance).
10.10.1.1 Tabnet
Tabnet can be run as part of mlrverse. In this example, the code runs locally but not when knitted by quarto. Hence, the eval function is set to false. This is due PyTorch.
library(tidyverse)library(mlr3verse)library(tabnet)library(recipes)library(yardstick)data("BreastCancer",package ="mlbench")#The Breast Cancer data contains NA as well as factors#note Class is benign or malignant of class factor#column Bare.nuclei removed due to NABreastCancer<-BreastCancer[,-c(1,7)] #%>% #mutate(Class=ifelse(Class=="malignant",1,0))#split data using caTools. #The next example will use createDataPartition from caretset.seed(123)split = caTools::sample.split(BreastCancer$Class, SplitRatio =0.75)Train =subset(BreastCancer, split ==TRUE)Test =subset(BreastCancer, split ==FALSE)rec <-recipe(Class ~ ., data = Train) %>%step_normalize(all_numeric(), -all_outcomes())#epoch is number of epochfit <-tabnet_fit(rec, Train, epochs =30, valid_split=0.1, learn_rate =5e-3)
cbind(Test, predict(fit, Test, type ="prob")) %>%#check the data frame as the argument is taken from the truth factor Class #and contains benign and malignantroc_auc(Class, .pred_benign)#roc_auc binary 0.9219298
10.10.1.1.1 Attention heatmap
Explain model on Test data with with attention heatmap
TabPFN is a transformer neural network. It does not need preprocessing of the data. It performs its own preprocessing, including z-score normalization and outlier handling. There is no R implementation of TabPFN at this stage.
10.10.2 CNN
Convolution neural network or CNN is an artifical neural network method that is well suited to classification of image data. CNN is able to develop an internal representation of the image.
10.10.3 RNN
Recurrent neural network or RNN is an artifical neural network method that is well suited to data with repeated patterns such as natural language processing. However, this architecture is less suited for tabular or imaging data.
10.10.4 Reinforcement learning
Reinforcement learning is an unsupervised method which uses trial and error for agent to learn and adapt.
library(ReinforcementLearning)
Brunet, Jean-Philippe, Pablo Tamayo, Todd R. Golub, and Jill P. Mesirov. 2004. “Metagenes and Molecular Pattern Discovery Using Matrix Factorization.”Proceedings of the National Academy of Sciences 101 (12): 4164–69. https://doi.org/10.1073/pnas.0308531101.
Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. 2022. “Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?”ArXiv abs/2207.08815. https://doi.org/10.48550/arXiv.2207.08815.
Lee, Seung, D. 1999. “Learning the Parts of Objects by Non-Negative Matrix Factorization.”Nature 401: 788–91. https://doi.org/https://doi.org/10.1038/44565.
McElfresh, Duncan C., Sujay Khandagale, Jonathan Valverde, C. VishakPrasad, Ben Feuer, Chinmay Hegde, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. 2023. “When Do Neural Nets Outperform Boosted Trees on Tabular Data?”ArXiv abs/2305.02997. https://doi.org/10.48550/arXiv.2305.02997.
Molnar, Christoph, Bernd Bischl, and Giuseppe Casalicchio. 2018. “Iml: An r Package for Interpretable Machine Learning.”JOSS 3 (26): 786. https://doi.org/10.21105/joss.00786.
Phan, T. G., J. Chen, S. Singhal, H. Ma, B. B. Clissold, J. Ly, and R. Beare. 2018. “Exploratory Use of Decision Tree Analysis in Classification of Outcome in Hypoxic-Ischemic Brain Injury.”Front Neurol 9: 126.
Phan, T. G., T. Kooblal, C. Matley, S. Singhal, B. Clissold, J. Ly, A. G. Thrift, V. Srikanth, and H. Ma. 2019. “Stroke Severity Versus Dysphagia Screen as Driver for Post-stroke Pneumonia.”Front Neurol 10: 16.
