Parameters
This page provides the API reference of torchensemble
. Below is a list of functions supported by all ensembles.
fit()
: Training stage of the ensembleevaluate()
: Evaluating stage of the ensemblepredict()
: Return the predictions of the ensembleforward()
: Data forward process of the ensembleset_optimizer()
: Set the parameter optimizer for training the ensembleset_scheduler()
: Set the learning rate scheduler for training the ensemble
Fusion
In fusion-based ensemble methods, the predictions from all base estimators are first aggregated as an average output. After then, the training loss is computed based on this average output and the ground-truth. The training loss is then back-propagated to all base estimators simultaneously.
FusionClassifier
- class torchensemble.fusion.FusionClassifier(*args: Any, **kwargs: Any)
Bases:
BaseClassifier
Implementation on the FusionClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in FusionClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for FusionClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for FusionClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for FusionClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of FusionClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
FusionRegressor
- class torchensemble.fusion.FusionRegressor(*args: Any, **kwargs: Any)
Bases:
BaseRegressor
Implementation on the FusionRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in FusionRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for FusionRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for FusionRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for FusionRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of FusionRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Voting
In voting-based ensemble methods, each base estimator is trained independently, and the final prediction takes the average over predictions from all base estimators.
VotingClassifier
- class torchensemble.voting.VotingClassifier(*args: Any, **kwargs: Any)
Bases:
BaseClassifier
Implementation on the VotingClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in VotingClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for VotingClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for VotingClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for VotingClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of VotingClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
VotingRegressor
- class torchensemble.voting.VotingRegressor(*args: Any, **kwargs: Any)
Bases:
BaseRegressor
Implementation on the VotingRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in VotingRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for VotingRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for VotingRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for VotingRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of VotingRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Bagging
In bagging-based ensemble methods, each base estimator is trained independently. In addition, sampling with replacement is conducted on the training data to further encourage the diversity between different base estimators in the ensemble.
BaggingClassifier
- class torchensemble.bagging.BaggingClassifier(*args: Any, **kwargs: Any)
Bases:
BaseClassifier
Implementation on the BaggingClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in BaggingClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for BaggingClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for BaggingClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for BaggingClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of BaggingClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
BaggingRegressor
- class torchensemble.bagging.BaggingRegressor(*args: Any, **kwargs: Any)
Bases:
BaseRegressor
Implementation on the BaggingRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in BaggingRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for BaggingRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for BaggingRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for BaggingRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of BaggingRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Gradient Boosting
Gradient boosting is a classic sequential ensemble method. At each iteration, the learning target of a new base estimator is to fit the pseudo residual computed based on the ground truth and the output from base estimators fitted before, using ordinary least square.
Tip
The input argument shrinkage_rate
in gradient_boosting
is also known as learning rate in other gradient boosting libraries such as XGBoost. However, its meaning is totally different from the meaning of learning rate in the context of parameter optimizer in deep learning.
GradientBoostingClassifier
- class torchensemble.gradient_boosting.GradientBoostingClassifier(*args: Any, **kwargs: Any)
Bases:
_BaseGradientBoosting
,BaseClassifier
Implementation on the GradientBoostingClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.
cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for GradientBoostingClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for GradientBoostingClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, early_stopping_rounds=2, save_model=True, save_dir=None)
Implementation on the training stage of GradientBoostingClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.
epochs (int, default=100) – The number of training epochs per base estimator.
use_reduction_sum (bool, default=True) – Whether to set
reduction="sum"
for the internal mean squared error used to fit each base estimator.log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A data loader that contains the evaluating data.
If
None
, no validation is conducted after each base estimator being trained.If not
None
, the ensemble will be evaluated on this dataloader after each base estimator being trained.
early_stopping_rounds (int, default=2) – Specify the number of tolerant rounds for early stopping. When the validation performance of the ensemble does not improve after adding the base estimator fitted in current iteration, the internal counter on early stopping will increase by one. When the value of the internal counter reaches
early_stopping_rounds
, the training stage will terminate instantly.save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble containingn_estimators
base estimators will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- forward(*x)
Implementation on the data forwarding in GradientBoostingClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
GradientBoostingRegressor
- class torchensemble.gradient_boosting.GradientBoostingRegressor(*args: Any, **kwargs: Any)
Bases:
_BaseGradientBoosting
,BaseRegressor
Implementation on the GradientBoostingRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.
cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for GradientBoostingRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for GradientBoostingRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, early_stopping_rounds=2, save_model=True, save_dir=None)
Implementation on the training stage of GradientBoostingRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.
epochs (int, default=100) – The number of training epochs per base estimator.
use_reduction_sum (bool, default=True) – Whether to set
reduction="sum"
for the internal mean squared error used to fit each base estimator.log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A data loader that contains the evaluating data.
If
None
, no validation is conducted after each base estimator being trained.If not
None
, the ensemble will be evaluated on this dataloader after each base estimator being trained.
early_stopping_rounds (int, default=2) – Specify the number of tolerant rounds for early stopping. When the validation performance of the ensemble does not improve after adding the base estimator fitted in current iteration, the internal counter on early stopping will increase by one. When the value of the internal counter reaches
early_stopping_rounds
, the training stage will terminate instantly.save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble containingn_estimators
base estimators will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- forward(*x)
Implementation on the data forwarding in GradientBoostingRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Neural Tree Ensemble
Neural tree ensemble are extensions of voting and gradient boosting, which uses the neural tree as the base estimator. Neural trees are differentiable trees that uses the logistic regression in internal nodes to split samples into child nodes with different probabilities. Model details are available at Distilling a neural network into a soft decision tree.
NeuralForestClassifier
- class torchensemble.voting.NeuralForestClassifier(*args: Any, **kwargs: Any)
Bases:
BaseTreeEnsemble
,VotingClassifier
Implementation on the NeuralForestClassifier.
- Parameters:
n_estimators (int) – The number of neural trees in the ensemble.
depth (int, default=5) – The depth of neural tree. A tree with depth
d
is with \(2^d\) leaf nodes and \(2^d-1\) internal nodes.lamda (float, default=1e-3) –
The coefficient of the regularization term when training neural trees, proposed in the paper: Distilling a neural network into a soft decision tree.
cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in NeuralForestClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for NeuralForestClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for NeuralForestClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for NeuralForestClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of NeuralForestClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
NeuralForestRegressor
- class torchensemble.voting.NeuralForestRegressor(*args: Any, **kwargs: Any)
Bases:
BaseTreeEnsemble
,VotingRegressor
Implementation on the NeuralForestRegressor.
- Parameters:
n_estimators (int) – The number of neural trees in the ensemble.
depth (int, default=5) – The depth of neural tree. A tree with depth
d
is with \(2^d\) leaf nodes and \(2^d-1\) internal nodes.lamda (float, default=1e-3) –
The coefficient of the regularization term when training neural trees, proposed in the paper: Distilling a neural network into a soft decision tree.
cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in NeuralForestRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for NeuralForestRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for NeuralForestRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for NeuralForestRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of NeuralForestRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted during the training stage.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
Snapshot Ensemble
Snapshot ensemble generates many base estimators by enforcing a base estimator to converge to its local minima many times and save the model parameters at that point as a snapshot. The final prediction takes the average over predictions from all snapshot models.
- Reference:
G. Huang, Y.-X. Li, G. Pleiss et al., Snapshot Ensemble: Train 1, and M for free, ICLR, 2017.
SnapshotEnsembleClassifier
- class torchensemble.snapshot_ensemble.SnapshotEnsembleClassifier(*args: Any, **kwargs: Any)
Bases:
_BaseSnapshotEnsemble
,BaseClassifier
Implementation on the SnapshotEnsembleClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in SnapshotEnsembleClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for SnapshotEnsembleClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_criterion(criterion)
Set the training criterion for SnapshotEnsembleClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, lr_clip=None, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of SnapshotEnsembleClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
DataLoader
container that contains the training data.lr_clip (list or tuple, default=None) –
Specify the accepted range of learning rate. When the learning rate determined by the scheduler is out of this range, it will be clipped.
The first element should be the lower bound of learning rate.
The second element should be the upper bound of learning rate.
epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
DataLoader
container that contains the evaluating data.If
None
, no validation is conducted after each snapshot being generated.If not
None
, the ensemble will be evaluated on this dataloader after each snapshot being generated.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble withn_estimators
base estimators will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
SnapshotEnsembleRegressor
- class torchensemble.snapshot_ensemble.SnapshotEnsembleRegressor(*args: Any, **kwargs: Any)
Bases:
_BaseSnapshotEnsemble
,BaseRegressor
Implementation on the SnapshotEnsembleRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in SnapshotEnsembleRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for SnapshotEnsembleRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_criterion(criterion)
Set the training criterion for SnapshotEnsembleRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, lr_clip=None, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of SnapshotEnsembleRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
DataLoader
container that contains the training data.lr_clip (list or tuple, default=None) –
Specify the accepted range of learning rate. When the learning rate determined by the scheduler is out of this range, it will be clipped.
The first element should be the lower bound of learning rate.
The second element should be the upper bound of learning rate.
epochs (int, default=100) – The number of training epochs.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
DataLoader
container that contains the evaluating data.If
None
, no validation is conducted after each snapshot being generated.If not
None
, the ensemble will be evaluated on this dataloader after each snapshot being generated.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble withn_estimators
base estimators will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Adversarial Training
Adversarial training is able to improve the performance of an ensemble by treating adversarial samples as the augmented training data. The fast gradient sign method (FGSM) is used to generate adversarial samples.
- Reference:
B. Lakshminarayanan, A. Pritzel, C. Blundell., Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, NIPS 2017.
Warning
When your base estimator is under-fit on the dataset, it is not recommended to use the AdversarialTrainingClassifier
or AdversarialTrainingRegressor
, because they may deteriorate the performance further.
AdversarialTrainingClassifier
- class torchensemble.adversarial_training.AdversarialTrainingClassifier(*args: Any, **kwargs: Any)
Bases:
_BaseAdversarialTraining
,BaseClassifier
Implementation on the AdversarialTrainingClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in AdversarialTrainingClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for AdversarialTrainingClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for AdversarialTrainingClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for AdversarialTrainingClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, epsilon=0.5, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of AdversarialTrainingClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
epsilon (float, default=0.01) – The step used to generate adversarial samples in the fast gradient sign method (FGSM), which should be in the range [0, 1].
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted after each training epoch.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
AdversarialTrainingRegressor
- class torchensemble.adversarial_training.AdversarialTrainingRegressor(*args: Any, **kwargs: Any)
Bases:
_BaseAdversarialTraining
,BaseRegressor
Implementation on the AdversarialTrainingRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in AdversarialTrainingRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for AdversarialTrainingRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for AdversarialTrainingRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for AdversarialTrainingRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, epsilon=0.5, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of AdversarialTrainingRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
torch.utils.data.DataLoader
container that contains the training data.epochs (int, default=100) – The number of training epochs.
epsilon (float, default=0.01) – The step used to generate adversarial samples in the fast gradient sign method (FGSM), which should be in the range [0, 1].
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
torch.utils.data.DataLoader
container that contains the evaluating data.If
None
, no validation is conducted after each training epoch.If not
None
, the ensemble will be evaluated on this dataloader after each training epoch.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Fast Geometric Ensemble
Motivated by geometric insights on the loss surface of deep neural networks, Fast Geometric Ensembling (FGE) is an efficient ensemble that uses a customized learning rate scheduler to generate base estimators, similar to snapshot ensemble.
- Reference:
T. Garipov, P. Izmailov, D. Podoprikhin et al., Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, NeurIPS, 2018.
FastGeometricClassifier
- class torchensemble.fast_geometric.FastGeometricClassifier(*args: Any, **kwargs: Any)
Bases:
_BaseFastGeometric
,BaseClassifier
Implementation on the FastGeometricClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in FastGeometricClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for FastGeometricClassifier. Notice that keyword arguments specified here will also be used in the ensembling stage except the learning rate..
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for FastGeometricClassifier. Notice that this scheduler will only be used in the stage on fitting the dummy base estimator.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for FastGeometricClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, cycle=4, lr_1=0.05, lr_2=0.0001, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of FastGeometricClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
DataLoader
container that contains the training data.cycle (int, default=4) – The number of cycles used to build each base estimator in the ensemble.
lr_1 (float, default=5e-2) –
alpha_1
in original paper used to adjust the learning rate, also serves as the initial learning rate of the internal optimizer.lr_2 (float, default=1e-4) –
alpha_2
in original paper used to adjust the learning rate, also serves as the smallest learning rate of the internal optimizer.epochs (int, default=100) – The number of training epochs used to fit the dummy base estimator.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
DataLoader
container that contains the evaluating data.If
None
, no validation is conducted after each real base estimator being generated.If not
None
, the ensemble will be evaluated on this dataloader after each base estimator being generated.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
FastGeometricRegressor
- class torchensemble.fast_geometric.FastGeometricRegressor(*args: Any, **kwargs: Any)
Bases:
_BaseFastGeometric
,BaseRegressor
Implementation on the FastGeometricRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- forward(*x)
Implementation on the data forwarding in FastGeometricRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for FastGeometricRegressor. Notice that keyword arguments specified here will also be used in the ensembling stage except the learning rate.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for FastGeometricRegressor. Notice that this scheduler will only be used in the stage on fitting the dummy base estimator.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for FastGeometricRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, cycle=4, lr_1=0.05, lr_2=0.0001, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of FastGeometricRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A
DataLoader
container that contains the training data.cycle (int, default=4) – The number of cycles used to build each base estimator in the ensemble.
lr_1 (float, default=5e-2) –
alpha_1
in original paper used to adjust the learning rate, also serves as the initial learning rate of the internal optimizer.lr_2 (float, default=1e-4) –
alpha_2
in original paper used to adjust the learning rate, also serves as the smallest learning rate of the internal optimizer.epochs (int, default=100) – The number of training epochs used to fit the dummy base estimator.
log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A
DataLoader
container that contains the evaluating data.If
None
, no validation is conducted after each real base estimator being generated.If not
None
, the ensemble will be evaluated on this dataloader after each base estimator being generated.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble fully trained will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
Soft Gradient Boosting
In soft gradient boosting, all base estimators could be simultaneously fitted, while achieving the similar boosting improvements as in gradient boosting.
- Reference:
Feng, Y.-X. Xu et al., Soft Gradient Boosting Machine, ArXiv, 2020.
SoftGradientBoostingClassifier
- class torchensemble.soft_gradient_boosting.SoftGradientBoostingClassifier(*args: Any, **kwargs: Any)
Bases:
_BaseSoftGradientBoosting
,BaseClassifier
Implementation on the SoftGradientBoostingClassifier.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.
cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for SoftGradientBoostingClassifier.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for SoftGradientBoostingClassifier.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for SoftGradientBoostingClassifier.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of SoftGradientBoostingClassifier.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.
epochs (int, default=100) – The number of training epochs per base estimator.
use_reduction_sum (bool, default=True) – Whether to set
reduction="sum"
for the internal mean squared error used to fit each base estimator.log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A data loader that contains the evaluating data.
If
None
, no validation is conducted after each base estimator being trained.If not
None
, the ensemble will be evaluated on this dataloader after each base estimator being trained.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble containingn_estimators
base estimators will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- forward(*x)
Implementation on the data forwarding in SoftGradientBoostingClassifier.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
proba – The predicted class distribution.
- Return type:
tensor of shape (batch_size, n_classes)
- evaluate(test_loader, return_loss=False)
Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the
test_loader
.
- Returns:
accuracy (float) – The classification accuracy of the fitted ensemble on
test_loader
.loss (float) – The average cross-entropy loss of the fitted ensemble on
test_loader
, only available whenreturn_loss
is True.
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)
SoftGradientBoostingRegressor
- class torchensemble.soft_gradient_boosting.SoftGradientBoostingRegressor(*args: Any, **kwargs: Any)
Bases:
_BaseSoftGradientBoosting
,BaseRegressor
Implementation on the SoftGradientBoostingRegressor.
- Parameters:
estimator (torch.nn.Module) –
The class or object of your base estimator.
If
class
, it should inherit fromtorch.nn.Module
.If
object
, it should be instantiated from a class inherited fromtorch.nn.Module
.
n_estimators (int) – The number of base estimators in the ensemble.
estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if
estimator
is a base estimator object after instantiation.shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.
cuda (bool, default=True) –
If
True
, use GPU to train and evaluate the ensemble.If
False
, use CPU to train and evaluate the ensemble.
n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as
voting
andbagging
. Setting it to an integer larger than1
enablesn_jobs
base estimators to be trained simultaneously.
- estimators_
An internal container that stores all fitted base estimators.
- Type:
torch.nn.ModuleList
- set_optimizer(optimizer_name, **kwargs)
Set the attributes on optimizer for SoftGradientBoostingRegressor.
- Parameters:
optimizer_name (string) – The name of the optimizer, should be one of {
Adadelta
,Adagrad
,Adam
,AdamW
,Adamax
,ASGD
,RMSprop
,Rprop
,SGD
}.**kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form:
lr=1e-3, weight_decay=5e-4, ...
. These keyword arguments will be directly passed totorch.optim.Optimizer
.
- set_scheduler(scheduler_name, **kwargs)
Set the attributes on scheduler for SoftGradientBoostingRegressor.
- Parameters:
scheduler_name (string) – The name of the scheduler, should be one of {
LambdaLR
,MultiplicativeLR
,StepLR
,MultiStepLR
,ExponentialLR
,CosineAnnealingLR
,ReduceLROnPlateau
}.**kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to
torch.optim.lr_scheduler
.
- set_criterion(criterion)
Set the training criterion for SoftGradientBoostingRegressor.
- Parameters:
criterion (torch.nn.loss) – The customized training criterion object.
- fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, save_model=True, save_dir=None)
Implementation on the training stage of SoftGradientBoostingRegressor.
- Parameters:
train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.
epochs (int, default=100) – The number of training epochs per base estimator.
use_reduction_sum (bool, default=True) – Whether to set
reduction="sum"
for the internal mean squared error used to fit each base estimator.log_interval (int, default=100) – The number of batches to wait before logging the training status.
test_loader (torch.utils.data.DataLoader, default=None) –
A data loader that contains the evaluating data.
If
None
, no validation is conducted after each base estimator being trained.If not
None
, the ensemble will be evaluated on this dataloader after each base estimator being trained.
save_model (bool, default=True) –
Specify whether to save the model parameters.
If test_loader is
None
, the ensemble containingn_estimators
base estimators will be saved.If test_loader is not
None
, the ensemble with the best validation performance will be saved.
save_dir (string, default=None) –
Specify where to save the model parameters.
If
None
, the model will be saved in the current directory.If not
None
, the model will be saved in the specified directory:save_dir
.
- forward(*x)
Implementation on the data forwarding in SoftGradientBoostingRegressor.
- Parameters:
X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.
- Returns:
pred – The predicted values.
- Return type:
tensor of shape (batch_size, n_outputs)
- evaluate(test_loader)
Compute the mean squared error (MSE) of the ensemble given the testing dataloader.
- Parameters:
test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.
- Returns:
mse – The testing mean squared error of the fitted ensemble on
test_loader
.- Return type:
float
- predict(*x)
Return the predictions of the ensemble given the testing data.
- Parameters:
X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.
- Returns:
pred – For classifiers,
n_outputs
is the number of distinct classes. For regressors,n_output
is the number of target variables.- Return type:
tensor of shape (n_samples, n_outputs)