Parameters

This page provides the API reference of torchensemble. Below is a list of functions supported by all ensembles.

  • fit(): Training stage of the ensemble

  • evaluate(): Evaluating stage of the ensemble

  • predict(): Return the predictions of the ensemble

  • forward(): Data forward process of the ensemble

  • set_optimizer(): Set the parameter optimizer for training the ensemble

  • set_scheduler(): Set the learning rate scheduler for training the ensemble

Fusion

In fusion-based ensemble methods, the predictions from all base estimators are first aggregated as an average output. After then, the training loss is computed based on this average output and the ground-truth. The training loss is then back-propagated to all base estimators simultaneously.

FusionClassifier

class torchensemble.fusion.FusionClassifier(*args: Any, **kwargs: Any)

Bases: BaseClassifier

Implementation on the FusionClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in FusionClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for FusionClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for FusionClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for FusionClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of FusionClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

FusionRegressor

class torchensemble.fusion.FusionRegressor(*args: Any, **kwargs: Any)

Bases: BaseRegressor

Implementation on the FusionRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in FusionRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for FusionRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for FusionRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for FusionRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of FusionRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Voting

In voting-based ensemble methods, each base estimator is trained independently, and the final prediction takes the average over predictions from all base estimators.

VotingClassifier

class torchensemble.voting.VotingClassifier(*args: Any, **kwargs: Any)

Bases: BaseClassifier

Implementation on the VotingClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in VotingClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for VotingClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for VotingClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for VotingClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of VotingClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

VotingRegressor

class torchensemble.voting.VotingRegressor(*args: Any, **kwargs: Any)

Bases: BaseRegressor

Implementation on the VotingRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in VotingRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for VotingRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for VotingRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for VotingRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of VotingRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Bagging

In bagging-based ensemble methods, each base estimator is trained independently. In addition, sampling with replacement is conducted on the training data to further encourage the diversity between different base estimators in the ensemble.

BaggingClassifier

class torchensemble.bagging.BaggingClassifier(*args: Any, **kwargs: Any)

Bases: BaseClassifier

Implementation on the BaggingClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in BaggingClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for BaggingClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for BaggingClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for BaggingClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of BaggingClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

BaggingRegressor

class torchensemble.bagging.BaggingRegressor(*args: Any, **kwargs: Any)

Bases: BaseRegressor

Implementation on the BaggingRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in BaggingRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for BaggingRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for BaggingRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for BaggingRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of BaggingRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Gradient Boosting

Gradient boosting is a classic sequential ensemble method. At each iteration, the learning target of a new base estimator is to fit the pseudo residual computed based on the ground truth and the output from base estimators fitted before, using ordinary least square.

Tip

The input argument shrinkage_rate in gradient_boosting is also known as learning rate in other gradient boosting libraries such as XGBoost. However, its meaning is totally different from the meaning of learning rate in the context of parameter optimizer in deep learning.

GradientBoostingClassifier

class torchensemble.gradient_boosting.GradientBoostingClassifier(*args: Any, **kwargs: Any)

Bases: _BaseGradientBoosting, BaseClassifier

Implementation on the GradientBoostingClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for GradientBoostingClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for GradientBoostingClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, early_stopping_rounds=2, save_model=True, save_dir=None)

Implementation on the training stage of GradientBoostingClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.

  • epochs (int, default=100) – The number of training epochs per base estimator.

  • use_reduction_sum (bool, default=True) – Whether to set reduction="sum" for the internal mean squared error used to fit each base estimator.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A data loader that contains the evaluating data.

    • If None, no validation is conducted after each base estimator being trained.

    • If not None, the ensemble will be evaluated on this dataloader after each base estimator being trained.

  • early_stopping_rounds (int, default=2) – Specify the number of tolerant rounds for early stopping. When the validation performance of the ensemble does not improve after adding the base estimator fitted in current iteration, the internal counter on early stopping will increase by one. When the value of the internal counter reaches early_stopping_rounds, the training stage will terminate instantly.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble containing n_estimators base estimators will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

forward(*x)

Implementation on the data forwarding in GradientBoostingClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

GradientBoostingRegressor

class torchensemble.gradient_boosting.GradientBoostingRegressor(*args: Any, **kwargs: Any)

Bases: _BaseGradientBoosting, BaseRegressor

Implementation on the GradientBoostingRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for GradientBoostingRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for GradientBoostingRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, early_stopping_rounds=2, save_model=True, save_dir=None)

Implementation on the training stage of GradientBoostingRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.

  • epochs (int, default=100) – The number of training epochs per base estimator.

  • use_reduction_sum (bool, default=True) – Whether to set reduction="sum" for the internal mean squared error used to fit each base estimator.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A data loader that contains the evaluating data.

    • If None, no validation is conducted after each base estimator being trained.

    • If not None, the ensemble will be evaluated on this dataloader after each base estimator being trained.

  • early_stopping_rounds (int, default=2) – Specify the number of tolerant rounds for early stopping. When the validation performance of the ensemble does not improve after adding the base estimator fitted in current iteration, the internal counter on early stopping will increase by one. When the value of the internal counter reaches early_stopping_rounds, the training stage will terminate instantly.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble containing n_estimators base estimators will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

forward(*x)

Implementation on the data forwarding in GradientBoostingRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Neural Tree Ensemble

Neural tree ensemble are extensions of voting and gradient boosting, which uses the neural tree as the base estimator. Neural trees are differentiable trees that uses the logistic regression in internal nodes to split samples into child nodes with different probabilities. Model details are available at Distilling a neural network into a soft decision tree.

NeuralForestClassifier

class torchensemble.voting.NeuralForestClassifier(*args: Any, **kwargs: Any)

Bases: BaseTreeEnsemble, VotingClassifier

Implementation on the NeuralForestClassifier.

Parameters:

  • n_estimators (int) – The number of neural trees in the ensemble.

  • depth (int, default=5) – The depth of neural tree. A tree with depth d is with \(2^d\) leaf nodes and \(2^d-1\) internal nodes.

  • lamda (float, default=1e-3) –

    The coefficient of the regularization term when training neural trees, proposed in the paper: Distilling a neural network into a soft decision tree.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in NeuralForestClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for NeuralForestClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for NeuralForestClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for NeuralForestClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of NeuralForestClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

NeuralForestRegressor

class torchensemble.voting.NeuralForestRegressor(*args: Any, **kwargs: Any)

Bases: BaseTreeEnsemble, VotingRegressor

Implementation on the NeuralForestRegressor.

Parameters:

  • n_estimators (int) – The number of neural trees in the ensemble.

  • depth (int, default=5) – The depth of neural tree. A tree with depth d is with \(2^d\) leaf nodes and \(2^d-1\) internal nodes.

  • lamda (float, default=1e-3) –

    The coefficient of the regularization term when training neural trees, proposed in the paper: Distilling a neural network into a soft decision tree.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in NeuralForestRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for NeuralForestRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for NeuralForestRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for NeuralForestRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of NeuralForestRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted during the training stage.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

Snapshot Ensemble

Snapshot ensemble generates many base estimators by enforcing a base estimator to converge to its local minima many times and save the model parameters at that point as a snapshot. The final prediction takes the average over predictions from all snapshot models.

Reference:

G. Huang, Y.-X. Li, G. Pleiss et al., Snapshot Ensemble: Train 1, and M for free, ICLR, 2017.

SnapshotEnsembleClassifier

class torchensemble.snapshot_ensemble.SnapshotEnsembleClassifier(*args: Any, **kwargs: Any)

Bases: _BaseSnapshotEnsemble, BaseClassifier

Implementation on the SnapshotEnsembleClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in SnapshotEnsembleClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for SnapshotEnsembleClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_criterion(criterion)

Set the training criterion for SnapshotEnsembleClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, lr_clip=None, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of SnapshotEnsembleClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A DataLoader container that contains the training data.

  • lr_clip (list or tuple, default=None) –

    Specify the accepted range of learning rate. When the learning rate determined by the scheduler is out of this range, it will be clipped.

    • The first element should be the lower bound of learning rate.

    • The second element should be the upper bound of learning rate.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A DataLoader container that contains the evaluating data.

    • If None, no validation is conducted after each snapshot being generated.

    • If not None, the ensemble will be evaluated on this dataloader after each snapshot being generated.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble with n_estimators base estimators will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

SnapshotEnsembleRegressor

class torchensemble.snapshot_ensemble.SnapshotEnsembleRegressor(*args: Any, **kwargs: Any)

Bases: _BaseSnapshotEnsemble, BaseRegressor

Implementation on the SnapshotEnsembleRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in SnapshotEnsembleRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for SnapshotEnsembleRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_criterion(criterion)

Set the training criterion for SnapshotEnsembleRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, lr_clip=None, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of SnapshotEnsembleRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A DataLoader container that contains the training data.

  • lr_clip (list or tuple, default=None) –

    Specify the accepted range of learning rate. When the learning rate determined by the scheduler is out of this range, it will be clipped.

    • The first element should be the lower bound of learning rate.

    • The second element should be the upper bound of learning rate.

  • epochs (int, default=100) – The number of training epochs.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A DataLoader container that contains the evaluating data.

    • If None, no validation is conducted after each snapshot being generated.

    • If not None, the ensemble will be evaluated on this dataloader after each snapshot being generated.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble with n_estimators base estimators will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Adversarial Training

Adversarial training is able to improve the performance of an ensemble by treating adversarial samples as the augmented training data. The fast gradient sign method (FGSM) is used to generate adversarial samples.

Reference:

B. Lakshminarayanan, A. Pritzel, C. Blundell., Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, NIPS 2017.

Warning

When your base estimator is under-fit on the dataset, it is not recommended to use the AdversarialTrainingClassifier or AdversarialTrainingRegressor, because they may deteriorate the performance further.

AdversarialTrainingClassifier

class torchensemble.adversarial_training.AdversarialTrainingClassifier(*args: Any, **kwargs: Any)

Bases: _BaseAdversarialTraining, BaseClassifier

Implementation on the AdversarialTrainingClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in AdversarialTrainingClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for AdversarialTrainingClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for AdversarialTrainingClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for AdversarialTrainingClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, epsilon=0.5, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of AdversarialTrainingClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • epsilon (float, default=0.01) – The step used to generate adversarial samples in the fast gradient sign method (FGSM), which should be in the range [0, 1].

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted after each training epoch.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

AdversarialTrainingRegressor

class torchensemble.adversarial_training.AdversarialTrainingRegressor(*args: Any, **kwargs: Any)

Bases: _BaseAdversarialTraining, BaseRegressor

Implementation on the AdversarialTrainingRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in AdversarialTrainingRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for AdversarialTrainingRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for AdversarialTrainingRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for AdversarialTrainingRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, epsilon=0.5, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of AdversarialTrainingRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A torch.utils.data.DataLoader container that contains the training data.

  • epochs (int, default=100) – The number of training epochs.

  • epsilon (float, default=0.01) – The step used to generate adversarial samples in the fast gradient sign method (FGSM), which should be in the range [0, 1].

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A torch.utils.data.DataLoader container that contains the evaluating data.

    • If None, no validation is conducted after each training epoch.

    • If not None, the ensemble will be evaluated on this dataloader after each training epoch.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Fast Geometric Ensemble

Motivated by geometric insights on the loss surface of deep neural networks, Fast Geometric Ensembling (FGE) is an efficient ensemble that uses a customized learning rate scheduler to generate base estimators, similar to snapshot ensemble.

Reference:

T. Garipov, P. Izmailov, D. Podoprikhin et al., Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, NeurIPS, 2018.

FastGeometricClassifier

class torchensemble.fast_geometric.FastGeometricClassifier(*args: Any, **kwargs: Any)

Bases: _BaseFastGeometric, BaseClassifier

Implementation on the FastGeometricClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in FastGeometricClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for FastGeometricClassifier. Notice that keyword arguments specified here will also be used in the ensembling stage except the learning rate..

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for FastGeometricClassifier. Notice that this scheduler will only be used in the stage on fitting the dummy base estimator.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for FastGeometricClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, cycle=4, lr_1=0.05, lr_2=0.0001, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of FastGeometricClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A DataLoader container that contains the training data.

  • cycle (int, default=4) – The number of cycles used to build each base estimator in the ensemble.

  • lr_1 (float, default=5e-2) – alpha_1 in original paper used to adjust the learning rate, also serves as the initial learning rate of the internal optimizer.

  • lr_2 (float, default=1e-4) – alpha_2 in original paper used to adjust the learning rate, also serves as the smallest learning rate of the internal optimizer.

  • epochs (int, default=100) – The number of training epochs used to fit the dummy base estimator.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A DataLoader container that contains the evaluating data.

    • If None, no validation is conducted after each real base estimator being generated.

    • If not None, the ensemble will be evaluated on this dataloader after each base estimator being generated.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

FastGeometricRegressor

class torchensemble.fast_geometric.FastGeometricRegressor(*args: Any, **kwargs: Any)

Bases: _BaseFastGeometric, BaseRegressor

Implementation on the FastGeometricRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

forward(*x)

Implementation on the data forwarding in FastGeometricRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for FastGeometricRegressor. Notice that keyword arguments specified here will also be used in the ensembling stage except the learning rate.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for FastGeometricRegressor. Notice that this scheduler will only be used in the stage on fitting the dummy base estimator.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for FastGeometricRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, cycle=4, lr_1=0.05, lr_2=0.0001, epochs=100, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of FastGeometricRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A DataLoader container that contains the training data.

  • cycle (int, default=4) – The number of cycles used to build each base estimator in the ensemble.

  • lr_1 (float, default=5e-2) – alpha_1 in original paper used to adjust the learning rate, also serves as the initial learning rate of the internal optimizer.

  • lr_2 (float, default=1e-4) – alpha_2 in original paper used to adjust the learning rate, also serves as the smallest learning rate of the internal optimizer.

  • epochs (int, default=100) – The number of training epochs used to fit the dummy base estimator.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A DataLoader container that contains the evaluating data.

    • If None, no validation is conducted after each real base estimator being generated.

    • If not None, the ensemble will be evaluated on this dataloader after each base estimator being generated.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble fully trained will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

Soft Gradient Boosting

In soft gradient boosting, all base estimators could be simultaneously fitted, while achieving the similar boosting improvements as in gradient boosting.

Reference:

  1. Feng, Y.-X. Xu et al., Soft Gradient Boosting Machine, ArXiv, 2020.

SoftGradientBoostingClassifier

class torchensemble.soft_gradient_boosting.SoftGradientBoostingClassifier(*args: Any, **kwargs: Any)

Bases: _BaseSoftGradientBoosting, BaseClassifier

Implementation on the SoftGradientBoostingClassifier.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for SoftGradientBoostingClassifier.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for SoftGradientBoostingClassifier.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for SoftGradientBoostingClassifier.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of SoftGradientBoostingClassifier.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.

  • epochs (int, default=100) – The number of training epochs per base estimator.

  • use_reduction_sum (bool, default=True) – Whether to set reduction="sum" for the internal mean squared error used to fit each base estimator.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A data loader that contains the evaluating data.

    • If None, no validation is conducted after each base estimator being trained.

    • If not None, the ensemble will be evaluated on this dataloader after each base estimator being trained.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble containing n_estimators base estimators will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

forward(*x)

Implementation on the data forwarding in SoftGradientBoostingClassifier.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

proba – The predicted class distribution.

Return type:

tensor of shape (batch_size, n_classes)

evaluate(test_loader, return_loss=False)

Compute the classification accuracy of the ensemble given the testing dataloader and optionally the average cross-entropy loss.

Parameters:

  • test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

  • return_loss (bool, default=False) – Whether to return the average cross-entropy loss over all batches in the test_loader.

Returns:

  • accuracy (float) – The classification accuracy of the fitted ensemble on test_loader.

  • loss (float) – The average cross-entropy loss of the fitted ensemble on test_loader, only available when return_loss is True.

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)

SoftGradientBoostingRegressor

class torchensemble.soft_gradient_boosting.SoftGradientBoostingRegressor(*args: Any, **kwargs: Any)

Bases: _BaseSoftGradientBoosting, BaseRegressor

Implementation on the SoftGradientBoostingRegressor.

Parameters:

  • estimator (torch.nn.Module) –

    The class or object of your base estimator.

    • If class, it should inherit from torch.nn.Module.

    • If object, it should be instantiated from a class inherited from torch.nn.Module.

  • n_estimators (int) – The number of base estimators in the ensemble.

  • estimator_args (dict, default=None) – The dictionary of hyper-parameters used to instantiate base estimators. This parameter will have no effect if estimator is a base estimator object after instantiation.

  • shrinkage_rate (float, default=1) – The shrinkage rate used in gradient boosting.

  • cuda (bool, default=True) –

    • If True, use GPU to train and evaluate the ensemble.

    • If False, use CPU to train and evaluate the ensemble.

  • n_jobs (int, default=None) – The number of workers for training the ensemble. This input argument is used for parallel ensemble methods such as voting and bagging. Setting it to an integer larger than 1 enables n_jobs base estimators to be trained simultaneously.

estimators_

An internal container that stores all fitted base estimators.

Type:

torch.nn.ModuleList

set_optimizer(optimizer_name, **kwargs)

Set the attributes on optimizer for SoftGradientBoostingRegressor.

Parameters:

  • optimizer_name (string) – The name of the optimizer, should be one of {Adadelta, Adagrad, Adam, AdamW, Adamax, ASGD, RMSprop, Rprop, SGD}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the optimizer, should be in the form: lr=1e-3, weight_decay=5e-4, .... These keyword arguments will be directly passed to torch.optim.Optimizer.

set_scheduler(scheduler_name, **kwargs)

Set the attributes on scheduler for SoftGradientBoostingRegressor.

Parameters:

  • scheduler_name (string) – The name of the scheduler, should be one of {LambdaLR, MultiplicativeLR, StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, ReduceLROnPlateau}.

  • **kwargs (keyword arguments) – Keyword arguments on setting the scheduler. These keyword arguments will be directly passed to torch.optim.lr_scheduler.

set_criterion(criterion)

Set the training criterion for SoftGradientBoostingRegressor.

Parameters:

criterion (torch.nn.loss) – The customized training criterion object.

fit(train_loader, epochs=100, use_reduction_sum=True, log_interval=100, test_loader=None, save_model=True, save_dir=None)

Implementation on the training stage of SoftGradientBoostingRegressor.

Parameters:

  • train_loader (torch.utils.data.DataLoader) – A data loader that contains the training data.

  • epochs (int, default=100) – The number of training epochs per base estimator.

  • use_reduction_sum (bool, default=True) – Whether to set reduction="sum" for the internal mean squared error used to fit each base estimator.

  • log_interval (int, default=100) – The number of batches to wait before logging the training status.

  • test_loader (torch.utils.data.DataLoader, default=None) –

    A data loader that contains the evaluating data.

    • If None, no validation is conducted after each base estimator being trained.

    • If not None, the ensemble will be evaluated on this dataloader after each base estimator being trained.

  • save_model (bool, default=True) –

    Specify whether to save the model parameters.

    • If test_loader is None, the ensemble containing n_estimators base estimators will be saved.

    • If test_loader is not None, the ensemble with the best validation performance will be saved.

  • save_dir (string, default=None) –

    Specify where to save the model parameters.

    • If None, the model will be saved in the current directory.

    • If not None, the model will be saved in the specified directory: save_dir.

forward(*x)

Implementation on the data forwarding in SoftGradientBoostingRegressor.

Parameters:

X (tensor) – An input batch of data, which should be a valid input data batch for base estimators in the ensemble.

Returns:

pred – The predicted values.

Return type:

tensor of shape (batch_size, n_outputs)

evaluate(test_loader)

Compute the mean squared error (MSE) of the ensemble given the testing dataloader.

Parameters:

test_loader (torch.utils.data.DataLoader) – A data loader that contains the testing data.

Returns:

mse – The testing mean squared error of the fitted ensemble on test_loader.

Return type:

float

predict(*x)

Return the predictions of the ensemble given the testing data.

Parameters:

X ({tensor, numpy array}) – A data batch in the form of tensor or numpy array.

Returns:

pred – For classifiers, n_outputs is the number of distinct classes. For regressors, n_output is the number of target variables.

Return type:

tensor of shape (n_samples, n_outputs)