Callbacks

Callbacks are the primary mechanism by which one can embed event hooks into the training process. Many useful callbacks are provided out of the box but in all likelihood you will want to implement your own to execute actions based on training events. To do so, simply extend the pywick.callbacks.Callback class and overwrite functions that you are interested in acting upon.

CSVLogger

class pywick.callbacks.CSVLogger.CSVLogger(file, separator=', ', append=False, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Logs epoch-level metrics to a CSV file

Parameters:
  • file – (string) path to csv file
  • separator – (string) delimiter for file
  • append – (bool) whether to append result to existing file or make new file

Callback

class pywick.callbacks.Callback.Callback(params: dict = None, trainer=None, **_)[source]

Bases: object

Abstract base class used to build new callbacks. Extend this class to build your own callbacks and overwrite functions that you want to monitor. Functions will be called automatically from the trainer once per relevant training event (e.g. at the beginning of epoch, end of epoch, beginning of batch, end of batch etc.)

is_enabled()[source]

Returns the status of this callback. Default = True but can be turned off with param[‘is_enabled’] = False :return:

on_batch_begin(batch: int, logs: Collection = None)[source]

Called at the beginning of a new batch :param batch: batch number :param logs: collection of logs to process / parse / add to :return:

on_batch_end(batch: int, logs: Collection = None)[source]

Called at the end of an epoch :param batch: batch number :param logs: collection of logs to process / parse / add to :return:

on_epoch_begin(epoch: int, logs: Collection = None)[source]

Called at the beginning of a new epoch :param epoch: epoch number :param logs: collection of logs to process / parse / add to :return:

on_epoch_end(epoch: int, logs: Collection = None)[source]

Called at the end of an epoch :param epoch: epoch number :param logs: collection of logs to process / parse / add to :return:

on_train_begin(logs: Collection = None)[source]

Called at the beginning of a new training run :param logs: collection of logs to process / parse / add to :return:

on_train_end(logs: Collection = None)[source]

Called at the end of a training run :param logs: collection of logs to process / parse / add to :return:

CyclicLRScheduler

class pywick.callbacks.CyclicLRScheduler.CyclicLRScheduler(optimizer, base_lr=0.001, max_lr=0.006, step_size=2000, mode='triangular', gamma=1.0, scale_fn=None, scale_mode='cycle', verbose=True, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks. The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis.

Cyclical learning rate policy changes the learning rate after every batch. batch_step should be called after a batch has been used for training. To resume training, save last_batch_iteration and use it to instantiate CycleLR.

This class has three built-in policies, as put forth in the paper: “triangular”:

A basic triangular cycle w/ no amplitude scaling.
“triangular2”:
A basic triangular cycle that scales initial amplitude by half each cycle.
“exp_range”:
A cycle that scales initial amplitude by gamma**(cycle iterations) at each cycle iteration.

This implementation was adapted from the github repo: bckenstler/CLR

Parameters:
  • optimizer – (Optimizer): Wrapped optimizer.
  • base_lr – (float or list): Initial learning rate which is the lower boundary in the cycle for eachparam groups. Default: 0.001
  • max_lr – (float or list): Upper boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_lr - base_lr). The lr at any cycle is the sum of base_lr and some scaling of the amplitude; therefore max_lr may not actually be reached depending on scaling function. Default: 0.006
  • step_size – (int): Number of training iterations per half cycle. Authors suggest setting step_size 2-8 x training iterations in epoch. Default: 2000
  • mode – (str): One of {triangular, triangular2, exp_range}. Values correspond to policies detailed above. If scale_fn is not None, this argument is ignored. Default: ‘triangular’
  • gamma – (float): Constant in ‘exp_range’ scaling function: gamma**(cycle iterations) Default: 1.0
  • scale_fn – (function): Custom scaling policy defined by a single argument lambda function, where 0 <= scale_fn(x) <= 1 for all x >= 0. mode paramater is ignored Default: None
  • scale_mode – (str): {‘cycle’, ‘iterations’}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: ‘cycle’
  • verbose – (bool): Whether to produce some output during initialization Default: True
Example:
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = torch.optim.CyclicLR(optimizer)
>>> data_loader = torch.utils.data.DataLoader(...)
>>> for epoch in range(10):
>>>     for batch in data_loader:
>>>         scheduler.batch_step()
>>>         train_batch(...)

EarlyStopping

class pywick.callbacks.EarlyStopping.EarlyStopping(monitor='val_loss', min_delta=0, patience=5, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Early Stopping to terminate training early under certain conditions

EarlyStopping callback to exit the training loop if training or
validation loss does not improve by a certain amount for a certain number of epochs
Parameters:
  • monitor – (string in {‘val_loss’, ‘loss’}): whether to monitor train or val loss
  • min_delta – (float): minimum change in monitored value to qualify as improvement. This number should be positive.
  • patience – (int): number of epochs to wait for improvment before terminating. the counter be reset after each improvment

ExperimentLogger

class pywick.callbacks.ExperimentLogger.ExperimentLogger(directory, filename='Experiment_Logger.csv', save_prefix='Model_', separator=', ', append=True, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Generic logger callback for dumping experiment data. Can be extended for more utility.

History

class pywick.callbacks.History.History(trainer, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Callback that records events into a History object.

This callback is automatically applied to every SuperModule.

LRScheduler

class pywick.callbacks.LRScheduler.LRScheduler(schedule, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Schedule the learning rate according to some function of the current epoch index, current learning rate, and current train/val loss.

Parameters:schedule – (callable): should return a number of learning rates equal to the number of optimizer.param_groups. It should take the epoch index and **kwargs (or logs) as argument. **kwargs (or logs) will return the epoch logs such as mean training and validation loss from the epoch

LambdaCallback

class pywick.callbacks.LambdaCallback.LambdaCallback(on_epoch_begin=None, on_epoch_end=None, on_batch_begin=None, on_batch_end=None, on_train_begin=None, on_train_end=None, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Callback for creating simple, custom callbacks on-the-fly.

ModelCheckpoint

class pywick.callbacks.ModelCheckpoint.ModelCheckpoint(run_id, monitored_log_key, save_dir, addl_k_v=None, epoch_log_keys=None, save_interval=5, save_best_only=False, max_saves=5, custom_func=None, do_minimize=True, verbose=False, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Model Checkpoint to save model weights during training. ‘Best’ is determined by minimizing (or maximizing) the value found under monitored_log_key in the logs Saved checkpoints contain these keys by default:

‘run_id’ ‘epoch’ ‘loss_type’ ‘loss_val’ ‘best_epoch’ - plus any additional key/value pairs produced by custom_func
Additionally saves a .json file with statistics about the run such as:
‘run_id’ ‘num_epochs’ ‘best_epoch’ ‘best_loss_or_gain’ ‘metric_name’ - plus any additional key/value pairs produced by custom_func
Parameters:
  • run_id – (string): Uniquely identifies the run
  • monitored_log_key – (string): Name of the key in the logs that will contain the value we want to minimize (and thus that will dictate whether the model is ‘best’)
  • save_dir – (string): Path indicating where to save the checkpoint
  • addl_k_v – (dict): dictionary of additional key/value pairs to save with the model. Typically these include some initialization parameters, name of the model etc. (e.g. from the initialization dictionary ‘opt’), as well as other useful params (e.g. mean, std, proc_type: gpu/cpu etc)
  • epoch_log_keys – (list): list of keys to save from the epoch log dictionary (Note: the logs dictionary is automatically provided by the learning framework)
  • save_interval – (int): How often to save the model (if none then will default to every 5 iterations)
  • save_best_only – (bool): Whether only to save the best result (and overwrite all previous) Default: False
  • max_saves – (integer > 0 or -1): the max number of models to save. Older model checkpoints will be overwritten if necessary. Set equal to -1 to have no limit. Default: 5
  • custom_func – func(k_v_dict, logs, out_dict, monitored_log_key, is_end_training): Custom function for performing any additional logic (to add values to the model). The function will be passed the addl_k_v dictionary, the event logs dictionary, an output dictionary to process, the monitored_log_key and a bool indicating whether the training is finished. The function is expected to modify the output dictionary in order to preserve values across epochs. The function will be called at the end of each epoch and at the end of the training (with is_end_traing = True)
  • do_minimize – (bool): whether to minimize or maximize the ‘monitored_log_key’ value
  • verbose – (bool): verbosity of the console output Default: False

OneCycleLRScheduler

Callback wrapper around the Pytorch OneCycleLRScheduler

class pywick.callbacks.OneCycleLRScheduler.OnceCycleLRScheduler(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=-1, verbose=True, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.

Parameters:optimizer – (Optimizer) : - Wrapped optimizer.

:param max_lr (float or list) : – Upper learning rate boundaries in the cycle for each parameter group. :param total_steps (int) : – The total number of steps in the cycle. Note that if a value is not provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None :param epochs (int) : – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None :param steps_per_epoch (int) : – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None :param pct_start (float) : – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.3 :param anneal_strategy (str) : – {‘cos’, ‘linear’} Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: ‘cos’ :param cycle_momentum (bool) : – If True, momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Default: True :param base_momentum (float or list) : – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Default: 0.85 :param max_momentum (float or list) : – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Default: 0.95 :param div_factor (float) : – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25 :param final_div_factor (float) : – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4 :param last_epoch (int) : – The index of the last batch. This parameter is used when resuming a training job. Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1

Example:
>>> import torch
>>> data_loader = torch.utils.data.DataLoader(...)
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)
>>> for epoch in range(10):
>>>     for batch in data_loader:
>>>         train_batch(...)
>>>         scheduler.step()

ReduceLROnPlateau

class pywick.callbacks.ReduceLROnPlateau.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, epsilon=0, cooldown=0, min_lr=0, verbose=0, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Reduce the learning rate if the train or validation loss plateaus

Parameters:
  • monitor – (string in {‘loss’, ‘val_loss’}): which metric to monitor
  • factor – (float): factor to decrease learning rate by
  • patience – (int): number of epochs to wait for loss improvement before reducing lr
  • epsilon – (float): how much improvement must be made to reset patience
  • cooldown – (int): number of epochs to cooldown after a lr reduction
  • min_lr – (float): minimum value to ever let the learning rate decrease to
  • verbose – (int): whether to print reduction to console

SimpleModelCheckpoint

class pywick.callbacks.SimpleModelCheckpoint.SimpleModelCheckpoint(directory, filename='ckpt.pth.tar', monitor='val_loss', save_best_only=False, save_weights_only=True, max_save=-1, verbose=0, **kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Simple Checkpoint to save model weights during training. This class is mostly superceded by ModelCheckpoint which provides flexible saving functionality.

Parameters:
  • file – (string): file to which model will be saved. It can be written ‘filename_{epoch}_{loss}’ and those values will be filled in before saving.
  • monitor – (string in {‘val_loss’, ‘loss’}): whether to monitor train or val loss
  • save_best_only – (bool): whether to only save if monitored value has improved
  • save_weights_only – (bool): whether to save entire model or just weights NOTE: only True is supported at the moment
  • max_save – (integer > 0 or -1): the max number of models to save. Older model checkpoints will be overwritten if necessary. Set equal to -1 to have no limit
  • verbose – (integer in {0, 1}): verbosity level
save_checkpoint(epoch, file, is_best=False)[source]

Saves checkpoint to file :param epoch: (int): epoch number :param file: (string): file location :param is_best: (bool): whether this is the best result seen thus far :return:

TQDM

class pywick.callbacks.TQDM.TQDM(**kwargs)[source]

Bases: pywick.callbacks.Callback.Callback

Callback that displays progress bar and useful statistics in terminal