Callbacks¶
Callbacks are the primary mechanism by which one can embed event hooks into the training process. Many useful callbacks are provided out of the box but in all likelihood you will want to implement your own to execute actions based on training events. To do so, simply extend the pywick.callbacks.Callback class and overwrite functions that you are interested in acting upon.
CSVLogger¶

class
pywick.callbacks.CSVLogger.
CSVLogger
(file, separator=', ', append=False, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Logs epochlevel metrics to a CSV file
Parameters:  file – (string) path to csv file
 separator – (string) delimiter for file
 append – (bool) whether to append result to existing file or make new file
Callback¶

class
pywick.callbacks.Callback.
Callback
(params: dict = None, trainer=None, **_)[source]¶ Bases:
object
Abstract base class used to build new callbacks. Extend this class to build your own callbacks and overwrite functions that you want to monitor. Functions will be called automatically from the trainer once per relevant training event (e.g. at the beginning of epoch, end of epoch, beginning of batch, end of batch etc.)

is_enabled
()[source]¶ Returns the status of this callback. Default = True but can be turned off with param[‘is_enabled’] = False :return:

on_batch_begin
(batch: int, logs: Collection = None)[source]¶ Called at the beginning of a new batch :param batch: batch number :param logs: collection of logs to process / parse / add to :return:

on_batch_end
(batch: int, logs: Collection = None)[source]¶ Called at the end of an epoch :param batch: batch number :param logs: collection of logs to process / parse / add to :return:

on_epoch_begin
(epoch: int, logs: Collection = None)[source]¶ Called at the beginning of a new epoch :param epoch: epoch number :param logs: collection of logs to process / parse / add to :return:

on_epoch_end
(epoch: int, logs: Collection = None)[source]¶ Called at the end of an epoch :param epoch: epoch number :param logs: collection of logs to process / parse / add to :return:

CyclicLRScheduler¶

class
pywick.callbacks.CyclicLRScheduler.
CyclicLRScheduler
(optimizer, base_lr=0.001, max_lr=0.006, step_size=2000, mode='triangular', gamma=1.0, scale_fn=None, scale_mode='cycle', verbose=True, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks. The distance between the two boundaries can be scaled on a periteration or percycle basis.
Cyclical learning rate policy changes the learning rate after every batch. batch_step should be called after a batch has been used for training. To resume training, save last_batch_iteration and use it to instantiate CycleLR.
This class has three builtin policies, as put forth in the paper: “triangular”:
A basic triangular cycle w/ no amplitude scaling. “triangular2”:
 A basic triangular cycle that scales initial amplitude by half each cycle.
 “exp_range”:
 A cycle that scales initial amplitude by gamma**(cycle iterations) at each cycle iteration.
This implementation was adapted from the github repo: bckenstler/CLR
Parameters:  optimizer – (Optimizer): Wrapped optimizer.
 base_lr – (float or list): Initial learning rate which is the lower boundary in the cycle for eachparam groups. Default: 0.001
 max_lr – (float or list): Upper boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_lr  base_lr). The lr at any cycle is the sum of base_lr and some scaling of the amplitude; therefore max_lr may not actually be reached depending on scaling function. Default: 0.006
 step_size – (int): Number of training iterations per half cycle. Authors suggest setting step_size 28 x training iterations in epoch. Default: 2000
 mode – (str): One of {triangular, triangular2, exp_range}. Values correspond to policies detailed above. If scale_fn is not None, this argument is ignored. Default: ‘triangular’
 gamma – (float): Constant in ‘exp_range’ scaling function: gamma**(cycle iterations) Default: 1.0
 scale_fn – (function): Custom scaling policy defined by a single argument lambda function, where 0 <= scale_fn(x) <= 1 for all x >= 0. mode paramater is ignored Default: None
 scale_mode – (str): {‘cycle’, ‘iterations’}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: ‘cycle’
 verbose – (bool): Whether to produce some output during initialization Default: True
 Example:
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = torch.optim.CyclicLR(optimizer) >>> data_loader = torch.utils.data.DataLoader(...) >>> for epoch in range(10): >>> for batch in data_loader: >>> scheduler.batch_step() >>> train_batch(...)
EarlyStopping¶

class
pywick.callbacks.EarlyStopping.
EarlyStopping
(monitor='val_loss', min_delta=0, patience=5, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Early Stopping to terminate training early under certain conditions
 EarlyStopping callback to exit the training loop if training or
 validation loss does not improve by a certain amount for a certain number of epochs
Parameters:  monitor – (string in {‘val_loss’, ‘loss’}): whether to monitor train or val loss
 min_delta – (float): minimum change in monitored value to qualify as improvement. This number should be positive.
 patience – (int): number of epochs to wait for improvment before terminating. the counter be reset after each improvment
ExperimentLogger¶

class
pywick.callbacks.ExperimentLogger.
ExperimentLogger
(directory, filename='Experiment_Logger.csv', save_prefix='Model_', separator=', ', append=True, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Generic logger callback for dumping experiment data. Can be extended for more utility.
History¶

class
pywick.callbacks.History.
History
(trainer, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Callback that records events into a History object.
This callback is automatically applied to every SuperModule.
LRScheduler¶

class
pywick.callbacks.LRScheduler.
LRScheduler
(schedule, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Schedule the learning rate according to some function of the current epoch index, current learning rate, and current train/val loss.
Parameters: schedule – (callable): should return a number of learning rates equal to the number of optimizer.param_groups. It should take the epoch index and **kwargs (or logs) as argument. **kwargs (or logs) will return the epoch logs such as mean training and validation loss from the epoch
LambdaCallback¶

class
pywick.callbacks.LambdaCallback.
LambdaCallback
(on_epoch_begin=None, on_epoch_end=None, on_batch_begin=None, on_batch_end=None, on_train_begin=None, on_train_end=None, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Callback for creating simple, custom callbacks onthefly.
ModelCheckpoint¶

class
pywick.callbacks.ModelCheckpoint.
ModelCheckpoint
(run_id, monitored_log_key, save_dir, addl_k_v=None, epoch_log_keys=None, save_interval=5, save_best_only=False, max_saves=5, custom_func=None, do_minimize=True, verbose=False, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Model Checkpoint to save model weights during training. ‘Best’ is determined by minimizing (or maximizing) the value found under monitored_log_key in the logs Saved checkpoints contain these keys by default:
‘run_id’ ‘epoch’ ‘loss_type’ ‘loss_val’ ‘best_epoch’  plus any additional key/value pairs produced by custom_func Additionally saves a .json file with statistics about the run such as:
 ‘run_id’ ‘num_epochs’ ‘best_epoch’ ‘best_loss_or_gain’ ‘metric_name’  plus any additional key/value pairs produced by custom_func
Parameters:  run_id – (string): Uniquely identifies the run
 monitored_log_key – (string): Name of the key in the logs that will contain the value we want to minimize (and thus that will dictate whether the model is ‘best’)
 save_dir – (string): Path indicating where to save the checkpoint
 addl_k_v – (dict): dictionary of additional key/value pairs to save with the model. Typically these include some initialization parameters, name of the model etc. (e.g. from the initialization dictionary ‘opt’), as well as other useful params (e.g. mean, std, proc_type: gpu/cpu etc)
 epoch_log_keys – (list): list of keys to save from the epoch log dictionary (Note: the logs dictionary is automatically provided by the learning framework)
 save_interval – (int): How often to save the model (if none then will default to every 5 iterations)
 save_best_only – (bool): Whether only to save the best result (and overwrite all previous) Default: False
 max_saves – (integer > 0 or 1): the max number of models to save. Older model checkpoints will be overwritten if necessary. Set equal to 1 to have no limit. Default: 5
 custom_func – func(k_v_dict, logs, out_dict, monitored_log_key, is_end_training): Custom function for performing any additional logic (to add values to the model). The function will be passed the addl_k_v dictionary, the event logs dictionary, an output dictionary to process, the monitored_log_key and a bool indicating whether the training is finished. The function is expected to modify the output dictionary in order to preserve values across epochs. The function will be called at the end of each epoch and at the end of the training (with is_end_traing = True)
 do_minimize – (bool): whether to minimize or maximize the ‘monitored_log_key’ value
 verbose – (bool): verbosity of the console output Default: False
OneCycleLRScheduler¶
Callback wrapper around the Pytorch OneCycleLRScheduler

class
pywick.callbacks.OneCycleLRScheduler.
OnceCycleLRScheduler
(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=1, verbose=True, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper SuperConvergence: Very Fast Training of Neural Networks Using Large Learning Rates.
Parameters: optimizer – (Optimizer) :  Wrapped optimizer. :param max_lr (float or list) : – Upper learning rate boundaries in the cycle for each parameter group. :param total_steps (int) : – The total number of steps in the cycle. Note that if a value is not provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None :param epochs (int) : – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None :param steps_per_epoch (int) : – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None :param pct_start (float) : – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.3 :param anneal_strategy (str) : – {‘cos’, ‘linear’} Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: ‘cos’ :param cycle_momentum (bool) : – If True, momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Default: True :param base_momentum (float or list) : – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Default: 0.85 :param max_momentum (float or list) : – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum  base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Default: 0.95 :param div_factor (float) : – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25 :param final_div_factor (float) : – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4 :param last_epoch (int) : – The index of the last batch. This parameter is used when resuming a training job. Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=1, the schedule is started from the beginning. Default: 1
 Example:
>>> import torch >>> data_loader = torch.utils.data.DataLoader(...) >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10) >>> for epoch in range(10): >>> for batch in data_loader: >>> train_batch(...) >>> scheduler.step()
ReduceLROnPlateau¶

class
pywick.callbacks.ReduceLROnPlateau.
ReduceLROnPlateau
(monitor='val_loss', factor=0.1, patience=10, epsilon=0, cooldown=0, min_lr=0, verbose=0, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Reduce the learning rate if the train or validation loss plateaus
Parameters:  monitor – (string in {‘loss’, ‘val_loss’}): which metric to monitor
 factor – (float): factor to decrease learning rate by
 patience – (int): number of epochs to wait for loss improvement before reducing lr
 epsilon – (float): how much improvement must be made to reset patience
 cooldown – (int): number of epochs to cooldown after a lr reduction
 min_lr – (float): minimum value to ever let the learning rate decrease to
 verbose – (int): whether to print reduction to console
SimpleModelCheckpoint¶

class
pywick.callbacks.SimpleModelCheckpoint.
SimpleModelCheckpoint
(directory, filename='ckpt.pth.tar', monitor='val_loss', save_best_only=False, save_weights_only=True, max_save=1, verbose=0, **kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Simple Checkpoint to save model weights during training. This class is mostly superceded by ModelCheckpoint which provides flexible saving functionality.
Parameters:  file – (string): file to which model will be saved. It can be written ‘filename_{epoch}_{loss}’ and those values will be filled in before saving.
 monitor – (string in {‘val_loss’, ‘loss’}): whether to monitor train or val loss
 save_best_only – (bool): whether to only save if monitored value has improved
 save_weights_only – (bool): whether to save entire model or just weights NOTE: only True is supported at the moment
 max_save – (integer > 0 or 1): the max number of models to save. Older model checkpoints will be overwritten if necessary. Set equal to 1 to have no limit
 verbose – (integer in {0, 1}): verbosity level
TQDM¶

class
pywick.callbacks.TQDM.
TQDM
(**kwargs)[source]¶ Bases:
pywick.callbacks.Callback.Callback
Callback that displays progress bar and useful statistics in terminal