Functions¶
Here you can find a collection of functions that are used in neural networks. One of the most important aspects of a neural network is a good activation function. Pytorch already has a solid collection of activation functions but here are a few more experimental ones to play around with.
CyclicLR¶
-
class
pywick.functions.cyclicLR.
CyclicLR
(optimizer, base_lr=0.001, max_lr=0.006, step_size=2000, mode='triangular', gamma=1.0, scale_fn=None, scale_mode='cycle', last_batch_iteration=-1)[source]¶ Bases:
object
Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks. The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis.
Cyclical learning rate policy changes the learning rate after every batch. batch_step should be called after a batch has been used for training. To resume training, save last_batch_iteration and use it to instantiate CycleLR.
This class has three built-in policies, as put forth in the paper:
triangular: A basic triangular cycle w/ no amplitude scaling.
triangular2: A basic triangular cycle that scales initial amplitude by half each cycle.
exp_range: A cycle that scales initial amplitude by gamma**(cycle iterations) at each cycle iteration.
This implementation was adapted from the github repo: bckenstler/CLR
Parameters: - optimizer – (Optimizer): Wrapped optimizer.
- base_lr – (float or list): Initial learning rate which is the lower boundary in the cycle for each param groups. Default: 0.001
- max_lr – (float or list): Upper boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_lr - base_lr). The lr at any cycle is the sum of base_lr and some scaling of the amplitude; therefore max_lr may not actually be reached depending on scaling function. Default: 0.006
- step_size – (int): Number of training iterations per half cycle. Authors suggest setting step_size 2-8 x training iterations in epoch. Default: 2000
- mode – (str): One of {triangular, triangular2, exp_range}. Values correspond to policies detailed above. If scale_fn is not None, this argument is ignored. Default: ‘triangular’
- gamma – (float): Constant in ‘exp_range’ scaling function: gamma**(cycle iterations) Default: 1.0
- scale_fn – (function): Custom scaling policy defined by a single argument lambda function, where 0 <= scale_fn(x) <= 1 for all x >= 0. mode paramater is ignored Default: None
- scale_mode – (str): {‘cycle’, ‘iterations’}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: ‘cycle’
- last_batch_iteration – (int): The index of the last batch. Default: -1
- Example:
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = torch.optim.CyclicLR(optimizer) >>> data_loader = torch.utils.data.DataLoader(...) >>> for epoch in range(10): >>> for batch in data_loader: >>> scheduler.batch_step() >>> train_batch(...)
Mish¶
-
class
pywick.functions.mish.
Mish
(inplace: bool = False)[source]¶ Bases:
sphinx.ext.autodoc.importer._MockObject
Mish - “Mish: A Self Regularized Non-Monotonic Neural Activation Function” https://arxiv.org/abs/1908.08681v1 implemented for PyTorch / FastAI by lessw2020 github: https://github.com/lessw2020/mish
-
pywick.functions.mish.
mish
(x, inplace: bool = False)[source]¶ Mish: A Self Regularized Non-Monotonic Neural Activation Function - https://arxiv.org/abs/1908.08681
Swish + Aria¶
-
class
pywick.functions.swish.
Aria
(A=0, K=1.0, B=1.0, v=1.0, C=1.0, Q=1.0)[source]¶ Bases:
sphinx.ext.autodoc.importer._MockObject
Aria activation function described in this paper.
-
class
pywick.functions.swish.
Aria2
(a=1.5, b=2.0)[source]¶ Bases:
sphinx.ext.autodoc.importer._MockObject
ARiA2 activation function, a special case of ARiA, for ARiA = f(x, 1, 0, 1, 1, b, 1/a)
-
class
pywick.functions.swish.
HardSwish
(inplace: bool = False)[source]¶ Bases:
sphinx.ext.autodoc.importer._MockObject