Here you can find a collection of functions that are used in neural networks. One of the most important aspects of a neural network is a good activation function. Pytorch already has a solid collection of activation functions but here are a few more experimental ones to play around with.


class pywick.functions.cyclicLR.CyclicLR(optimizer, base_lr=0.001, max_lr=0.006, step_size=2000, mode='triangular', gamma=1.0, scale_fn=None, scale_mode='cycle', last_batch_iteration=-1)[source]

Bases: object

Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks. The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis.

Cyclical learning rate policy changes the learning rate after every batch. batch_step should be called after a batch has been used for training. To resume training, save last_batch_iteration and use it to instantiate CycleLR.

This class has three built-in policies, as put forth in the paper:

triangular: A basic triangular cycle w/ no amplitude scaling.

triangular2: A basic triangular cycle that scales initial amplitude by half each cycle.

exp_range: A cycle that scales initial amplitude by gamma**(cycle iterations) at each cycle iteration.

This implementation was adapted from the github repo: bckenstler/CLR

  • optimizer – (Optimizer): Wrapped optimizer.
  • base_lr – (float or list): Initial learning rate which is the lower boundary in the cycle for each param groups. Default: 0.001
  • max_lr – (float or list): Upper boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_lr - base_lr). The lr at any cycle is the sum of base_lr and some scaling of the amplitude; therefore max_lr may not actually be reached depending on scaling function. Default: 0.006
  • step_size – (int): Number of training iterations per half cycle. Authors suggest setting step_size 2-8 x training iterations in epoch. Default: 2000
  • mode – (str): One of {triangular, triangular2, exp_range}. Values correspond to policies detailed above. If scale_fn is not None, this argument is ignored. Default: ‘triangular’
  • gamma – (float): Constant in ‘exp_range’ scaling function: gamma**(cycle iterations) Default: 1.0
  • scale_fn – (function): Custom scaling policy defined by a single argument lambda function, where 0 <= scale_fn(x) <= 1 for all x >= 0. mode paramater is ignored Default: None
  • scale_mode – (str): {‘cycle’, ‘iterations’}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: ‘cycle’
  • last_batch_iteration – (int): The index of the last batch. Default: -1
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = torch.optim.CyclicLR(optimizer)
>>> data_loader =
>>> for epoch in range(10):
>>>     for batch in data_loader:
>>>         scheduler.batch_step()
>>>         train_batch(...)


class pywick.functions.mish.Mish(inplace: bool = False)[source]

Bases: sphinx.ext.autodoc.importer._MockObject

Mish - “Mish: A Self Regularized Non-Monotonic Neural Activation Function” implemented for PyTorch / FastAI by lessw2020 github:

pywick.functions.mish.mish(x, inplace: bool = False)[source]

Mish: A Self Regularized Non-Monotonic Neural Activation Function -

Swish + Aria

class pywick.functions.swish.Aria(A=0, K=1.0, B=1.0, v=1.0, C=1.0, Q=1.0)[source]

Bases: sphinx.ext.autodoc.importer._MockObject

Aria activation function described in this paper.

class pywick.functions.swish.Aria2(a=1.5, b=2.0)[source]

Bases: sphinx.ext.autodoc.importer._MockObject

ARiA2 activation function, a special case of ARiA, for ARiA = f(x, 1, 0, 1, 1, b, 1/a)

class pywick.functions.swish.HardSwish(inplace: bool = False)[source]

Bases: sphinx.ext.autodoc.importer._MockObject

class pywick.functions.swish.Swish(b=1.0)[source]

Bases: sphinx.ext.autodoc.importer._MockObject

Swish activation function, a special case of ARiA, for ARiA = f(x, 1, 0, 1, 1, b, 1)

pywick.functions.swish.hard_swish(x, inplace: bool = False)[source]