Activation Functions: Let’s see new activation functions

less than 1 minute read

Cover Page

GELU (Gaussian Error Linear Unit)

Cover Page

The GELU nonlinearity is the expected transformation of a stochastic regularizer which randomly applies the identity or zero map to a neuron’s input

1. Equation:

Cover Page

2. GELU Experiments

Classfication Experiment: MNIST classification

Cover Page

Autoencoder Experiment: MNIST Autoencoder

Cover Page

Reference:

https://arxiv.org/pdf/1606.08415.pdf https://github.com/hendrycks/GELUs

LiSHT (Linearly Scaled Hyperbolic Tangent Activation)

Cover Page

1. Equation:

Cover Page

2. LiSHT Experiments

Classification Experiment: MNIST & CIFAR10

Cover Page

Sentiment Classification Results using LSTM

Cover Page

Reference

https://arxiv.org/pdf/1901.05894.pdf

SWISH

Cover Page

1. Equation:

Cover Page

2. SWISH Experiments

Machine Translation

Cover Page

Reference

https://arxiv.org/pdf/1710.05941.pdf

Mish

Cover Page

1. Equation:

Cover Page

Cover Page

2. Mish Experiments

Output Landscape of a Random Neural Network

Cover Page

Testing Accuracy v/s Number of Layers on MNIST

Cover Page

Test Accuracy v/s Batch Size on CIFAR-10

Cover Page

Reference

https://arxiv.org/pdf/1908.08681.pdf

Other Activation Functions

Rectified Activations: https://arxiv.org/pdf/1505.00853.pdf

Sparsemax: https://arxiv.org/pdf/1602.02068.pdf

counter free

Comments