We've all dealt with activation functions while working with neural...
Akshay 🚀@akshay_pachaar
57 views
Jun 05, 2025
~1 min read
Advertisement
1
We've all dealt with activation functions while working with neural nets.
- Sigmoid
- Tanh
- ReLu & Leaky ReLu
- Gelu
Ever wondered why they are so important❓🤔
Let me explain it to you in this 🧵👇
- Sigmoid
- Tanh
- ReLu & Leaky ReLu
- Gelu
Ever wondered why they are so important❓🤔
Let me explain it to you in this 🧵👇
2
Before we proceed I want you to understand something!
You can think of a layer in neural net as a function & multiple layers makes the network a composite function.
Now, a composite function consisting of individual linear functions is also linear.
Check this 👇
You can think of a layer in neural net as a function & multiple layers makes the network a composite function.
Now, a composite function consisting of individual linear functions is also linear.
Check this 👇
3
We have a simple neural net that does binary classification.
Scenario 1:
- Linear decision boundary
- Linear Activation function
Observe how the neural net is able to quickly learn & loss converges to zero.
Watch this 👇
Scenario 1:
- Linear decision boundary
- Linear Activation function
Observe how the neural net is able to quickly learn & loss converges to zero.
Watch this 👇
0:29
4
Scenario 2:
- Non Linear decision boundary
- Linear Activation function
Observe how the neural net struggles to learn & the loss consistently remains high!
With linear activations it's unable to create a non-linear decision boundary.
Watch this 👇
- Non Linear decision boundary
- Linear Activation function
Observe how the neural net struggles to learn & the loss consistently remains high!
With linear activations it's unable to create a non-linear decision boundary.
Watch this 👇
0:46
5
Scenario 3:
- Non Linear decision boundary
- Non-linear Activation function (Sigmoid)
Observe how the neural net performs well this time.
With a non-linear activation function we give the network ability to create a non-linear decision boundary.
Watch this 👇
- Non Linear decision boundary
- Non-linear Activation function (Sigmoid)
Observe how the neural net performs well this time.
With a non-linear activation function we give the network ability to create a non-linear decision boundary.
Watch this 👇
0:42
6
Now we understand why activation functions are important.
Next time we see why do we need different flavours of these non-linear activation functions.
What are the advantages of one over other.
You can play around like i did in the videos here 👇
playground.tensorflow.org
Next time we see why do we need different flavours of these non-linear activation functions.
What are the advantages of one over other.
You can play around like i did in the videos here 👇
playground.tensorflow.org
7
That's a wrap!
If you interested in:
- Python 🐍
- Data Science 📈
- Machine Learning 🤖
- Maths for ML 🧮
- MLOps 🛠
- NLP 🗣
- Computer Vision 🎥
- LLMs 🧠
I'm sharing daily content over here, follow me → @akshay_pachaar if you haven't already!!
Cheers!! 🙂
If you interested in:
- Python 🐍
- Data Science 📈
- Machine Learning 🤖
- Maths for ML 🧮
- MLOps 🛠
- NLP 🗣
- Computer Vision 🎥
- LLMs 🧠
I'm sharing daily content over here, follow me → @akshay_pachaar if you haven't already!!
Cheers!! 🙂