## 8- TRAINING A NEURAL NETWORK: Implementing backpropagation and gradient descent from scratch

In this video, I implement backpropagation and gradient descent from scratch using the Python programming language. I also train the network to perform an incredibly hard task: the arithmetic sum 😀

Code:
https://github.com/musikalkemist/DeepLearningForAudioWithPython/tree/master/8-%20Training%20a%20neural%20network:%20Implementing%20back%20propagation%20from%20scratch/code

Melodrive:
https://melodrive.com/

Source

### Comment List

• Valerio Velardo - The Sound of AI
December 19, 2020

Probably the only backprop tutorial I've been able to follow all the way through, thank you so much 🙂

• Valerio Velardo - The Sound of AI
December 19, 2020

16:19 Sir, why are you using _sigmoid_derivative(activations) instead of _sigmoid_derivative(h_{i+1}); am I missing something?

• Valerio Velardo - The Sound of AI
December 19, 2020

at 21:07, instead of doing the reshape, can you not just do [[x] for x in self.activations[i]]?

• Valerio Velardo - The Sound of AI
December 19, 2020

Hi, great how to include biases in line with weight matrices and ReLU in line with Sigmoid activation?

• Valerio Velardo - The Sound of AI
December 19, 2020

Excellent course
13/11/2020

• Valerio Velardo - The Sound of AI
December 19, 2020

I have question in MSE backprop don't we need to do derivative of error before sending to backprop function?

• Valerio Velardo - The Sound of AI
December 19, 2020

I wish I could pay you the money worth of the time and money you have saved for me. I never imagined that I would find such an solid and brilliant implementation of nural network on YouTube. I am paying my deep gratitude to you for putting your time and brilliant work in the video. Wishing you the best and please keep teaching us.

• Valerio Velardo - The Sound of AI
December 19, 2020

You didn’t incorporate bias in the network. How would you reflect it since the bias needs updating in back prop?

• Valerio Velardo - The Sound of AI
December 19, 2020

In the previous video you had mentioned that we should go down the gradient in order to get to the global minima, in this video we are adding it up to the sum_error instead subtracting it .
Is that becuase in the previous one we were considering error = a(3) – y and here we are taking it as y – a(3) … ?hence the addition ?
Also one more Doubt :
if activations = sigmoid(x) <– this returns a single float value but @ 20:18 we are considering the activations to be a ndarray. Where do we acutally change the structure
Edit 3: understood that sigmoid(x) takes in a numpy array and computes the activations and returns a numpy array itself !
Thanks!

• Valerio Velardo - The Sound of AI
December 19, 2020

Definitely the best code for entry level MLP and you seem to have the knack of explaining the basics and coding it in the most elegant way. Your approach of Teach By Example (data) worked out very well for me, since I am a Learn By Example person. Thanks for your time putting this series. Definitely learnt few things.

• Valerio Velardo - The Sound of AI
December 19, 2020

You, Sir, are a true Legend, a splendid tutorial boss. Godspeed m/

• Valerio Velardo - The Sound of AI
December 19, 2020

56:21 Epic moment.

What a great example Valerio, I'm having a really good feeling that I'm getting all the concept, Thanks!

• Valerio Velardo - The Sound of AI
December 19, 2020

Please anyone teach me why there are no Bias terms in this Neural network model.

• Valerio Velardo - The Sound of AI
December 19, 2020

Thank you for this video

• Valerio Velardo - The Sound of AI
December 19, 2020

how come there is no bias vector in this network?

• Valerio Velardo - The Sound of AI
December 19, 2020

Thanks Valerio for your step-by-step and comprehensive explanation. I copied your code, got up to the end of training perfectly. However, when I run the code with a new set of inputs (executing the forward_propagate function only with a new dataset), I do not get the result I am expecting which is strange as the weights have been adjusted post training and the expected results should be generated. Any advice?

• Valerio Velardo - The Sound of AI
December 19, 2020

Dear, this tutor is very much interesting in training of a neural net. It is highly appreciated how to create or supply array input datasets say using nested for loop method to train and implement forward/backward propagation in ANN. In this case, say 4 raster input datasets (R1, R2, R3 and R4) of which 2 datasets (R1 and R2) having time, longitude and latitude dimensions, and 2 of them (R3 and R4) have no time dimension. The shape of the Raster data is (time =1000, latitude =310, longitude =458). Thank you in advance for your time and knowledge sharing.

• Valerio Velardo - The Sound of AI
December 19, 2020

It works as is, but I can't change the number of input neurons, I tried to go from 2 to 4 in MLP() , I changed the inputs value as well (4 elements in each samples) but got a shape error between "activations" and "w" in np.dot()

• Valerio Velardo - The Sound of AI
December 19, 2020

Woahh, I got it running!!! Thank you so much , Valerio!

• Valerio Velardo - The Sound of AI
December 19, 2020

Finally, a video that talks to engineers. Well done.

• Valerio Velardo - The Sound of AI
December 19, 2020

This is one of the best NN series I've even watched! Very well explained! Thanks!