Training Convnet – Deep Learning and Neural Networks with Python and Pytorch p.6




[ad_1]

Welcome to part 6 of the deep learning with Python and Pytorch tutorials. Leading up to this tutorial, we’ve covered how to make a basic neural network, and now we’re going to cover how to make a slightly more complex neural network: The convolutional neural network, or Convnet/CNN.

Text-based tutorials and sample code: https://pythonprogramming.net/convnet-model-deep-learning-neural-network-pytorch/

Linode Cloud GPUs $20 credit: https://linode.com/sentdex

Channel membership: https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ/join
Discord: https://discord.gg/sentdex
Support the content: https://pythonprogramming.net/support-donate/
Twitter: https://twitter.com/sentdex
Instagram: https://instagram.com/sentdex
Facebook: https://www.facebook.com/pythonprogramming.net/
Twitch: https://www.twitch.tv/sentdex

#pytorch #deeplearning #machinelearning

Source


[ad_2]

Comment List

  • sentdex
    November 21, 2020

    How does the neural network work? 20:58

  • sentdex
    November 21, 2020

    Why not train_test_split at 23:34

  • sentdex
    November 21, 2020

    how can i train this. Do i just keep running it?

  • sentdex
    November 21, 2020

    Gonna reach 1Mil soon!!!

  • sentdex
    November 21, 2020

    I do not think it is a good idea to use MSE loss for classification problem because MSE is built for regression problem. In the book PRML, it has justified the natural rise of cross-entropy loss in classification problem.

  • sentdex
    November 21, 2020

    I get the error:

    val_acc, val_loss = fwd_pass(X.view(-1,1,32,32).to(device), y.to(device))
    TypeError: NoneType object is not iterable

    does anyone have a clue Thanks!

  • sentdex
    November 21, 2020

    To know about F.softmax(x,dim=1) see this link the accepted answer :https://stackoverflow.com/questions/52513802/pytorch-softmax-with-dim/52518139

  • sentdex
    November 21, 2020

    how do you put your jupyter notebook in dark mode

  • sentdex
    November 21, 2020

    RuntimeError: Given input size: (128x1x1). Calculated output size: (128x0x0). Ouput size is too small. Does anyone also had this error. I commeted the conv3
    out and then this error accured The size of tensor a (5) must match the size of tensor b (10) at non-singleton dimension 1
    Thank you for your help

  • sentdex
    November 21, 2020

    Awesome video

  • sentdex
    November 21, 2020

    Here's how to work out the size of the output of the convolutional layers instead of using the _to_linear hack:

    (Assuming you are using 1 channel here – the first parameter to Conv2d)

    Each convolution outputs a shape (1, outputs, image-size – kernel-size + 1, image-size – kernel-size + 1)

    So a 50×50 image with a kernel size of 5 would output (1, outputs, 46, 46)

    This makes sense if you think about it. A sliding window of x in an image of size x gives you one option. Add another pixel and you get two options. So it is image-size – x + 1. This assumes you are using a stride of 1 (the amount it is sliding along by each time).

    The relu function doesn't change the shape.

    With the max_pool pools the results are again aggregated using sliding windows. There is a 2×2 window and the stride is the same as the kernel size (2), so this time the number of outputs is inputs / 2 as we are sliding two cells at a time.

    So now the output is (1, outputs, 23, 23)

    Repeat this for the next 2 layers and you get

    (1, 64, 19, 19) # conv 23 – 4 = 19

    (1, 64, 9, 9) # max pool 19 / 2 = 9

    (1, 128, 5, 5) # conv 5 = 9 – 4

    (1, 128, 2, 2) # max pool 2 = 5 / 2

    If you flatten 128 x 2 x 2 you get 512 which is the size that was calculated for the Linear layers.

  • sentdex
    November 21, 2020

    Hi, I am coming from a C/C++ background and I get a little lost by what python assumes and executes without prompt. Can someone tell me how the network is feeding data forward without the forward function ever being called? I'd have assumed you would need a line saying net.forward(x)? I thought maybe python assumes the function to use based on the parameters past but both convs and forward have jsut the x parameter? what am I not understanding?

  • sentdex
    November 21, 2020

    We actually do need activation functions, as they allow the network to learn a non-linear function. If we did not use activations, it would be like a simple linear classifier and thus won't work as expected. Moreover the final layer requires an activation function like softmax to convert the outputs to a probability distribution (values sum to 1) so that it is comparable to the one-hot encoded target that we're passing. Also loss function that is generally used for classification problems is the cross-entropy loss which is kind of a measure of dissimilarity between two distributions and if our target is a probability distribution then we should scale our outputs to be a probability distribution as well, and softmax just guarantees that.

  • sentdex
    November 21, 2020

    This tutorial has really helped please how can I print out the features extracted

  • sentdex
    November 21, 2020

    I am getting 48% accuracy.
    Even with 3 epochs, my loss isnt going down. Its constant at 0.25000
    I also verified this by copy-pasting your code from your website, didnt work
    Can you help?

  • sentdex
    November 21, 2020

    18:35 The reason why activation functions are used is to make the neural network a nonlinear function. If the entire network consisted of just linear functions, then the best function that your neural network is going to represent is a linear function. If that works for your problem, you should be using linear fitting methods instead of neural networks.

  • sentdex
    November 21, 2020

    I am getting an error called 'NoneType' object has no attribute 'view', how can i solve this problem? batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50) it's for this line

  • sentdex
    November 21, 2020

    Plz make this video again using torch.flatten() method..

  • sentdex
    November 21, 2020

    Wait I don't get it, you never called the forward method/function

  • sentdex
    November 21, 2020

    what is the point of the "forward" method if its never used?

  • sentdex
    November 21, 2020

    nice skull cup! 🙂

  • sentdex
    November 21, 2020

    getting runtime error at loss.backward() what to do ???
    RuntimeError: expected dtype Double but got dtype Float (validate_dtype at ..atensrcATennativeTensorIterator.cpp:143)

    (no backtrace available)

  • sentdex
    November 21, 2020

    I am getting the following error while training the model:

    RuntimeError: The size of tensor a (2) must match the size of tensor b (50) at non-singleton dimension 2

    and it points out the loss_function but I don't see any problem there. Here is my code:

    BATCH_SIZE = 100

    EPOCHS = 1

    for epoch in range(EPOCHS):

    for i in tqdm(range(0, len(train_X), BATCH_SIZE)):

    #print(i, i+BATCH_SIZE)

    batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50)

    batch_y = train_y[i:i+BATCH_SIZE]

    net.zero_grad()

    outputs = net(batch_X)

    loss = loss_function(outputs, batch_y)

    loss.backward()

    optimizer.step()

    print(loss)

    Can anyone point me out what am I doing wrong here?

  • sentdex
    November 21, 2020

    Hello, there is something I don't understand :
    net_out = net(test_X[i].view(-1,1,50,50))[0]

    In the previous line, why should put the index [0] ? isn't the output already a tensor like (0.23 , 0.77) (for example) ?

  • sentdex
    November 21, 2020

    Why are you using X, y .. big X, and small y it's very easy to confuse them?

  • sentdex
    November 21, 2020

    What are optimizer, like Adam ?

  • sentdex
    November 21, 2020

    I'm not into pytorch at all but isn't the batch implementation wrong as you call optimizer.step() in the inner loop? I suspect that optimizer.step() is doing the parameter update. I might be wrong but if this is the case, then when you are doing batching, what you really want is to update parameters after each batch rather than after each example.

  • sentdex
    November 21, 2020

    One note: If you have only two classes, it is better to have a single output neuron/value. I recommend to use a simple sigmood function which provides something between 0 and 1 which can be interpreted as a probability already. The nice thing is that you can use an optimized threshold in here. Once your training is done, you can search for the threshold which delivers the best accuracy for the training and or test set (be sure to keep some additional validation set of you optimize here for test set, too). This way, you get a pretty neat binary classifier.

  • sentdex
    November 21, 2020

    Also, I wanted to mention that the lack of a "flatten" function can be good as it forces the programmer to come up with an own way which means that he is at least aware of how this works. There might be a problem regarding "flatten" as this is not unambigious. This can be done arbitrarily. As long as you stay with the exact same framework, this does not matter at all but I often find that people do not even know how the data is processed by the network… Question to some guy who passed me a bunch of neural network parameters to be implemented in a C model: "How did you do the flattening exactly?" "Uhhh, I don't know; used the flatten function." Great.

  • sentdex
    November 21, 2020

    Regarding the switch from convolutional part to fully connected part: I think it is OK to assume that whoever uses neural networks knows the theory behind it. My rule is: If you don't understand what you are using, don't use it! It might get someone into a lot of trouble if it doesn't work out and he doesn't know why because he doesn't have any clue what he is doing… That said you must analyze how the image sizes changes from layer to layer. The numbers which you are passing to the Conv2d layers are just the number of input and output images to/from this layer. If you are doing 5×5 convolutions and use the "inner convolution" mode, the image size reduces by 4 in both, X and Y dimensions (you take kernel dimension size minus 1). E.g. if you have 64×64 images as input to the convolutional layer, you end up with 60×60 images at the output (assuming 5×5 kernels). If this layer provides 50 output images, you have a total of 60x60x50 output pixels. In the one example that you have shown, it seems as though 4×4 is the size of the output images at this particular convolutional layer. Thus, this is multiplied by 50 which is the number of output images. You do not need to pass fake data to determine this. It is better if your code can handle this on it''s own based on the theory.

  • sentdex
    November 21, 2020

    Hey, with the convolutional layers: It really depends on your data. If you go for image stuff, sure convolutional layers are the natural choice. Generally, if your features (which are pixels in images) must underlie some sort of locality principle because otherwise it does not make sense to use convolutional layers at all. Therefore, if you have derived a bunch of own features which are more or less independent of each other via some preprocessing (e.g. for other types of classification), you must use something else.

  • sentdex
    November 21, 2020

    Could you please create a GitHub account and put the code there?

  • sentdex
    November 21, 2020

    Can you please make a shirt with your rant in 20:50 ?
    Loved it.

  • sentdex
    November 21, 2020

    Why MSE loss is used for classification here?

  • sentdex
    November 21, 2020

    my face when I didn't make typo and sentdex did

  • sentdex
    November 21, 2020

    These videos have taught a valuable lesson. To take with a can of salt any predictions that in 20 to 40 years we will have sentient AI. It's pretty hard to make an image comparative program let alone A sentient android.

  • sentdex
    November 21, 2020
  • sentdex
    November 21, 2020

    THIS IS HOW TO CALCULATE THE FLATTEN INPUT SIZE:

    In your case each:
    – Conv2D layer output size is (InputSize – KernelSize + 1) => (InputSize + 4). Attention, this only applies for stride=1 and padding=0
    – MaxPool2D layer output size is floor(InputSize/PoolSize) => floor(InputSize/2). Attention, this only applies for stride=KernelSize and padding=0

    For your case you start with 50×50 image size, then:
    50x50x1 –conv2d–> 46x46x32 –pool–> 23x23x32 –conv2d–> 19x19x64 –pool–> 9x9x64 –conv2d–> 5x5x128 –pool–> 2x2x128

    This way you get 2x2x128 as input for the FC-layer which is 512 inputs in total

  • sentdex
    November 21, 2020

    Hi, just wondering – how is PyTorch is more convenient than Keras? In keras I've never had manually to deal with flattening, nor with gradient zeroing. I mean – if PyTorch is used, there must be an advantage over Keras. But if it's not the syntax and not the run time then what is it?

  • sentdex
    November 21, 2020

    In tensorflow we don't have to mention the input size to the next convolutional layers right? We just write the number of filters? But here I see that we have to mention input size same as the output of previous layer.

  • sentdex
    November 21, 2020

    Just by looking through some of the data manually, you see there is some serious junk in it… Love it… ^^
    E.g. There is a Yahoo label declared as a dog which obviously is not the case because they are not cute…

  • sentdex
    November 21, 2020

    can you do tutorial on RNN with pytorch

  • sentdex
    November 21, 2020

    if i have 3 color channel do i need to put self.conv1=nn.Conv2d(3,32,5)? please help

  • sentdex
    November 21, 2020

    "expected dtype Double but got dtype Float"……….. I am repeatedly getting this error message in the segment where we backpropagate. i.e. loss.backward(). [32:51 in this video]. Please help!

  • sentdex
    November 21, 2020

    could have liked it twice, but sadly there is no option.

  • sentdex
    November 21, 2020

    See the mathematical formula of softmax to understand what this dim argument is doig

  • sentdex
    November 21, 2020

    Thanks for the tutorial! A little advice regarding code style:

    instead of

    x[0].shape[0]*x[0].shape[1]*x[0].shape[2]

    better make

    np.prod(x[0].shape)

  • sentdex
    November 21, 2020

    I ran this with an NVME on a quad core laptop and speed was much faster than here. I think the IO was the dependency, not CPU/GPU, but heh.

  • sentdex
    November 21, 2020

    is your cpu just rather slow or is it because you are also recording at the same time, i just ran this on a laptop and got way faster results and it is just a 9th gen i7 laptop cpu

Write a comment