Loading in your own data – Deep Learning basics with Python, TensorFlow and Keras p.2




[ad_1]

Welcome to a tutorial where we’ll be discussing how to load in our own outside datasets, which comes with all sorts of challenges!

First, we need a dataset. Let’s grab the Dogs vs Cats dataset from Microsoft: https://www.microsoft.com/en-us/download/confirmation.aspx?id=54765

Text tutorials and sample code: https://pythonprogramming.net/loading-custom-data-deep-learning-python-tensorflow-keras/

Discord: https://discord.gg/sentdex
Support the content: https://pythonprogramming.net/support-donate/
Twitter: https://twitter.com/sentdex
Facebook: https://www.facebook.com/pythonprogramming.net/
Twitch: https://www.twitch.tv/sentdex
G+: https://plus.google.com/+sentdex

Source


[ad_2]

Comment List

  • sentdex
    November 15, 2020

    Can you make video on how to create raster(.geotiff) dataset in python

  • sentdex
    November 15, 2020

    I get the following error:
    ValueError: Data cardinality is ambiguous:

    x sizes: 22451

    y sizes: 0

    Please provide data which shares the same first dimension.

    İ used codes:
    ______________________________________________________________________________________________________________

    import numpy as np

    import matplotlib.pyplot as plt

    import os

    import cv2

    from tqdm import tqdm

    DATADIR = "dataset/"

    CATEGORIES = ["dog", "cat"]

    for category in CATEGORIES: # do dogs and cats

    path = os.path.join(DATADIR,category) # create path to dogs and cats

    for img in os.listdir(path): # iterate over each image per dogs and cats

    img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE) # convert to array

    plt.imshow(img_array, cmap='gray') # graph it

    plt.show() # display!

    break # we just want one for now so break

    break #…and one more!

    print(img_array.shape)

    training_data = []

    IMG_SIZE=50

    def create_training_data():

    for category in CATEGORIES: # do dogs and cats

    path = os.path.join(DATADIR,category) # create path to dogs and cats

    class_num = CATEGORIES.index(category) # get the classification (0 or a 1). 0=dog 1=cat

    for img in tqdm(os.listdir(path)): # iterate over each image per dogs and cats

    try:

    img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE) # convert to array

    new_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE)) # resize to normalize data size

    training_data.append([new_array, class_num]) # add this to our training_data

    except Exception as e: # in the interest in keeping the output clean…

    pass

    #except OSError as e:

    # print("OSErrroBad img most likely", e, os.path.join(path,img))

    #except Exception as e:

    # print("general exception", e, os.path.join(path,img))

    create_training_data()

    #print(len(training_data))

    import random

    random.shuffle(training_data)

    for sample in training_data[:10]:

    print(sample[1])

    X = []

    y = []

    y=np.array(y)

    for features, label in training_data:

    X.append(features)

    #y.append(label)

    np.array((y,label))

    #print(X[0].reshape(-1, IMG_SIZE, IMG_SIZE, 1))

    X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1)

    #X[3]

    import pickle

    pickle_out = open("X.pickle","wb")

    pickle.dump(X, pickle_out)

    pickle_out.close()

    pickle_out = open("y.pickle","wb")

    pickle.dump(y, pickle_out)

    pickle_out.close()

    pickle_in = open("X.pickle","rb")

    X = pickle.load(pickle_in)

    pickle_in = open("y.pickle","rb")

    y = pickle.load(pickle_in)

    # In[79]:

    import tensorflow as tf

    from tensorflow.keras.datasets import cifar10

    from tensorflow.keras.preprocessing.image import ImageDataGenerator

    from tensorflow.keras.models import Sequential

    from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten

    from tensorflow.keras.layers import Conv2D, MaxPooling2D

    import pickle

    pickle_in = open("X.pickle","rb")

    X = pickle.load(pickle_in)

    pickle_in = open("y.pickle","rb")

    y = pickle.load(pickle_in)

    X = X/255.0

    model = Sequential()

    model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))

    model.add(Activation('relu'))

    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Conv2D(64, (3, 3)))

    model.add(Activation('relu'))

    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Flatten()) # this converts our 3D feature maps to 1D feature vectors

    model.add(Dense(64))

    model.add(Dense(1))

    model.add(Activation('sigmoid'))

    model.compile(loss='binary_crossentropy',

    optimizer='adam',

    metrics=['accuracy'])

    model.fit(X, y, batch_size=32, epochs=3, validation_split=.1)

  • sentdex
    November 15, 2020

    import numpy as np

    import matplotlib.pyplot as plt

    import os

    import cv2

    DATADIR = "E:/BHARGABH_PLC/Tensorflow Logic/PetImages/Dog"

    CATEGORIES = ["Cat","Dog"]

    for category in CATEGORIES:

    path = os.path.join(DATADIR,category) #path to dogs or cats dir

    for img in os.listdir(path):

    img_array = cv2.imread(os.path.join(path,img),cv2.IMREAD_GRAYSCALE)

    Dear sir why i am got this error
    FileNotFoundError: [Errno 2] No such file or directory: 'E:/BHARGABH_PLC/Tensorflow Logic/PetImages/Dog/Cat'
    i have download and save the file on this dir E:BHARGABH_PLCTensorflow LogicPetImages

  • sentdex
    November 15, 2020

    how to download tflite file after its conversation from h5 file?

  • sentdex
    November 15, 2020

    Чо он там говорил на английском нихуя не понял
    А что он писал понял

  • sentdex
    November 15, 2020

    can I download the cc of the video?

  • sentdex
    November 15, 2020

    Where are we using here the code from the first film? What about it?

  • sentdex
    November 15, 2020

    the plylist is amazing, however i came across this issues after running part 2 and part 3 back to back..
    the y also needs to be an array, so the model.fit in part 3 can run…
    thank you once more 🙂

  • sentdex
    November 15, 2020

    sir how i appy data Augmentation when i use this loading system

  • sentdex
    November 15, 2020

    Have encountered thgis error in imshow?
    TypeError: Image data of dtype object cannot be converted to float

  • sentdex
    November 15, 2020

    'cat and raptor or something' 😀 😀

  • sentdex
    November 15, 2020

    does this work with a multiclass problem?

  • sentdex
    November 15, 2020

    My problem is i can't locate my file LOL is there any online dataset?

  • sentdex
    November 15, 2020

    I found for the Y labels you have to make it a numpy array if not the model will not take them. Other than that this is an amazing tutorial

  • sentdex
    November 15, 2020

    That was a great tutorial, Thanks a billion, However I have faced an error when I was trying to fit the model on my X and y:
    Failed to find data adapter that can handle input: <class 'numpy.ndarray'>, (<class 'list'> containing values of types {"<class 'int'>"})

    Could you please help me?

  • sentdex
    November 15, 2020

    why don't u use ImageDataGenerator

  • sentdex
    November 15, 2020

    Please make a video on how to make a dataset of images from scratch.

  • sentdex
    November 15, 2020

    I've been looking for ways to upload 40k images to my Drive for 3 days. You in one word: you are perfect

  • sentdex
    November 15, 2020

    Hi sentdex,Can you guide me how to read the image from a folder and store its image data into pandadataframe

  • sentdex
    November 15, 2020

    Thanks a lott🙏🏽🙏🏽 u just saved me

  • sentdex
    November 15, 2020

    hye can you tell me how can we make datasets of images (for face recognition in python OpenCV) by using bing search API

  • sentdex
    November 15, 2020

    I am getting an error with the x = x / 255.0
    Memory Error:

    I converted x to a numpy array before the pickle_out.close.
    Any help? I think it's because the data set is so large

  • sentdex
    November 15, 2020

    Hi Sir,
    I found you posted X, y empty lists to store the features and labels. However, I have no idea how you distinguish the corrected features and labels from training datasets. Please kindly advise me. Thanks in advance!

  • sentdex
    November 15, 2020

    Thank you man

  • sentdex
    November 15, 2020

    Great video as usual. But you need some silent fans for your computer!! 🙂

  • sentdex
    November 15, 2020

    Greate tutorial, thanks. You already know that, you did not run line 42, that is why, I think X gave error there.

  • sentdex
    November 15, 2020

    What are the things to do and consider to prepare my own data images. I'm a beginner.

  • sentdex
    November 15, 2020

    Can I do this on GoogleColab, please?

  • sentdex
    November 15, 2020

    i am going to learn image processing in python please guide me the best youtube videos for it

  • sentdex
    November 15, 2020

    "Corrupt JPEG data: 128 extraneous bytes before marker 0xd9" error shows while running create_training_data function. I can't remove that error.

  • sentdex
    November 15, 2020

    Hey great tutorial I've got a problem tho I keep getting file not found but I've named the file and put it in the directory the same way you have is there a way to fix this?

  • sentdex
    November 15, 2020

    hey can anyone help me at 3:23 , what if i don't want to convert it into gray scale, should i just skip that step?

  • sentdex
    November 15, 2020

    Python по скайпу. Научу мыслить нестандартно. Решаем задачки, строим утилиты, игры. Data Science и всё, что с эти связано. Телега у меня в контактах. Напиши мне

  • sentdex
    November 15, 2020

    7:00 its a paw!!!

Write a comment