Loading in your own data – Deep Learning basics with Python, TensorFlow and Keras p.2
[ad_1]
Welcome to a tutorial where we’ll be discussing how to load in our own outside datasets, which comes with all sorts of challenges!
First, we need a dataset. Let’s grab the Dogs vs Cats dataset from Microsoft: https://www.microsoft.com/en-us/download/confirmation.aspx?id=54765
Text tutorials and sample code: https://pythonprogramming.net/loading-custom-data-deep-learning-python-tensorflow-keras/
Discord: https://discord.gg/sentdex
Support the content: https://pythonprogramming.net/support-donate/
Twitter: https://twitter.com/sentdex
Facebook: https://www.facebook.com/pythonprogramming.net/
Twitch: https://www.twitch.tv/sentdex
G+: https://plus.google.com/+sentdex
Source
[ad_2]
Can you make video on how to create raster(.geotiff) dataset in python
I get the following error:
ValueError: Data cardinality is ambiguous:
x sizes: 22451
y sizes: 0
Please provide data which shares the same first dimension.
İ used codes:
______________________________________________________________________________________________________________
import numpy as np
import matplotlib.pyplot as plt
import os
import cv2
from tqdm import tqdm
DATADIR = "dataset/"
CATEGORIES = ["dog", "cat"]
for category in CATEGORIES: # do dogs and cats
path = os.path.join(DATADIR,category) # create path to dogs and cats
for img in os.listdir(path): # iterate over each image per dogs and cats
img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE) # convert to array
plt.imshow(img_array, cmap='gray') # graph it
plt.show() # display!
break # we just want one for now so break
break #…and one more!
print(img_array.shape)
training_data = []
IMG_SIZE=50
def create_training_data():
for category in CATEGORIES: # do dogs and cats
path = os.path.join(DATADIR,category) # create path to dogs and cats
class_num = CATEGORIES.index(category) # get the classification (0 or a 1). 0=dog 1=cat
for img in tqdm(os.listdir(path)): # iterate over each image per dogs and cats
try:
img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE) # convert to array
new_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE)) # resize to normalize data size
training_data.append([new_array, class_num]) # add this to our training_data
except Exception as e: # in the interest in keeping the output clean…
pass
#except OSError as e:
# print("OSErrroBad img most likely", e, os.path.join(path,img))
#except Exception as e:
# print("general exception", e, os.path.join(path,img))
create_training_data()
#print(len(training_data))
import random
random.shuffle(training_data)
for sample in training_data[:10]:
print(sample[1])
X = []
y = []
y=np.array(y)
for features, label in training_data:
X.append(features)
#y.append(label)
np.array((y,label))
#print(X[0].reshape(-1, IMG_SIZE, IMG_SIZE, 1))
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1)
#X[3]
import pickle
pickle_out = open("X.pickle","wb")
pickle.dump(X, pickle_out)
pickle_out.close()
pickle_out = open("y.pickle","wb")
pickle.dump(y, pickle_out)
pickle_out.close()
pickle_in = open("X.pickle","rb")
X = pickle.load(pickle_in)
pickle_in = open("y.pickle","rb")
y = pickle.load(pickle_in)
# In[79]:
import tensorflow as tf
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten
from tensorflow.keras.layers import Conv2D, MaxPooling2D
import pickle
pickle_in = open("X.pickle","rb")
X = pickle.load(pickle_in)
pickle_in = open("y.pickle","rb")
y = pickle.load(pickle_in)
X = X/255.0
model = Sequential()
model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten()) # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(X, y, batch_size=32, epochs=3, validation_split=.1)
import numpy as np
import matplotlib.pyplot as plt
import os
import cv2
DATADIR = "E:/BHARGABH_PLC/Tensorflow Logic/PetImages/Dog"
CATEGORIES = ["Cat","Dog"]
for category in CATEGORIES:
path = os.path.join(DATADIR,category) #path to dogs or cats dir
for img in os.listdir(path):
img_array = cv2.imread(os.path.join(path,img),cv2.IMREAD_GRAYSCALE)
Dear sir why i am got this error
FileNotFoundError: [Errno 2] No such file or directory: 'E:/BHARGABH_PLC/Tensorflow Logic/PetImages/Dog/Cat'
i have download and save the file on this dir E:BHARGABH_PLCTensorflow LogicPetImages
how to download tflite file after its conversation from h5 file?
Чо он там говорил на английском нихуя не понял
А что он писал понял
can I download the cc of the video?
Where are we using here the code from the first film? What about it?
the plylist is amazing, however i came across this issues after running part 2 and part 3 back to back..
the y also needs to be an array, so the model.fit in part 3 can run…
thank you once more 🙂
sir how i appy data Augmentation when i use this loading system
Have encountered thgis error in imshow?
TypeError: Image data of dtype object cannot be converted to float
'cat and raptor or something' 😀 😀
does this work with a multiclass problem?
My problem is i can't locate my file LOL is there any online dataset?
I found for the Y labels you have to make it a numpy array if not the model will not take them. Other than that this is an amazing tutorial
That was a great tutorial, Thanks a billion, However I have faced an error when I was trying to fit the model on my X and y:
Failed to find data adapter that can handle input: <class 'numpy.ndarray'>, (<class 'list'> containing values of types {"<class 'int'>"})
Could you please help me?
why don't u use ImageDataGenerator
Please make a video on how to make a dataset of images from scratch.
I've been looking for ways to upload 40k images to my Drive for 3 days. You in one word: you are perfect
Hi sentdex,Can you guide me how to read the image from a folder and store its image data into pandadataframe
Thanks a lott🙏🏽🙏🏽 u just saved me
hye can you tell me how can we make datasets of images (for face recognition in python OpenCV) by using bing search API
I am getting an error with the x = x / 255.0
Memory Error:
I converted x to a numpy array before the pickle_out.close.
Any help? I think it's because the data set is so large
Hi Sir,
I found you posted X, y empty lists to store the features and labels. However, I have no idea how you distinguish the corrected features and labels from training datasets. Please kindly advise me. Thanks in advance!
Thank you man
Great video as usual. But you need some silent fans for your computer!! 🙂
Greate tutorial, thanks. You already know that, you did not run line 42, that is why, I think X gave error there.
What are the things to do and consider to prepare my own data images. I'm a beginner.
Can I do this on GoogleColab, please?
i am going to learn image processing in python please guide me the best youtube videos for it
"Corrupt JPEG data: 128 extraneous bytes before marker 0xd9" error shows while running create_training_data function. I can't remove that error.
Hey great tutorial I've got a problem tho I keep getting file not found but I've named the file and put it in the directory the same way you have is there a way to fix this?
hey can anyone help me at 3:23 , what if i don't want to convert it into gray scale, should i just skip that step?
Python по скайпу. Научу мыслить нестандартно. Решаем задачки, строим утилиты, игры. Data Science и всё, что с эти связано. Телега у меня в контактах. Напиши мне
7:00 its a paw!!!