Machine Learning with Synthetic Data | @Unity + @TensorFlow




[ad_1]

In this augmented reality video we are going to train our own object detection machine learning model with Tensorflow using fully synthetic data created in Unity.

#MachineLearning #Tensorflow #Unity

Github Project:
https://github.com/MatthewHallberg/SyntheticDataGenerator

Tensorflow Models install instructions:
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md

Tensorflow Models directory for 1.8.0:
https://github.com/tensorflow/models/tree/d530ac540b0103caa194b4824af353f1b073553b

Jameson Toole’s article:
https://heartbeat.fritz.ai/synthetic-data-a-bridge-over-the-data-moat-29f392a52f27

Adam Kelly’s project:
https://www.youtube.com/watch?v=3O42e4-u7hM

Videos that inspired this:
Jabrils – WRITING MY FIRST MACHINE LEARNING GAME! (1/4)
https://www.youtube.com/watch?v=ZX2Hyu5WoFg
Jabrils – I MADE BETTER AI THAN NINTENDO
https://www.youtube.com/watch?v=DGZs2xREwSU
Jabrils – Making AI App to Save Lives
https://www.youtube.com/watch?v=E-uk9DtcOS8

MEET SINGLE AR DEVELOPERS HERE (Discord):
https://discord.gg/rCfV3Bd
and here
https://www.linkedin.com/groups/10490047/

Music:
https://www.epidemicsound.com/

Follow me on:
Facebook: https://www.facebook.com/matthewchallberg
Instagram: https://www.instagram.com/matthewhallberg
Twitter: https://twitter.com/MatthewHallberg

Source


[ad_2]

Comment List

  • MatthewHallberg
    December 12, 2020

    two words: transfer learning :0

  • MatthewHallberg
    December 12, 2020

    someone knows of any tutorial

  • MatthewHallberg
    December 12, 2020

    Lol I had this exact problem with the bounds! I used basically the exact* same approach you did with the mesh sweep.

    Also, I think that's sweet how you tuck your laptop in.

  • MatthewHallberg
    December 12, 2020

    What is the name of mobile 3d images app

  • MatthewHallberg
    December 12, 2020

    Was recently asked to estimate the work need to make a mobile app that could detect specific (known) objects. I remembered watching this video a while back.
    Can some of this be combined with your previous ML video?
    Would more computer power be a partial solution to this?
    Would more lighting variations enrich your data set?

    Thanks so much for this video. You're my go-to guy for all those off the wall projects.

  • MatthewHallberg
    December 12, 2020

    Amazing!!!
    Never seen your tutorials before and I usually do not Subscribe on YouTube but you just got a new Subscribe (and a fan)

  • MatthewHallberg
    December 12, 2020

    You don't neet 20000 epochs to train the model with such less data,.. it's over fits the model and the model can not predict all the labels with equally accuracy!!!, You have to train untill the loss is stable,.. thats it!! When your detecting every pichachu in the world,.. you need different types of pichachu with different background images,….. Don't worry about the huge data.., all you need is 1000 images each!!,.. and epochs not more than 500

  • MatthewHallberg
    December 12, 2020

    Hi Matthew, I have been following your tutorials for quite a while. Iโ€™m trying to do object detection of real objects with an iPhone. I know iPhones, Unity and window donโ€™t always work well but still want to give it a try. I can use photogrammetry for real objects but Iโ€™m not sure on how to do machine training once I have these images. Any suggestions would be appreciated!

  • MatthewHallberg
    December 12, 2020

    How to place 3d model on detected object in unity3d with tensorflow.

  • MatthewHallberg
    December 12, 2020

    You need to remove the green bbox outline from images and then it will work.

  • MatthewHallberg
    December 12, 2020

    You could have used Google colab, they give 12gb ram and GPU/TPU for free. You can keep it running even if you close the browser or shut down the computer

  • MatthewHallberg
    December 12, 2020

    Hi Mattehw, How can I do to enter a augmented reality house but first put the house in a specific place and then scale it?

  • MatthewHallberg
    December 12, 2020

    great stuff bro, i was searching to make something like wanna kicks app. Do you have any idea how to track the feet and put the shoes on legs like that? i know it's AR but i don't have any insights about how to do it ๐Ÿ™

  • MatthewHallberg
    December 12, 2020

    Impressed. Keep up the good work. Appreciate your insight.

  • MatthewHallberg
    December 12, 2020

    Great video ! What apps did you use to scan object to 3d models?!

  • MatthewHallberg
    December 12, 2020

    Lol "It was so hard to figure out how to convert bounding boxes from world space to screen space…". Really?

  • MatthewHallberg
    December 12, 2020

    " i have no idea what that is, And i don't even want to know " –
    MatthewHallberg 2:30
    .
    .
    .
    .
    Much love and support for you bro , such a great content <3

  • MatthewHallberg
    December 12, 2020

    Always try to use Google Collab. It's worth it if you don't have a real monster pc. Your charmander / pikachu not being detected may be caused by over/underfitting, you should check your data for angles etc,(but that is a hellish nightmare though) and look for the recommended amount of samplesizes for training.
    And don't sweat not understanding ML. In the end almost no one goes beyond those little concepts of "its kinda in a boundary so it creates a propability for that area".
    I tried putting together a custom 4 layered CNN for image recognition and it was a fucking dumpsterfire. Informatics/mathematicians who actually make this stuff are real fucking hardcore.

  • MatthewHallberg
    December 12, 2020

    Add jarvis to the play store!!!!!

  • MatthewHallberg
    December 12, 2020

    Love the project! Starting on something similar, hopefully it will work as well.
    If you'd like to improve this, i have a few suggestions that might work:
    – If you use Keras in tensorflow and try ImageDataGenerator, it automatically does the data augmentation for you before you put it into the network. Brightness, shearing, window sizes etc.
    – Also, try dynamic learning rates such as cyclic learning rates to improve your accuracy. If you'd use tensorboard, the loss function graph could tell you if you're over- or underfitting.
    – REALLY Look at different network architectures to see which one fits your needs best. It can literally make or break your project.
    – Moreover, use class imbalance techniques to put focus on your objects that you want to detect(Which are probably wayyyy less images than your " nothing in here" class. Check your confusion matrix from your test set to assess whether the loss function you use focuses mostly on True Negatives("No, nothing in here") while you want it to focus on True positives(" Yes, Charmander is present!).
    – On stackoverflow there are a lot of loss functions optimized for this purpose.

    Good luck, thanks for the awesome and relatable video!

  • MatthewHallberg
    December 12, 2020

    Did you have the green borders in all your rendered training images (7:57)?

  • MatthewHallberg
    December 12, 2020

    Why… ๐Ÿ˜‘

  • MatthewHallberg
    December 12, 2020

    I need help with the coronavirus ar app

  • MatthewHallberg
    December 12, 2020

    Am so confused when can I start learning about ML …
    Do I need like probability theory and statistics class first after that intro to AI and read few books after that I can jump to ML

    What introductory courses would you recommend

    Side note : Iโ€™m coming from 2+ years with java and about few months with python and am learning flask now (just for the heck of it )

    Do you think am ready to start doing stuff with tenser-flow or scikit maybe ???

    And which one to start with ??

  • MatthewHallberg
    December 12, 2020

    Great job Matthew. I think u should contact Jabrils for this project since he is good at AI and ML I think. I'm learning ML this Sem can't wait to explore its uses through unity.

  • MatthewHallberg
    December 12, 2020

    I think you overfitted to your training data which may be why the first model didn't work. Cant be sure though.

  • MatthewHallberg
    December 12, 2020

    i use vuforia app to scan the 3d object and use it to create augmented reality app just like this one… i think it provide better experience and faster workflow..

  • MatthewHallberg
    December 12, 2020

    Idea: maybe try the 3d scan in a completely diffuse lighting setup. So you do not have hard shadows on the textures of your 3d scans. I think that could mess up the recognition of the shapes. Or did you try that already? Anyway , apart from the bulbosaur(?) (not a pokemon expert here) recognition seems quite good already.

  • MatthewHallberg
    December 12, 2020

    you should join the Two Minute Papers Discord!

  • MatthewHallberg
    December 12, 2020

    Great vid, too bad you have no real computer ๐Ÿ™‚

  • MatthewHallberg
    December 12, 2020

    Literally loved this …..๐Ÿ˜๐Ÿ˜

Write a comment