Profiling and optimizing your Python code | Python tricks




[ad_1]

[https://www.buymeacoffee.com/cogsci] In this video, I show how you can profile Python code using the cProfile module, and how you can use this information to optimize your code, resulting (sometimes) in massive performance improvements.

The Jupyter notebook is available from https://osf.io/upav8/

Source


[ad_2]

Comment List

  • Sebastiaan Mathôt
    November 22, 2020

    your video is very useful, thanks for the same.

  • Sebastiaan Mathôt
    November 22, 2020

    13:30 Who else pause the video and challenge yourselves? But beware of the tendency to jump into redesign the solution before profiling.

  • Sebastiaan Mathôt
    November 22, 2020

    This is super nice video, thank you sir

  • Sebastiaan Mathôt
    November 22, 2020

    I love this so much

  • Sebastiaan Mathôt
    November 22, 2020

    Careful… he is a hero 🙌

  • Sebastiaan Mathôt
    November 22, 2020

    This channel needs a couple of millions subscribers… I always come back to it to learn those marvellous tips and tricks!

  • Sebastiaan Mathôt
    November 22, 2020

    Good video but If I understand correctly the final change in code change does not account for a movie title that is duplicated more than once. So the first two iterations of the code are doing more functionality.
    All in all nice video, I learned something new watching it. so thank you for that.

  • Sebastiaan Mathôt
    November 22, 2020

    15:30 auto corection !
    Nice video BTW 😀

  • Sebastiaan Mathôt
    November 22, 2020

    Superb! Thanks

  • Sebastiaan Mathôt
    November 22, 2020

    Great explanation. Lucid and fundamental. It is indeed helpful.

  • Sebastiaan Mathôt
    November 22, 2020

    Where is the movies.txt? Please provide it, thanks.

  • Sebastiaan Mathôt
    November 22, 2020

    Man your videos are really really helpful! Best explanation of cProfile, profiling and optimization in python. Please keep posting videos…

  • Sebastiaan Mathôt
    November 22, 2020

    I like your video, and i subscribed, but is not what i was searching. Im searching for cpu and mem profiling (my actual problem is: i need to set parameters of an app, cause i need to deply that app in a cluster). Maybe next time you can specify more detailed the title of the video, than saying "profiling" is to general porpose, and be more specific. Any way good video.

  • Sebastiaan Mathôt
    November 22, 2020

    Love your shades 🤘

  • Sebastiaan Mathôt
    November 22, 2020

    Finally some optimization that is not too complex. Thanks.

  • Sebastiaan Mathôt
    November 22, 2020

    In case anyone in the future reads this
    I found that this method executes even faster than the method he ended up with.
    from collections import Counter

    def find_duplicate_words_counter(src='movies.txt'):
    return [movie for movie, count in Counter([movie.lower() for movie in read_movies(src)]).items() if count>1]

  • Sebastiaan Mathôt
    November 22, 2020

    This helped me optimize code by about 50-75% depending on the file contents being scanned. Earlier it was consuming about a minute on a large scan, while now it takes about 20 at max. The average speed is reduced to 10secs from 25secs. All I did after analyzing was change my Pandas Series objects (generated from Google Spreadsheets) to tuples (lists would also have the same effect but my data never changes in a single run).
    Using cprofiler I could see that Pandas library was consuming loads of resources just to fetch values based on an index number. Thanks a ton!!

  • Sebastiaan Mathôt
    November 22, 2020

    Real treasure… bunch of thanx

  • Sebastiaan Mathôt
    November 22, 2020

    Keep coming back here when my code starts running slow 😛

  • Sebastiaan Mathôt
    November 22, 2020

    Awesome work! I just subscribed

  • Sebastiaan Mathôt
    November 22, 2020

    Great Video…Thanks for explanation 🙂

  • Sebastiaan Mathôt
    November 22, 2020

    is there any tool to identify how much memory is consumed by the code ?

  • Sebastiaan Mathôt
    November 22, 2020

    I'll gladly give you all my money if you make a Datascience bootcamp course on Udemy.

  • Sebastiaan Mathôt
    November 22, 2020

    I used but I get this error:
    File "/usr/lib/python2.7/pstats.py", line 351, in print_stats
    print >> self.stream, indent, self.total_calls, "function calls",
    TypeError: unicode argument expected, got 'str'

  • Sebastiaan Mathôt
    November 22, 2020

    very good video! thanks!

  • Sebastiaan Mathôt
    November 22, 2020

    Awesome video! Well explained, with a simple, clear and typical hands-on example illustration.
    Great work.

  • Sebastiaan Mathôt
    November 22, 2020

    Thank you for that explanation. Neat solution with the zip and slices too! ps The link for the movies file is now out of date though.

Write a comment