Python Web Scraping – Should I use Selenium, Beautiful Soup or Scrapy? [2020]




[ad_1]

In this video, you’ll learn the pros and cons of the three Python web scraping frameworks you should know – beautiful soup, selenium, and scrapy. I’ll also give you scenarios of when a certain framework is more effective than the others. Let’s jump in.

⭐ Kite is a free AI-powered coding assistant that will help you code faster and smarter. The Kite plugin integrates with all the top editors and IDEs to give you smart completions and documentation while you’re typing. We made this YouTube channel and Kite to help you be more productive: https://kite.com/download/?utm_medium=referral&utm_source=youtube&utm_campaign=kite&utm_content=selenium-vs-scrapy

JOIN our online community of people who want to level up their developer skills ►
https://www.facebook.com/groups/505658083720291/

Subscribe ► https://www.youtube.com/c/KiteHQ?sub_confirmation=1
Twitter ►https://twitter.com/kitehq

***************************************
Additional Python Resources

Web Scraping with Cats! ►
https://youtu.be/ogPMCpcgb-E

6 Python Tips and Tricks YOU Should Know ►
https://youtu.be/qEr9iRX4K0o

Best IDEs for Python ►
https://www.youtube.com/watch?v=f7vJzFgUorc

***************************************
Be sure to subscribe for tutorials, project breakdowns and more!
https://www.youtube.com/c/KiteHQ?sub_confirmation=1

STAY TUNED:
Kite ► https://kite.com/
YouTube ► https://www.youtube.com/channel/UCxVRDu9ujwOrmDxu72V3ujQ

Source


[ad_2]

Comment List

  • Kite
    January 16, 2021

    Wow, the quality of the video and the editing is outstanding.

  • Kite
    January 16, 2021

    tldr
    Simple project? BeautifulSoup
    Complex? Scrappy
    Selenium? Better not use it 😁

  • Kite
    January 16, 2021

    Love this quick video summary. content is perfect and I got exactly what I came for. Some tiny constructive feedback on the delivery would be to speak about them in the same order throughout.
    0:39 – Selenium, BS, Scrapy
    main content – BS, Selenium, Scrapy (best order in my opinion)
    summary (4:34) – BS, Scrapy, Selenium

    Keep up the great work!

  • Kite
    January 16, 2021

    Excellent video! Kudos!

  • Kite
    January 16, 2021

    Hi!

    In my application I need to open a web page, fill a form and then click in a button, then get some data that will be loaded in the page. What would you recommend?

    Thanks in advance

  • Kite
    January 16, 2021

    This dude looks like his birthday is on May 2nd

  • Kite
    January 16, 2021

    This is great! Thanks a lot. By the way, could you do a guide on helium? It's a wrapper for selenium but easier to use though I can't get it to work for some reason. Would appreciate a guide video/series.

  • Kite
    January 16, 2021

    How can I enter a page that has recaptcha2?

  • Kite
    January 16, 2021

    Just use pyautogui lmao

  • Kite
    January 16, 2021

    Beautiful Soup
    + User Friendly
    + Easy to Learn & Master
    – Requires Dependencies
    – Inefficient

    Scrapy
    + Efficient
    + Portability
    – Not User Friendly

    Selenium
    + Versatile
    + Works well with Javascript
    – Not Meant to be a Web Scraper
    – Inefficient

  • Kite
    January 16, 2021

    Great content! Thanks! Quick note, maybe center yourself higher in the frame for the camera. Most people have only a little room from the top of the frame to the top of their head. When you center your head in frame as you did, it kinda just makes you look short. Like you could be 7 feet tall but that centering makes you look like a hobbit!

    Anyway, keep up the great work!

  • Kite
    January 16, 2021

    I came looking for copper and found gold

    btw great video as always, the edits are really cool~!

  • Kite
    January 16, 2021

    @Kite , i want to scrap 1 million records from a website and there are some javascript like some click event require, i know scrapy and selenium, so please tell me which what should i use , scrapy or selenium ?

  • Kite
    January 16, 2021

    Your video was so easy to understand and help me a lot to know which way to go

  • Kite
    January 16, 2021

    Why do you say bs4 needs dependencies and the others do not? To begin with, scrapy needs twisted, you even mention it yourself!

  • Kite
    January 16, 2021

    If inspect element not allowed for a page,how could we scrsp

  • Kite
    January 16, 2021

    nice video really informative

  • Kite
    January 16, 2021

    What's not to like about this video?

  • Kite
    January 16, 2021

    Summary: Use Scrapy if your data set will be large, else use BeautifulSoup

  • Kite
    January 16, 2021

    Amazing Content, New Sub Here!

  • Kite
    January 16, 2021

    Solid editing on this.

  • Kite
    January 16, 2021

    I actually downloaded kite as per your suggestion and am using it now with jupyterlab. It's pretty neat!

  • Kite
    January 16, 2021

    Can we use python requests library with scrapy to make post requests to api? I'm done with web scraper but stuck at api

  • Kite
    January 16, 2021

    Great tutorial, thanks for the insight, you saved me a lot of reading 😉

  • Kite
    January 16, 2021

    Very interesting video)) I like it =)

Write a comment