Python Web Scraping – Should I use Selenium, Beautiful Soup or Scrapy? [2020]
[ad_1]
In this video, you’ll learn the pros and cons of the three Python web scraping frameworks you should know – beautiful soup, selenium, and scrapy. I’ll also give you scenarios of when a certain framework is more effective than the others. Let’s jump in.
⭐ Kite is a free AI-powered coding assistant that will help you code faster and smarter. The Kite plugin integrates with all the top editors and IDEs to give you smart completions and documentation while you’re typing. We made this YouTube channel and Kite to help you be more productive: https://kite.com/download/?utm_medium=referral&utm_source=youtube&utm_campaign=kite&utm_content=selenium-vs-scrapy
JOIN our online community of people who want to level up their developer skills ►
https://www.facebook.com/groups/505658083720291/
Subscribe ► https://www.youtube.com/c/KiteHQ?sub_confirmation=1
Twitter ►https://twitter.com/kitehq
***************************************
Additional Python Resources
Web Scraping with Cats! ►
https://youtu.be/ogPMCpcgb-E
6 Python Tips and Tricks YOU Should Know ►
https://youtu.be/qEr9iRX4K0o
Best IDEs for Python ►
https://www.youtube.com/watch?v=f7vJzFgUorc
***************************************
Be sure to subscribe for tutorials, project breakdowns and more!
https://www.youtube.com/c/KiteHQ?sub_confirmation=1
STAY TUNED:
Kite ► https://kite.com/
YouTube ► https://www.youtube.com/channel/UCxVRDu9ujwOrmDxu72V3ujQ
Source
[ad_2]
Wow, the quality of the video and the editing is outstanding.
tldr
Simple project? BeautifulSoup
Complex? Scrappy
Selenium? Better not use it 😁
Love this quick video summary. content is perfect and I got exactly what I came for. Some tiny constructive feedback on the delivery would be to speak about them in the same order throughout.
0:39 – Selenium, BS, Scrapy
main content – BS, Selenium, Scrapy (best order in my opinion)
summary (4:34) – BS, Scrapy, Selenium
Keep up the great work!
Excellent video! Kudos!
Hi!
In my application I need to open a web page, fill a form and then click in a button, then get some data that will be loaded in the page. What would you recommend?
Thanks in advance
This dude looks like his birthday is on May 2nd
This is great! Thanks a lot. By the way, could you do a guide on helium? It's a wrapper for selenium but easier to use though I can't get it to work for some reason. Would appreciate a guide video/series.
How can I enter a page that has recaptcha2?
Just use pyautogui lmao
Beautiful Soup
+ User Friendly
+ Easy to Learn & Master
– Requires Dependencies
– Inefficient
Scrapy
+ Efficient
+ Portability
– Not User Friendly
Selenium
+ Versatile
+ Works well with Javascript
– Not Meant to be a Web Scraper
– Inefficient
Great content! Thanks! Quick note, maybe center yourself higher in the frame for the camera. Most people have only a little room from the top of the frame to the top of their head. When you center your head in frame as you did, it kinda just makes you look short. Like you could be 7 feet tall but that centering makes you look like a hobbit!
Anyway, keep up the great work!
I came looking for copper and found gold
btw great video as always, the edits are really cool~!
@Kite , i want to scrap 1 million records from a website and there are some javascript like some click event require, i know scrapy and selenium, so please tell me which what should i use , scrapy or selenium ?
Your video was so easy to understand and help me a lot to know which way to go
Why do you say bs4 needs dependencies and the others do not? To begin with, scrapy needs twisted, you even mention it yourself!
If inspect element not allowed for a page,how could we scrsp
nice video really informative
What's not to like about this video?
Summary: Use Scrapy if your data set will be large, else use BeautifulSoup
Amazing Content, New Sub Here!
Solid editing on this.
I actually downloaded kite as per your suggestion and am using it now with jupyterlab. It's pretty neat!
Can we use python requests library with scrapy to make post requests to api? I'm done with web scraper but stuck at api
Great tutorial, thanks for the insight, you saved me a lot of reading 😉
Very interesting video)) I like it =)