Scraping Data Off Twitter Using Python | Twitterscraper + NLP + Data Visualization




[ad_1]

Link to Code:
https://github.com/AlexTheAnalyst/PythonCode

Scraping Data from Donald Trump’s Twitter and creating visualizations with it has never been easier! I did this project a while back just to see what the results would be and I ended up thinking it was pretty cool. This is not a project that is in my current portfolio, but it could easily be altered to be a great starter project!
____________________________________________

SUBSCRIBE!
Do you want to become a Data Analyst? That’s what this channel is all about! My goal is to help you learn everything you need in order to start your career or even switch your career into Data Analytics. Be sure to subscribe to not miss out on any content!
____________________________________________

RESOURCES:

Udemy Courses:
Python for Data Analysis and Visualization- https://bit.ly/3hhX4LX
Statistics for Data Science – https://bit.ly/37jqDbq
SQL for Data Analysts (SSMS) – https://bit.ly/3fkqEij
NLP – Natural Language Processing with Python – https://bit.ly/30w7iCB

Coursera Courses:
Database and SQL for Data Science – https://bit.ly/2zUk0zL
Data Analysis with Python – https://bit.ly/3geiVDG
Data Analysis Specialization – https://bit.ly/3dYJ6w0
Tableau Data Visualization – https://bit.ly/2AHCNPd

Favorite Books:
Python for Data Analysis Book – https://amzn.to/3dDneGq
SQL Beginner’s Guide to Managing and Analyzing With SQL Book – https://amzn.to/2X5341B

*Please note I may earn a small commission for any purchase through these links – Thanks for supporting the channel!*
____________________________________________

SUPPORT MY CHANNEL – PATREON

Patreon Page – https://www.patreon.com/AlexTheAnalyst

Every dollar donated is put back into my channel to make my videos even better. Thank you all so much for your support!
____________________________________________

Websites:
GitHub: https://github.com/AlexTheAnalyst
Where I learned the TwitterScraper Module in this video: https://www.youtube.com/watch?v=zF_Q2v_9zKY
Credit to Ken Jee.
____________________________________________

*All opinions or statements in this video are my own and do not reflect the opinion of the company I work for or have ever worked for*

Source


[ad_2]

Comment List

  • Alex The Analyst
    November 14, 2020

    Hi alex. This tutorial is very helpful for beginners and intermidiates as well.
    I follow your code, but it don't show me any output! The dataframe is empty.
    All the code is run successfully but don't be able to fetch data.
    I use windows 7 and IDE is jupyter notebook.
    So plz tell me how I can fix that?

  • Alex The Analyst
    November 14, 2020

    Hi despite using the codes you've provided, I honestly don't receive any tweets in return. My return was only "_" and I've no idea whats wrong with it.

    And I've also run into error "df = df.loc[df['screen_name'] == user]" (KeyError: 'screen_name') How do i resolve it?

  • Alex The Analyst
    November 14, 2020

    This is great stuff. Question: is it possible to build a data scraper for social media sites like Reddit and Twitter to pick up chatter on what stocks people are talking about the most just before the market opens? I’m trying to figure out if there is a way to predict which stocks will move the most based on how many people are talking about that stock during pre market hours. Thanks.

  • Alex The Analyst
    November 14, 2020

    Hi Alex, you have done a very clean and sweet presentation. And from your video, i know how people scape data from twitter now. However, sometimes, when people post a tweet, they add pictures right below to explain themselves more clearly. By using the scraping bot, we can get the text but we missed the pictures.

    So, it is possible to output the data into a word file by keeping both the text and picture in order ?

    Thanks

  • Alex The Analyst
    November 14, 2020

    Is scraping tweets legal? I have been tinkering around with web scraping and I found out that its a very debatable topic and has its own pros and cons. I just want to learn all the allowed things before I can actually do things on my own. Can I please get some information on this?

  • Alex The Analyst
    November 14, 2020

    requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  • Alex The Analyst
    November 14, 2020

    Wow this is the first time I've seen a "practical" use of Python. Just started a Udemy course and the one thing I noticed is that its hard to see what you can do with Python because nobody really shows it in the beginning. This example is really cool and made me laugh a little based on some of the words!

  • Alex The Analyst
    November 14, 2020

    …no covfefe?

  • Alex The Analyst
    November 14, 2020

    I have installed python, twitterscraper and spyder to scrape data but whenever I try to run this scripit

    (from twitterscraper.query import query_tweets

    import datetime as dt

    import panda as pd

    begin_date = dt.datetime(2010,1,1)

    end_date = dt.datetime(2020,9,1)

    limit = 5000

    lang = 'urdu'

    #user = arynewsud

    tweets = query_tweets("پاکستان تحریک انصاف ", begindate = begin_date, enddate = end_date, limit = limit, lang = lang)

    )it gives

    "No module named 'twitterscraper'"

    can any onehelp ?

  • Alex The Analyst
    November 14, 2020

    KeyError: 'screen_name' ?

  • Alex The Analyst
    November 14, 2020

    twitter changed everything, making it impossible to scrape

  • Alex The Analyst
    November 14, 2020

    twitterscraper library doesnt work now? can you confirm?

  • Alex The Analyst
    November 14, 2020

    ImportError Traceback (most recent call last)
    <ipython-input-2-fed1e59594cf> in <module>()
    1 from twitterscraper import query_tweets
    —-> 2 from twitterscraper.query import query_tweets_from_user
    3 import datetime as dt
    4 import pandas as pd
    5

    ImportError: cannot import name 'query_tweets_from_user'

    What this error mean? 🙁 The code referral is at minute 1:00

  • Alex The Analyst
    November 14, 2020

    I am getting 0 tweets

  • Alex The Analyst
    November 14, 2020

    Hi, after trying this with another tutorial and now with this video im facing the common error that i dont gather any tweets. I installed and uninstalled twitterscraper multiple times, i run on the latest version and if i open the query file it seems like the header issue is already fixed within the newest version ( date 03.09.2020). I changed time periods and limits and even users but i always get 0 tweets.The code itself runs perfecly fine. Is this an issues within the package or are thjer eany tricks on ther installation of twitetrscraper pls help

  • Alex The Analyst
    November 14, 2020

    Sorry please, I am still having issues with the code, I also keep getting a keyerror, and I dont know where to do modification for the query.py, I cant find any query.py file

  • Alex The Analyst
    November 14, 2020

    I just emailed you …. 🙂

  • Alex The Analyst
    November 14, 2020

    Thanks Alex

  • Alex The Analyst
    November 14, 2020

    I've been trying to get this to work for a couple of hours. Lots of issues. Have you tried to run it yourself lately?

  • Alex The Analyst
    November 14, 2020

    Guys, if you have trouble regarding the line "nlp = spacy.load('en_core_web_lg')" then try to run these commands in cmd
    python -m spacy download en

    python -m spacy download en_core_web_lg
    Hope this helps 🙂

  • Alex The Analyst
    November 14, 2020

    Hi Alex,
    This is awesome stuff.
    However I seem to have run into a problem.
    On my Jupyter notebook running on Python 3.8 Windows 10, it says
    'no module named twitterscraper'
    Any suggestions would be great.

    Thanks in advance

  • Alex The Analyst
    November 14, 2020

    hi ive got an error, can you kindly solve it?

    df=pd.DataFrame(t._dict_ for t in tweets)

    df = df['text']

    df

    the error message that i got
    —-> 1 df = df['text']
    KeyError: 'text'

  • Alex The Analyst
    November 14, 2020

    hello.. it's error in..
    KeyError: 'screen_name'

  • Alex The Analyst
    November 14, 2020

    I am trying to search for a particular hashtag. Any advice on that

  • Alex The Analyst
    November 14, 2020

    Thanks for sharing Alex. I am working on a school project

  • Alex The Analyst
    November 14, 2020

    I am 29 and changing my career currently. From social work to business analytics. I just finished my first year of my degree plan. My question is how long did it take you to write this code? Did you sit down and just had an understanding or did you have to stack overflow your way through it? Programming has been the most difficult part of this change for me

  • Alex The Analyst
    November 14, 2020

    Can you kindly explain how to solve it?

    ModuleNotFoundError Traceback (most recent call last)
    <ipython-input-1-86445701e3dd> in <module>
    9 import numpy as np
    10 import re
    —> 11 import spacy
    12 nlp = spacy.load('en_core_web_lg')

    ModuleNotFoundError: No module named 'spacy'

  • Alex The Analyst
    November 14, 2020

    Nice video.

    I have an article about Python data visualization with Maptlotlib in Jupyter Notebook for Absolutely Beginner.
    You can check this link

    https://medium.com/@bahrulsg/data-visualization-with-matplotlib-for-absolute-beginner-part-i-655275855ec8

    Thank you.

  • Alex The Analyst
    November 14, 2020

    Great video, thank you very much!

    I used another scraping method as I am working on a similar project right now. I will publish this on my channel in the upcoming weeks. 
    My NLP results are different from yours as I am considering more tweets (roughly 1,5k).
    I will be using some of your techniques e.g. the NLP part and the seaborn visualization. I did some amendments but
    I will definitely give you the credits in my video.

    Nevertheless if that's a problem for you, please let me know.

    Kind Regards
    Algo

  • Alex The Analyst
    November 14, 2020

    Hey Alex! Quick question. I am looking into a Data Analyst Certification program through a University. I was wondering if you had any insight on how large companies value those programs. This specific one is through Northwestern in Chicago and offers career services and portfolio building. Programs vary, but I was just hoping to get an sense of how people in corporate jobs view these certificates.

  • Alex The Analyst
    November 14, 2020

    It doesn't work. Still 0 tweets

  • Alex The Analyst
    November 14, 2020

    Hi Alex, newbie here 🙂 Your work is great. It helps me a lot. But i have some issue. I hope you can help 🙂 When i try to scrape tweets from tweetscraper; each day, it starts crawling from 12pm to 00am. Due to limitation of API i think, it blocks me somewhere in mid of day(e.g. 2pm) I need all day not only 24pm-2pm interval. Do you have any suggestion for me?
    Somebody try to fix this by changing date to time. But i cant manage to run it. (using anaconda/spyder)
    link: https://github.com/taspinar/twitterscraper/pull/258
    Thanks in advance 🙂

  • Alex The Analyst
    November 14, 2020

    Your code does NOT work dude 😒

Write a comment