## Matplotlib Tutorial 8 – getting data from the internet

Aside from loading data from the files, another popular source for data is the internet. We can load data from the internet from a variety of ways, but, for us, we’re going to just simply read the source code of the website, then use simple splitting to separate the data.

sample code: http://pythonprogramming.net
http://hkinsley.com
http://sentdex.com
http://seaofbtc.com

Source

### Comment List

• sentdex
November 27, 2020

Unfortunately, Yahoo decided to remove their API. To handle for this, you can change the stock_price_url to the following in the code:

stock_price_url = 'https://pythonprogramming.net/yahoo_finance_replacement'

Only AAPL is supported. If you have your own source, feel free to adapt the code to suit that, or you can look into pandas for pulling data.

• sentdex
November 27, 2020

The below code may be useful for someone.

%matplotlib inline

from matplotlib.dates import bytespdate2num

import matplotlib.pyplot as plt

import numpy as np

import urllib

def graph_data(stock):

stock_price_url = 'https://pythonprogramming.net/yahoo_finance_replacement&#39;

stock_data = []

split_source = source_code.split('n')

for line in split_source:

split_line = line.split(',')

if len(split_line) == 7:

if 'Volume' not in line and 'labels' not in line:

stock_data.append(line)

delimiter=',',

unpack=True,

converters={0: bytespdate2num('%Y-%m-%d')})

plt.plot_date(date, closep, '-', label='Price')

plt.xlabel('Date')

plt.ylabel('Price')

plt.title('Interesting Graph nCheck it out')

plt.legend()

plt.show()

graph_data('TSLA')

• sentdex
November 27, 2020

How did you get the data on web looking like that? When I try to manually navigate to Teslas stocks, I can't see anywhere the option to open the data as csv

• sentdex
November 27, 2020

• sentdex
November 27, 2020

Solution for Python 3+
https://pastebin.com/KNJURDDR

• sentdex
November 27, 2020

module 'urllib' has no attribute 'request', it shows an error

• sentdex
November 27, 2020

I am trying to plot Arabic file, success to the plot but the result shows the inverted letters
my code:
import matplotlib.pyplot as plt

import numpy as np

X, Y = [], []

for line in open('example.txt', 'r',encoding="utf-8"):

values = [str(s) for s in line.split(':')]

X.append(values[0])

Y.append(values[1])

plt.plot(X, Y)

plt.show()

• sentdex
November 27, 2020

Too bad I didn't buy TSLA when this video came out. \$30/shr then, >\$1000 today.

• sentdex
November 27, 2020

Hey, how do you make new line just exactly under the end of previous line but not of its begining?

• sentdex
November 27, 2020

Hello sir, I am doing this using anaconda
since yahoo has suspended its api its being tough to run the code please help as i am being a beginner .

• sentdex
November 27, 2020

Hello sir, I am using Spyder IDE in anaconda. I could only get HTTPError: NOT FOUND. Can you help sir?

• sentdex
November 27, 2020

Thanks for posting this video, I got my code to work with the new url , but it was a bit confusing going thru the for loop. Might have to watch another time, regardless , your videos in general so far are GREAT. Thanks

• sentdex
November 27, 2020

Link is now different, which means the data is different, which mean the code in the video is wrong. I fixed all this before discovering Tomer Cna'an's comment below, but honestly, this video should either be redone, deleted, or have some sort of disclaimer upfront mentioning these issues…

• sentdex
November 27, 2020

import matplotlib.pyplot as plt

import numpy as np

import urllib

import matplotlib.dates as mdates

def graph_data(stock):

stock_price_url = 'https://pythonprogramming.net/yahoo_finance_replacement&#39;

stock_data = []

split_source = source_code.split('n')

#plt.title('MY PLOT')

plt.xlabel('x')

plt.ylabel('y')

plt.legend()

plt.show()

graph_data('TSLA')
this is my code exactly the same but when i run this its giving me a blank graph like an empty square box,help me out ples

• sentdex
November 27, 2020

My code:
import matplotlib.pyplot as plt
import numpy as np

import urllib
import matplotlib.dates as mdates

import pandas as pd

def graph_data(stock):

stock_price_url = 'https://pythonprogramming.net/yahoo_finance_replacement&#39;

stock_data = []

date = []
openP = []
highP = []
lowP = []
closeP = []
volume = []

split_source = source_code.split('n')

for line in split_source:
split_line = line.split(',')

if len(split_line) == 7:
if 'Date' not in line:
stock_data.append(line)
date.append(split_line[0])
openP.append(split_line[1])
highP.append(split_line[2])
lowP.append(split_line[3])
closeP.append(split_line[4])

volume.append(split_line[6])

date = date[:5]
highP = highP[:5]

plt.bar(date, highP, label='date vs max price', color='cyan')
plt.xlabel('date')
plt.ylabel('maximum price')
plt.legend()
plt.show()

graph_data('TSLA')

• sentdex
November 27, 2020

With the API changed this tutorial is now a mess. Please redo it or remove it. It's not beneficial in the state that it is in. Thanks.

• sentdex
November 27, 2020

how do i load the graph in an html page?

• sentdex
November 27, 2020

You can get data from Yahoo like this:

import pandas as pd

from pandas_datareader import data as wb

• sentdex
November 27, 2020

Can't you just use pandas read_csv function to make this a lot simpler?

• sentdex
November 27, 2020

thank you so much sentdex
you can use this too.
from datetime import datetime
data.get_data_yahoo('TSLA',start=datetime(2019, 4, 5),end=datetime(2019, 4, 7))

• sentdex
November 27, 2020

Upto which video I should watch in matplot to get knowledge enough to start machine learning

• sentdex
November 27, 2020

I find this rather confusing. I keep getting this error…. NameError: name 'date' is not defined
Anyone know why? Any help is appreciated.

Here's the code:
https://codeshare.io/axYEEN

• sentdex
November 27, 2020

import datetime as dt

import matplotlib.pyplot as plt
import pandas as pd

start = dt.datetime(2016,1,1)
end = dt.datetime(2018,1,1)

print(df)

• sentdex
November 27, 2020

can we use list in np.loadtxt, because in pycharm it says that 'expected a string , but got a list instead'. what should i do now?

• sentdex
November 27, 2020

For me this url i m trying to open;its not opening,saying server ip address cant be found,url=http://chartapi.finance.yahoo.com/instrument/1.0/'+stock+'/chardata;type=quote;range=10y/csv, in place of stock TSLA

• sentdex
November 27, 2020

i am trying to do the same as u did to get data from internet but i am getting an error, i am using python 2.7,what change i need to make.please help me out

• sentdex
November 27, 2020

Thanks for your video. Could you tell me please how did you move several columns to right at the same time
I am talking about plt.xlabel( ….? Thanks

• sentdex
November 27, 2020

hey,#sentdex
when the url is read the values are taken in the form of str and that one is not allowing further execution
can anyone help in resolving this………:(

• sentdex
November 27, 2020

Hi, I haven't had any problem with understanding your tutorials, but from Tutorial 8, I can't understand anything… what is 'urllib', what is 'request', what is 'urlopen' 'read()' 'decode()'.. the difficulty suddenly soared up to the advanced level from the beginner's level…

• sentdex
November 27, 2020

I wanted to know how to handle if the response in now a days in JSON. how to handle the ?like below

• sentdex
November 27, 2020

hey sentdex I know this is an old tuts, but quick question in the data we had to account for the 'values' line bc it had 6, but 'labels' line right above also has 6 and each one is a date but we didn't have to account for that line…. also if this is a csv then why does each line besides the actual values were looking for start with a title then a colon then the Csv data. KIND OF LOOKS LIKE JSON, example= values: date, close, high, low, open, volume is each line a Dict of some sort or is it JSON
—-also no matter what id do im getting a URLLIB open error. SSL cert verify failed!!!!

• sentdex
November 27, 2020

• sentdex
November 27, 2020

can some one explain in a little detail of the function bytespdate2num? I am totally lost…

• sentdex
November 27, 2020

can some one explain why after reading the data, the date is an array like this:
array([736536., 736535., 736534., …, 730124., 730123., 730122.])

Does the number represent tick?

• sentdex
November 27, 2020

Hi sentdex, thanks for the wonderful tutorial ðŸ™‚

If I may ask, in the line

where source_url = "https://pythonprogramming.net/yahoo_finance_replacement&quot;,
1) what does urllib.request.urlopen(source_url) do or return?
2) what does .read() and .decode() do respectively? why are they necessary?

Thank you!

• sentdex
November 27, 2020
• sentdex
November 27, 2020

To convert the milliseconds use: %f
# %f = milliseconds

2018-05-27 14:42:11.163881 This date should have the following format
bytespdate2num('%Y-%m-%d %H:%M:%S.%f')

• sentdex
November 27, 2020

• sentdex
November 27, 2020

First of all, great tutorial!

Since Yahoo has removed their API, could you make a video or some tutorials on how to use other currently available API, such as IEX API?

Thank you

• sentdex
November 27, 2020

thanks alot for easy explanation..And you saved lot of my time too..

• sentdex
November 27, 2020

I'm not sure what's wrong with my code.

import matplotlib.pyplot as plt
import numpy as np
import urllib
import matplotlib.dates as mdates

def bytespdate2num(fmt, encoding='utf-8'):
strconverter = mdates.strdate2num(fmt)
def bytesconverter(b):
s = b.decode(encoding)
return strconverter(s)
return bytesconverter

def graph_data(stock):

stock_price_url = 'https://pythonprogramming.net/yahoo_finance_replacement&#39;
stock_data = []
split_source = source_code.split('n')
for line in split_source:
split_line = line.split(',')
if len(split_line) == 6:
if 'values' not in line and 'labels' not in line:
stock_data.append(line)

date, closep, highp, lowp, openp, volume = np.loadtxt(stock_data,
delimiter=',',
unpack=True,
converters={0: bytespdate2num('%Y%m%d')})

plt.plot_date(date, closep)

plt.xlabel('Date')
plt.ylabel('Price')
plt.title('Intereting GraphnCheck it out')
plt.legend()
plt.show()

graph_data('AAPL')

• sentdex
November 27, 2020

This my by wrongheaded hairsplitting. I think this video title should be "Matplotlib Tutorial 8 – Getting data from the internet".

• sentdex
November 27, 2020

I've been losing my mind trying to learn Python and data. Your vids are THE BEST resource I've found so far. THANK YOU VERY MUCH.

• sentdex
November 27, 2020

you could filter out the un-needed data by checking for a number at the beginning of the line.

• sentdex
November 27, 2020
• sentdex
November 27, 2020

File "matplotlib010.py", line 33
plt.xlabel('date')
^
IndentationError: unexpected indent

I had defined 'import matplotlib.pyplot as plt' but still it gives this error.. plzzz helpme

• sentdex
November 27, 2020

Hi, Your tutorials are very useful, why not you use jupyter notebook?

• sentdex
November 27, 2020

Correct Code:
import matplotlib.pyplot as plt
import numpy as np
import urllib
from matplotlib.dates import bytespdate2num
stock_url='https://pythonprogramming.net/yahoo_finance_replacement&#39;

split_src=src.split('n')
data = []
for line in split_src:
split_line=line.split(',')
if len(split_line) ==7:
if 'Date' not in line:
data.append(line)

• sentdex
November 27, 2020

Huge thanks for great videos, I have a question that why we used decode() I didn't clearly understand…

• sentdex
November 27, 2020

Huge thanks for these videos! I found a way to pull data for this module.

1. First, you must have a current installation of pandas (check version with: import pandas as pd, pd.__version__).
3. Then, at least if you're running version 20.3, these lines will pull in the data:

df

My finished function (for this module) looked like this, and I didn't have to do all the date manipulation:

def graph_data(stock):