Reading data from a CSV file online in Python 3

Question

Just testing something out as practice. I have this huge CSV file online https://raw.github.com/datasets/gdp/master/data/gdp.csv And I want to read all the data and put it

Answer 1

You need to split the read CSV data by lines before passing it to the csv.reader():

datareader = csv.reader(webpage.read().decode('utf-8').splitlines())

The csv.reader() then takes care of the rest for you.

You could also have io.TextIOWrapper() take care of reading, decoding and line-handling for you:

import csv
import io
import urllib.request

url = "https://raw.github.com/datasets/gdp/master/data/gdp.csv"
webpage = urllib.request.urlopen(url)
datareader = csv.reader(io.TextIOWrapper(webpage))

There is little point in looping over the reader and adding rows to a list; you could just do:

data = list(datareader)

instead, but if all you want to do is print out the columns, loop directly over the reader and do so:

datareader = csv.reader(io.TextIOWrapper(webpage))
for row in datareader:
    print(row)

Either way, with splitting the lines yourself or using TextIOWrapper, the code now produces:

['Country Name', 'Country Code', 'Year', 'Value']
['Arab World', 'ARB', '1968', '32456179321.45']
['Arab World', 'ARB', '1969', '35797666653.6002']
['Arab World', 'ARB', '1970', '39062044200.4362']
['Arab World', 'ARB', '1971', '45271917893.3429']
['Arab World', 'ARB', '1972', '54936622019.8224']
['Arab World', 'ARB', '1973', '69564884441.8264']
['Arab World', 'ARB', '1974', '132123836511.468']
['Arab World', 'ARB', '1975', '147666389454.913']
['Arab World', 'ARB', '1976', '182208407088.856']
# ... etc. ...