python: reading file from URL

Question

What is the proper way to read text file from internet. For example text file here https://gist.githubusercontent.com/deekayen/4148741/raw/01c6252ccc5b5fb307c1bb899c95989a8a284616/1-1000.txt

Code below works but produces extra 'b in front of each word

from urllib.request import urlopen
#url = 'https://raw.githubusercontent.com/first20hours/google-10000-english/master/google-10000-english.txt'
url = 'https://gist.githubusercontent.com/deekayen/4148741/raw/01c6252ccc5b5fb307c1bb899c95989a8a284616/1-1000.txt'
#data = urlopen(url)
#print('H w')

# it's a file like object and works just like a file
l = set()
data = urlopen(url)
for line in data:  # files are iterable
    word = line.strip()
    print(word)
    l.add(word)

print(l)

Answer 1

You have to decode each byte object to unicode . For that you can use the method decode('utf-8') . Here's the code:

from urllib.request import urlopen
url = 'https://gist.githubusercontent.com/deekayen/4148741/raw/01c6252ccc5b5fb307c1bb899c95989a8a284616/1-1000.txt'

l = set()
data = urlopen(url)
for line in data:  # files are iterable
    word = line.strip().decode('utf-8') # decode the line into unicode
    print(word)
    l.add(word)

print(l)

Answer 2

It's simple using pandas. Just execute

import pandas as pd
pd.read_csv('https://gist.githubusercontent.com/deekayen/4148741/raw/01c6252ccc5b5fb307c1bb899c95989a8a284616/1-1000.txt')

and you are all set:)