从数据集中在线提取数据

Question

DataSet Link

I have a dataset that is available in Bupa.data as CSV at the link given above and the attributes data is given in point 7 at Bupa.name file.

https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/

I am confused how to combine both the links to create a dataframe from both the links as one contains header information and the other contain the data in csv format.

I am comfortable with python and started with the following code :

import pandas as pd
from bs4 import BeautifulSoup
from urllib.request import Request,urlopen
soup_link1 = BeautifulSoup(urlopen('https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.data'))
soup_link2 = BeautifulSoup(urlopen('https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.names'))
table_data = soup_link1.find('p')
table_header = soup2_link.find('p')

Please help further.

Answer 1

To create the code that is robust against different attribute names you can use a regular expression to get the data out of the files. In your particular case:

import pandas as pd
import numpy as np
import re
from bs4 import BeautifulSoup
from urllib.request import Request,urlopen
soup_link1 = BeautifulSoup(urlopen('https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.data'))
soup_link2 = BeautifulSoup(urlopen('https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.names'))
table_data = soup_link1.find('p')
table_header = soup2_link.find('p')

p = re.compile(r'(?<=\d\.\s)[a-z]+')
columns = p.findall(table_header.text)
data = list(i.split(',') for i in table_data.text.split('\n'))
df = pd.DataFrame(data, columns=columns).apply(pd.to_numeric, errors='ignore')

The huge problem with such data is that everything is a string so we have to do a lot of converting between string and float.

Answer 2

You can combine data from two URLs and create virtual csv file (you can then supply the created csv file to dataframe or otherwise process it):

import requests
import re
import csv
from io import StringIO # for Python2 use: from StringIO import StringIO

data_1 = requests.get('https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.data').text
data_2 = requests.get('https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.names').text

row_names = re.findall(r'\d+\.\s+([a-z]+)', data_2)
f = StringIO(','.join('"{}"'.format(v) for v in row_names) + '\n' + data_1)

cr = csv.reader(f, delimiter=',')  # cr is your created csv file

for row in cr:
    print(row)

Outputs:

['mcv', 'alkphos', 'sgpt', 'sgot', 'gammagt', 'drinks', 'selector']
['85', '92', '45', '27', '31', '0.0', '1']
['85', '64', '59', '32', '23', '0.0', '2']
['86', '54', '33', '16', '54', '0.0', '2']
['91', '78', '34', '24', '36', '0.0', '2']
['87', '70', '12', '28', '10', '0.0', '2']
['98', '55', '13', '17', '17', '0.0', '2']
['88', '62', '20', '17', '9', '0.5', '1']
['88', '67', '21', '11', '11', '0.5', '1']
['92', '54', '22', '20', '7', '0.5', '1']

...and so on.