如何转换在线 Txt 文件 padas Dataframe

Question

Im using requests and beautiful soup to navigate and download data from the Census Webpage. Im able to get the data into a result object, and if i want a soup object, but can not seem to convert it into a dataframe so that it can be appended with each of the other files. It is stored online as a .txt file.

from bs4 import BeautifulSoup
import pandas as pd
import csv
import requests 
from json import loads
from bs4.dammit import EncodingDetector 
url = 'https://www2.census.gov/econ/bps/Place/West%20Region/'
parser = 'html.parser'  # or 'lxml' (preferred) or 'html5lib', if installed
resp = requests.get(url)
http_encoding = resp.encoding if 'charset' in resp.headers.get('content-type', '').lower() else None
html_encoding = EncodingDetector.find_declared_encoding(resp.content, is_html=True)
encoding = html_encoding or http_encoding
region_soup = BeautifulSoup(resp.content, parser, from_encoding=encoding)
df = DataFrame()
for link in region_soup.find_all('a', href=True):
    links = str(link['href'])
    print(links)
    if links[-4:] == ".txt":
        result = requests.get(url + links).text
        df.append(pd.read_csv(result), ignore_index = True)

How do I convert the requests object into a dataframe, and define the column names etc

Answer 1

Off the bat, you import pandas as pd so you need use that when calling the DataFrame() method. Secondly, pandas is not parsing the text into a csv table. It would require a tad more manipulation to read in that text. Pandas can actually just read in the csv from a url though, so just do that directly.

Finally you need to store the appended dataframe. So change

df.append(pd.read_csv(result), ignore_index = True)

to

df = df.append(pd.read_csv(result), ignore_index = True)

Code:

from bs4 import BeautifulSoup
import pandas as pd
import csv
import requests 
from json import loads
from bs4.dammit import EncodingDetector 


url = 'https://www2.census.gov/econ/bps/Place/West%20Region/'
parser = 'html.parser'  # or 'lxml' (preferred) or 'html5lib', if installed
resp = requests.get(url)
http_encoding = resp.encoding if 'charset' in resp.headers.get('content-type', '').lower() else None
html_encoding = EncodingDetector.find_declared_encoding(resp.content, is_html=True)
encoding = html_encoding or http_encoding
region_soup = BeautifulSoup(resp.content, parser, from_encoding=encoding)
df = pd.DataFrame()
for link in region_soup.find_all('a', href=True):
    links = str(link['href'])
    print(links)
    if links[-4:] == ".txt":
        result = pd.read_csv(url + links)
        df = df.append(result, ignore_index = True)

Note:

You will get The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

So I'd rather:

df_list = []
for link in region_soup.find_all('a', href=True):
    links = str(link['href'])
    print(links)
    if links[-4:] == ".txt":
        result = pd.read_csv(url + links)
        df_list.append(result)
        
df = pd.concat(df_list, ignore_index=True)