python web抓取和excel人口

Question

I am relatively new to programming and completely new to stack overflow. I thought a good way to learn would be with a python & excel based project, but am stuck. My plan was to scrape a website of addresses using beautiful soup look up the zillow estimates of value for those addresses and populate them into tabular form in excel. I am unable to figure out how to get the addresses (the html on the site I am trying to scrape seems pretty messy), but was able to pull google address links from the site. Sorry if this is a very basic question, any advice would help though:

from bs4 import BeautifulSoup

from urllib.request import Request, 
urlopen

import re

import pandas as pd

req = Request("http://www.tjsc.com/Sales/TodaySales")
html_page = urlopen(req)

soup = BeautifulSoup(html_page, "lxml")

count = 0
links = []
for link in soup.findAll('a'):
    links.append(link.get('href'))
    count = count +1

print(links)
print("count is", count)

po = links

pd.DataFrame(po).to_excel('todaysale.xlsx', header=False, index=False)

Answer 1

you are on the right track. Instead of 'a', you need to use different html tag 'td' for the rows. Also 'th' for column names. here is one way to implement it. list_slide function converts each 14 elements to one row since the original table has 14 columns.

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

url = "http://www.tjsc.com/Sales/TodaySales"
r = requests.get(url, verify=False)
text = r.text
soup = bs(text, 'lxml')

# Get column headers from the html file
header = []
for c_name in  soup.findAll('th'):
    header.append(c_name)
# clean up the extracted header content
header = [h.contents[0].strip() for h in header]

# get each row of the table
row = []
for link in soup.find_all('td'):
    row.append(link.get_text().strip())

def list_slice(my_list, step):
"""This function takes any list, and divides it to chunks of size of "step"
"""
   return [my_list[x:x + step] for x in range(0, len(my_list), step)]

# creating the final dataframe
df = pd.DataFrame(list_slice(row, 14), columns=header[:14])