使用python从HTML网站上进行网页抓取股票

Question

I'm trying to scrape stock tickers from a website with a page source that looks like this:

<thead>
                            <tr>
                                <th>Company</th>
                                 <th>Symbol</th>
                                 <th>Weight</th>
                        </tr>
                    </thead>


                    <tbody>

                        <tr>
                            <td><a href="http://www.google.com/finance?q=AAPL">Apple Inc.</a></td>
                            <td><form action="/charts" method="post"> <div><input type="hidden" name="symbol" value="AAPL"/> <input type="submit" value="AAPL"/> </div></form></td>
                            <td>3.635302</td>
                        </tr>

So far , my python code (below) is only returning the name of the company ("Apple Inc.") , and the weight of 3.635 into the csv file - but I'd like to include the ticker 'AAPL' . On the website the tickers are formatted as a hyperlink- not sure how to scrape that data.

url = "http://slickcharts.com/sp500"
r = requests.get(url)
data = r.text
soup = BeautifulSoup(data, "html5lib")

table=soup.find_all('table')[0]
rows=table.find_all('tr')[1:]

data = {
    'Company' : [],
    'Symbol' : [],
    'Weight' : []
}

for row in rows:
    cols = row.find_all('td')
    data['Company'].append(cols[0].get_text())
    data['Symbol'].append(cols[1].get_text())
    data['Weight'].append(cols[2].get_text())

Answer 1

There is nothing in your cols[1].get_text()

you need data['Symbol'].append(cols[1].find('input')['value'])

Answer 2

You can get the ticker by finding the <a> tag and getting the href attribute as shown below and then splitting the link based on = would give a list with second value as the required AAPL

url = "http://slickcharts.com/sp500"
r = requests.get(url)
data = r.text
soup = BeautifulSoup(data, "html5lib")

table=soup.find_all('table')[0]
rows=table.find_all('tr')[1:]

data = {
    'Company' : [],
    'Symbol' : [],
    'Weight' : [],
    'q':[]
}
for row in rows:
    cols = row.find_all('td')
    data['Company'].append(cols[0].get_text())
    data['Symbol'].append(cols[1].get_text())
    data['Weight'].append(cols[2].get_text())
    data['q'].append(cols[0].find("a").get("href").split("=")[1])