Yahoo Finance-不完整的检索组件

Question

I am trying to write down a function in python to retrieve the list of components of an Index. So lets say I want to look at FTSE100 (^FTSE), I'd like to get all its components (100s of them) or even more information.

I can get more information about a components just by adding flags (see this ).

However, given the index, I can only retrieve the first 51 components (first page of this: http://finance.yahoo.com/q/cp?s=%5EFTSE&c=0 ).

My function is:

at = '%40'
def getListComponents(symbol):
    url = 'http://finance.yahoo.com/d/quotes.csv?s=%s%s&c=1&f=s' % (at, symbol)
    return urllib.urlopen(url).read().strip().strip('"')

Output example: 
'AAL.L"\r\n"ABF.L"\r\n"ADM.L"\r\n"ADN.L"\r\n"AGK.L"\r\n"AMEC.L"\r\n"ANTO.L"\r\n"ARM.L"\r\n"AV.L"\r\n"AZN.L"\r\n"BA.L"\r\n"BAB.L"\r\n"BARC.L"\r\n"BATS.L"\r\n"BG.L"\r\n"BLND.L"\r\n"BLT.L"\r\n"BNZL.L"\r\n"BP.L"\r\n"BRBY.L"\r\n"BSY.L"\r\n"BT-A.L"\r\n"CCL.L"\r\n"CNA.L"\r\n"CPG.L"\r\n"CPI.L"\r\n"CRDA.L"\r\n"CRH.L"\r\n"CSCG.L"\r\n"DGE.L"\r\n"ENRC.L"\r\n"EVR.L"\r\n"EXPN.L"\r\n"FRES.L"\r\n"GFS.L"\r\n"GKN.L"\r\n"GLEN.L"\r\n"GSK.L"\r\n"HL.L"\r\n"HMSO.L"\r\n"HSBA.L"\r\n"IAG.L"\r\n"IHG.L"\r\n"IMI.L"\r\n"IMT.L"\r\n"ITRK.L"\r\n"ITV.L"\r\n"JMAT.L"\r\n"KAZ.L"\r\n"KGF.L"\r\n"LAND.L'

This way getting parsing the components'titles is very easy.

How can I get the remaning 49 components? Take in consideration, that the components not retrieved could be even more in case I was looking at FTSE250 or higher.

THE NO ANSWER:

So I did some research, tried many combinations of flags, found and read this thread of comments: code.google.com/p/yahoo-finance-managed/wiki/csvQuotesDownload ; AND I concluded that it's not possible to download all the components of an index as CSV.

If you have/had the same problem than just use BeautifulSoup. You may not like this approach, but there's not another way.

Solution to most of my problems

Answer 1

If you're doing it that way, there's a little link at the top of the table that says last - which'll give you the last page number - http://finance.yahoo.com/q/cp?s=%5EFTSE&c=2 (from your example) then split that out to create a range range(number) to loop over and request pages similar to how you're doing at the moment.

1. Open initial page
2. Extract link via lxml.html or BeautifulSoup
3. Parse out the last page number
4. Loop over number of pages retrieving each

On a side note, I'm pretty sure Yahoo! must have an API for some of this?

Answer 2

I am new to Python and finding my feet.

I was looking for a solution to the same problem, but ended up writing my own. My code is inefficient, lengthy and ugly - but it works and I will use if rarely. I look forward to learning from someone wiser.

def getIndexComponents(symbol):

# function to retrieve the component list of equity index
# from Yahoo Finance, if available

import requests
p = 0

while p < 12:

    if p == 0:

        url = 'http://finance.yahoo.com/q/cp?s=%5E' + symbol
        text = requests.get(url).content
                                                                              # </a></b></td><td
        componentSubset = [text[n-10:n] for n in xrange(len(text)) if text.find('</a></b></td><td', n) == n]

        for comp in range(len(componentSubset)):

            componentSubset[comp] = componentSubset[comp][(1+componentSubset[comp].index('>')):]

        components = componentSubset

    else:

        url = 'http://finance.yahoo.com/q/cp?s=%5E' + symbol + '&c=' + str(p)
        text = requests.get(url).content

        componentSubset = [text[n-10:n] for n in xrange(len(text)) if text.find('</a></b></td><td', n) == n]

        for comp in range(len(componentSubset)):

            componentSubset[comp] = componentSubset[comp][(1+componentSubset[comp].index('>')):]

        components.extend(componentSubset)

    p = p + 1

components = set(components)

return components

seems to work

getIndexComponents('FTSE')