使用BS4 python进行刮擦

Question

我使用以下代码从网站上抓取数据。

from bs4 import BeautifulSoup
import urllib2
import re
for i in xrange(1,461,10):
  try:
    page = urllib2.urlopen("http://cms.onlinedemos.in/directory.php?click=n&startline={}#lst".format(i))
  except urllib2.HTTPError:
    continue
  else:
    pass
  finally:
    soup = BeautifulSoup(page)
    td1=soup.findAll('td', {'class':'comtext'})
    td2 = soup.findAll('td',{'class':'comuser'})
    td3 = soup.findAll('td',{'class':'com'})
    for td1s, td2s, td3s in zip(td1,td2,td3):
      data = [re.sub('\s+', '', text).strip().encode('utf8') for text in td1s.find_all(text=True) + td2s.find_all(text=True) + td3s.find_all(text=True)  if text.strip()]
      print ','.join(data)

我的输出是

A.T.E.EnterprisesPvt.Ltd.,,AnujBhagwati
A.T.E.Pvt.Ltd.,,AtulBhagwati
AalidhraTextileEngineersLtd.,,HansrajGondalia,Mumbai
AarBeeAssociates,Mr.Gopalsamy,022-22872245
ABCarterIndiaPvt.Ltd.,,B.B.Shetty,fort@ateindia.com
ABCCorporation,MittalPatel,Mumbai
ABCIndustrialFasteners,S.R.Sheth,022-22872245

但它应该是这样的

    A.T.E. Enterprises Pvt. Ltd.,   Anuj Bhagwati   Mumbai  022-22872245    fort@ateindia.com    

    A.T.E. Pvt. Ltd.,   Atul Bhagwati   Mumbai  022-22872245    fort@ateindia.com    

    Aalidhra Textile Engineers Ltd.,    Hansraj Gondalia    Surat   0261-2279520/30/40  aalidhra@aalidhra.com    

    Aar Bee Associates  Mr. Gopalsamy   Coimbatore  0422-2236250 / 2238560  aarbeeassociates@rediffmail.com  

所以你可以看到第一排价值Mumbai 022-22872245 fort@ateindia.com开始落在第三，第四和第五排。 它继续为所有人。 我知道我哪里出错了。

Answer 1

看一下这个页面的HTML，每行有3个columsn of class com 。 使用另一个包含30个项目的列表的10个项目的列表来压缩10个项目的列表将导致您获得的输出类型。

>>> len(td3)
30
>>> td3[0:3]
[<td class="com" width="100"></td>, <td class="com" width="160"></td>, <td class="com" width="185"></td>]
>>> td3[3:6]
[<td class="com" width="100">Mumbai</td>, <td class="com" width="160">022-22872245</td>, <td class="com" width="185">fort@ateindia.com</td>]

Answer 2

@VooDooNOFX是对的。 要相应地修改代码，请尝试以下方法：

from bs4 import BeautifulSoup
import urllib2
import re
for i in xrange(1,461,10):
  try:
    page = urllib2.urlopen("http://cms.onlinedemos.in/directory.php?click=n&startline={}#lst".format(i))
  except urllib2.HTTPError:
    continue
  else:
    pass
  finally:
    soup = BeautifulSoup(page) 
    td1=soup.findAll('td', {'class':'comtext'})    
    td2 = soup.findAll('td',{'class':'comuser'})
    td345 = soup.findAll('td',{'class':'com'})
    #for td3, td4, and td5, use slicing method: s[i:j:k] slice of s from i to j with step k
    td3 = td345[0::3]
    td4 = td345[1::3]
    td5 = td345[2::3]
    for td1s, td2s, td3s, td4s, td5s in zip(td1,td2,td3,td4,td5):
      data = [re.sub('\s+', ' ', text).strip().encode('utf8').replace(",", "") for text in td1s.find_all(text=True) + td2s.find_all(text=True) + td3s.find_all(text=True) + td4s.find_all(text=True) + td5s.find_all(text=True) if text.strip()]
      print ', '.join(data)

第一页的输出：

A.T.E. Enterprises Pvt. Ltd., Anuj Bhagwati, Mumbai, 022-22872245, fort@ateindia.com
A.T.E. Pvt. Ltd., Atul Bhagwati, Mumbai, 022-22872245, fort@ateindia.com
Aalidhra Textile Engineers Ltd., Hansraj Gondalia, Surat, 0261-2279520/30/40, aalidhra@aalidhra.com
Aar Bee Associates, Mr. Gopalsamy, Coimbatore, 0422-2236250 / 2238560, aarbeeassociates@rediffmail.com
AB Carter India Pvt. Ltd., B.B. Shetty, Mumbai, 022-66662961 / 62, shettybb@vsnl.net
ABC Corporation, Mittal Patel, Ahmedabad, 079-40068999 / 26582333, info@abctextile.com
ABC Industrial Fasteners, S.R. Sheth, Mumbai, 022-28470806 / 66923987, abc@precibolts.com
Abhishek Enterprises, N.C. Jain, Bhilwara, 01482-264250, jainabhl@sancharnet.in
Accurate Trans Heat Pvt. Ltd., Kedarmal Dargar, Surat, 0261-2397268, info@accurateindia.com