I have some problems with code, help me please.
Im took all links from page with this func:
def get_links(link):
soup = BeautifulSoup(link, 'lxml')
block = soup.find('div', class_='block-top-important')
projects = []
for string in block.find_all('a', class_='post-preview-text post-preview-text--no-padding'):
#print(string)
projects.append ({
string.get('href')
})
return(projects)
But when I send these links in 2nd func to crawl pages it returns me error in string 4:
def parse(html):
stranica = []
for link in html:
soup = BeautifulSoup(urllib.request.urlopen(link), 'lxml')
page = soup.find(class_= 'fb-quotable')
title = page.find('h1').text
for main_text in page:
main_text.text = page.find('p', class_='align-left')
stranica.append ((title, main_text))
My main func, which send links in code:
def main():
url = 'https://www.nur.kz/'
op = get_html(url)
take_links = get_links(op)
start_parse = parse(take_links)
if __name__ == '__main__':
main()
Full error trace:
Traceback (most recent call last):
File "C:\Users\admin\AppData\Roaming\Sublime Text 3\Local\New pars.py", line 66, in <module>
main()
File "C:\Users\admin\AppData\Roaming\Sublime Text 3\Local\New pars.py", line 62, in main
start_parse = parse(take_links)
File "C:\Users\admin\AppData\Roaming\Sublime Text 3\Local\New pars.py", line 47, in parse
soup = BeautifulSoup(urllib.request.urlopen(link), 'lxml')
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 515, in open
req.timeout = timeout
AttributeError: 'set' object has no attribute 'timeout'
[Finished in 4.4s]
Okay, I decided my problem, if someone will have same question:
Error was here:
projects.append ({
string.get('href')
})
My "for" tooks links in "{}", so that's why they are "set" objects, not "string" and urllib returns error.
How I fixed it:
projects.append (string.get('href'))
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.