[英]How to make web-scraping faster? Django project
I'm building a web-scraping application using the Django framework.我正在使用 Django 框架构建一个网络抓取应用程序。 I need some tips on how to speed up my application.我需要一些有关如何加快我的应用程序的提示。 As of right now, it takes almost a minute to load the page just parsing through 3 urls which is a problem.截至目前,加载页面仅解析 3 个 url 需要将近一分钟,这是一个问题。 I'm going to need to run a lot faster as I want to parse through up to 10 urls on my webpage.我需要更快地运行,因为我想在我的网页上解析多达 10 个 url。 As you can see, I'm only targeting one div with my code which is why my application is running so slowly.如您所见,我的代码只针对一个 div,这就是我的应用程序运行如此缓慢的原因。 I'm thinking I could try targeting multiple divs to narrow down my "soup" but I've had difficulty with that in the past so I'm hoping to get some pointers.我想我可以尝试针对多个 div 来缩小我的“汤”范围,但我过去在这方面遇到了困难,所以我希望得到一些指导。
def stats(request):
if 'user_id' not in request.session:
return redirect('/')
this_user = User.objects.filter(id = request.session['user_id'])
this_stock = Stock.objects.filter(user_id = request.session['user_id'])
progress_dict = []
for object in this_stock:
URL = object.nasdaq_url
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
progress = soup.find_all('div', class_='ln0Gqe')
for number in progress:
progress_dict.append(number.text)
context = {
"current_user" : this_user[0].first_name,
"progress_dict": progress_dict,
"this_stock": this_stock,
}
return render(request, "nasdaq.html", context)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.