解析后Python不创建csv文件 - 使用selenium和beautifulsoup

Question

我的代码分为两部分

1. 使用selenium打开浏览器并添加详细信息以从页面获取结果

2. 解析结果页面的html并将其写入csv文件。

问题第二部分只有在我下载页面并手动添加本地URL（在我的计算机上）时才有效。 如果我添加代码的第一部分，selenium会打开浏览器但不会导出csv文件。

我曾经写过的东西 - Ubuntu Mate 18.04 Pycharm编辑器Firefox浏览器

我打印了每个级别的代码并得到了正确的输出。 但是，输出在for循环后停止。

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
import pandas as pd
import csv
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

os.environ["PATH"] += os.pathsep + r'/home/pierre/PycharmProjects/scraping/venv'
browser = webdriver.Firefox()
browser.get('http://karresults.nic.in/indexPUC_2019.asp')

reg = browser.find_element_by_id('reg')
reg.send_keys('738286')

sub = browser.find_element_by_class_name('btn-default')

sub.click()

url = browser.current_url

my_url = url

uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")

results = []

for record in page_soup.findAll('tr'):
    for data in record.findAll('td'):
        results = results + [data.text.replace(u'\xa0', u'').strip()]

        print(results)

        with open('myfile.csv', 'w') as f:
            for item in results:
                f.write(item + ',')

Pycharm控制台上没有错误

Answer 1

无需重新请求更新的URL，您必须使用时间模块等待几秒钟。

from bs4 import BeautifulSoup
from selenium import webdriver
import time

os.environ["PATH"] += os.pathsep + r'/home/pierre/PycharmProjects/scraping/venv'
browser = webdriver.Firefox()
browser.get('http://karresults.nic.in/indexPUC_2019.asp')

reg = browser.find_element_by_id('reg')
reg.send_keys('738286')

sub = browser.find_element_by_class_name('btn-default')

sub.click()

time.sleep(3)

soup = BeautifulSoup(browser.page_source, 'lxml')
results = []
for record in soup.find_all('tr'):
    for data in record.find_all('td'):
        results = results + [data.text.replace(u'\xa0', u'').strip()]
        with open('myfile.csv', 'w') as f:
            for item in results:
                f.write(item + ',')

csv文件O / P：

Name,ANIKET ANIL BALEKUNDRI,Reg. No.,738286,ENGLISH,76,,76P,HINDI,76,,76P,Part A - TOTAL,152,PHYSICS,44,30,74P,CHEMISTRY,46,30,76P,MATHEMATICS,73,,73P,BIOLOGY,55,29,84P,Part B - TOTAL,307,GRAND TOTAL MARKS,459,FINAL RESULT,First Class,