解析后Python不创建csv文件 - 使用selenium和beautifulsoup

Question

My code works in two parts

1. Opens browser with selenium and adds in the details to get results from a page

2. Parse the html of the results page and write that to a csv file.

Problem The second part works only if I download the page and manually add the local url (on my computer). If I add the first part of the code, selenium opens the browser but no csv file is exported.

Things I've used to write this - Ubuntu Mate 18.04 Pycharm editor Firefox Browser

I have printed every level of the code and got the right output. However, output stops after the for loop.

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
import pandas as pd
import csv
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

os.environ["PATH"] += os.pathsep + r'/home/pierre/PycharmProjects/scraping/venv'
browser = webdriver.Firefox()
browser.get('http://karresults.nic.in/indexPUC_2019.asp')

reg = browser.find_element_by_id('reg')
reg.send_keys('738286')

sub = browser.find_element_by_class_name('btn-default')

sub.click()

url = browser.current_url

my_url = url

uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")

results = []

for record in page_soup.findAll('tr'):
    for data in record.findAll('td'):
        results = results + [data.text.replace(u'\xa0', u'').strip()]

        print(results)

        with open('myfile.csv', 'w') as f:
            for item in results:
                f.write(item + ',')

No errors on Pycharm console

Answer 1

There is no need to re-request for updated URL, you have to wait for a couple of seconds using a time module.

from bs4 import BeautifulSoup
from selenium import webdriver
import time

os.environ["PATH"] += os.pathsep + r'/home/pierre/PycharmProjects/scraping/venv'
browser = webdriver.Firefox()
browser.get('http://karresults.nic.in/indexPUC_2019.asp')

reg = browser.find_element_by_id('reg')
reg.send_keys('738286')

sub = browser.find_element_by_class_name('btn-default')

sub.click()

time.sleep(3)

soup = BeautifulSoup(browser.page_source, 'lxml')
results = []
for record in soup.find_all('tr'):
    for data in record.find_all('td'):
        results = results + [data.text.replace(u'\xa0', u'').strip()]
        with open('myfile.csv', 'w') as f:
            for item in results:
                f.write(item + ',')

csv file O/P:

Name,ANIKET ANIL BALEKUNDRI,Reg. No.,738286,ENGLISH,76,,76P,HINDI,76,,76P,Part A - TOTAL,152,PHYSICS,44,30,74P,CHEMISTRY,46,30,76P,MATHEMATICS,73,,73P,BIOLOGY,55,29,84P,Part B - TOTAL,307,GRAND TOTAL MARKS,459,FINAL RESULT,First Class,