如何使用“美丽汤”解析面糊的名称？

Question

I have had no issues grabbing three stats: hits, runs and rbi's. Here is the code I have been working with so far:

#import modules

from bs4 import BeautifulSoup
import requests, os
from selenium import webdriver

#start webdriver 

os.chdir('C:\webdrivers')
header = {'User-agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}
options = webdriver.ChromeOptions(); options.add_argument("--start- 
maximized")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://www.baseball-reference.com/leagues/MLB/2018-standard- 
batting.shtml')

#grab html

soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()

#parse three stats: rbi's, runs and hits

hits = [i.text for i in soup.find_all('td', {'data-stat': 'H'})]
runs = [i.text for i in soup.find_all('td', {'data-stat': 'R'})]
rbi = [i.text for i in soup.find_all('td', {'data-stat': 'RBI'})]

#print data

print(hits, runs, rbi)

The code above works great. When I try to grab the batter's names, however, I run into some problems. The batter's names are not parsed correctly. I would like just their first and last name if possible.

Here is what I tried:

print(soup.find_all('td', {'data-stat': 'player'}))

The batter's names are in the code but there is a lot of extra data. Also, my computer slowed down a lot when I tried this line of code. Any suggestions? Thanks in advance for any help you may offer!

Answer 1

How to get batter's names:

just request the player data directly, I found this url when I watch the page load, get player name from this url will very easy:
https://d3k2oh6evki4b7.cloudfront.net/short/inc/players_search_list.csv

How to speeder your code:

First: Using selenium to load the webdriver will cost the most part time in your code. For your grab target, I suggest you use requests directly instead selenium

Second: lxml parser will speeder than the html parser, but you should install it if you never use it, just run " pip install lxml " will help you.
installing-a-parser and summarizes the advantages and disadvantages of each parser library

for example:

import requests
from bs4 import BeautifulSoup

# start requests
target_url = 'https://www.baseball-reference.com/leagues/MLB/2018-standard-batting.shtml'
headers = {'User-agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}

page_source = requests.get(target_url, headers=headers).text

#grab html

soup = BeautifulSoup(page_source, 'lxml')

#parse three stats: rbi's, runs and hits

hits = [i.text for i in soup.find_all('td', {'data-stat': 'H'})]
runs = [i.text for i in soup.find_all('td', {'data-stat': 'R'})]
rbi = [i.text for i in soup.find_all('td', {'data-stat': 'RBI'})]

#print data

print(hits, runs, rbi)

Answer 2

The data is not in source page, please refer to this link: https://d3k2oh6evki4b7.cloudfront.net/short/inc/players_search_list.csv

this is the csv file you can directly download this file or you can fetch desired data with code as well.