用 python 抓取网页（'NoneType' 对象没有属性 'get_text'）

Question

I would like to extract multiple drug informations from multiple pages in https://www.medindia.net/doctors/drug_information/abacavir.htm , https://www.medindia.net/doctors/drug_information/talimogene_laherparepvec.htm , and etc

On each pages, The information that I would like to extract are as follows: General, Brands, Prescription Contraindications, Side effects, Dosage, How to Take, Warning and Storage.

By using Beautiful soup, I am able to identify the class needed for extraction. However, when i am trying to extract the information and store the information into a variable, it shows the 'NoneType' object has no attribute 'get_text' . It seems that there is no element with the class 'drug-content'. However, when I print the items it shows the class. Please help me. Below is my code:

import pandas as pd
import requests
import urllib.request
import time
from bs4 import BeautifulSoup

url = 'https://www.medindia.net/doctors/drug_information/abacavir.htm'
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
drug = soup.find(class_='mi-container__fluid')
print(drug)

# whole page contain drug content
items = drug.find_all(class_='drug-content')
print(items)

# extract drug information from drug content into individual variable
general = items[0].find(class_='drug-content').get_text(strip=True).replace("\n", "")
brand = items[1].find(class_='report-content').get_text(strip=True).replace("\n", "")
prescription = items[1].find(class_='drug-content').get_text(strip=True).replace("\n", "")
contraindications = items[2].find(class_='drug-content').get_text(strip=True).replace("\n", "")
side_effect = items[2].find(class_='drug-content').get_text(strip=True).replace("\n", "")
dosage = items[3].find(class_='drug-content').get_text(strip=True).replace("\n", "")
how_to_use = items[4].find(class_='drug-content').get_text(strip=True).replace("\n", "")
warnings = items[5].find(class_='drug-content').get_text(strip=True).replace("\n", "")
storage = items[7].find(class_='drug-content').get_text(strip=True).replace("\n", "")

I have try to change the class to 'report-content drug-widget'. However, with that class, I am unable to extract the general information. And also side-effect is unavailable for this drug. How can I put an NA into the variable if the information is not available for the drug.

# whole page contain drug content
items = drug.find_all(class_='report-content drug-widget')
print(items)

# extract drug information from drug content into individual variable
general = items.find(class_='drug-content').get_text(strip=True).replace("\n", "")
brand = items[0].find(class_='drug-content').get_text(strip=True).replace("\n", "")

Please advice how to extract the information and how can I put NA where information which I need are not available.

Answer 1

I can help you with the first one, it should help you get started on how to deal with non finds, and how to search for the pattern your looking for:

try:
  general = items[0].find('h3', attrs={'style': 'margin:0px!important'}).get_text(strip=True).replace("\n", "").replace("\xa0", " ")
except:
  general = "N/A"

You can slice the Generic Name: out since it's probably the same size for each answer by:

general = general[15:]  
print(general):
#'Abacavir'