Cannot extract text from webpage using beautifulsoup with python

Question

I am fairly new to webscraping and HTML text extraction. I want to extract text from this segment of the HTML code

I want to extract

"I have open my account...."

and put it in my code's review variable.

 import requests from bs4 import BeautifulSoup import csv URL = "https://www.mouthshut.com/product-reviews/HDFC-Bank-reviews-925004501" r = requests.get(URL) soup = BeautifulSoup(r.content, 'html5lib') reviews = [] # a list to store reviews table = soup.find('div', attrs={'class': 'container box-module ofinherit'}) for row in table.findAll('div', attrs={'class': 'row review-article'}): review = {'Review_Title': row.a.text, 'URL': row.a['href'], 'Review': row.p.text} reviews.append(review) print(reviews)

When I run the above code, Review remains empty. Would really appreciate help in this case. Thank You

Answer 1

After inspecting the link you provided, I believe this is the easiest solution

import requests
from bs4 import BeautifulSoup
import csv

URL = "https://www.mouthshut.com/product-reviews/HDFC-Bank-reviews-925004501"
r = requests.get(URL)
soup = BeautifulSoup(r.content, 'html5lib')
reviews = []  # a list to store reviews
# Use a CSS selector to extract all the review containers
review_divs = soup.select('div.col-10.review')
for element in review_divs :
    review = {'Review_Title': element .a.text, 'URL': element .a['href'], 'Review': element .find('div', {'class': ['more', 'reviewdata']}).text.strip()}
    reviews.append(review)

print(reviews)

Output:

[{'Review_Title': 'Worst internet banking',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-omnppuupuoo',
  'Review': "I have open my account in HDFC bank for more than a year now. The account opening procedure were simple and easy. No headache at all. Every thing was good on the bank untill I found that I can no more login to my internet banking.\n\nI generally don't use the phone banking app of HDFC but I will...Read More"},
 {'Review_Title': 'Very poor service in customer care and bank depart',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-ssrupmqouoo',
  'Review': 'Very poor service in customer care and bank department\n\nThe bank has deducted surplus charges from my bank accounts and no one is responding, this is the second time this has happened to me.\n\nI am not at all satisfied with the bank and the services of the bank\n\n...Read More'},
 {'Review_Title': 'Home Loan Not showing outstanding Principal Amount',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-suqroqlouoo',
  'Review': "After taking home loan from HDFC ltd and paying continuously emi for two years, I tried to check the outstanding principal amout but didn't find any way to get this not in website or mobile app.\n\nIn their website most of the places they have mentioned about the total loan taken but no where th...Read More"},
 {'Review_Title': 'HDFC, another name of froud.',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-utptqrpsqtn',
  'Review': 'I have a terrible experience with HDFC bank to share with you.\n\nWhen I take the credit card, HDFC agent came to my office and told me that they charge an annual fee of 1200 which is applicable only if we spent less than 30k a year.\n\nLater I received the card realized that it was 2.5 lacks an...Read More'},
 {'Review_Title': 'Pathetic,disgusting behaviour by HDFC branch staff',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-oppoqppntsn',
  'Review': 'I never thought I would post this but then I had no choice left.\n\nMy Father holds an account with HDFC Bank in Najibabad Uttar Pradesh Branch.\n\nI visited the branch on April 2nd, 2019 at around 11:15 AM in order to withdraw cash via cheque.\n\nThere was a long queue of customers waiting, I j...Read More'},
 {'Review_Title': 'Without intimation holding my saving account',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-tptomtuopsn',
  'Review': 'This is the worst experience I have been through in any of the banks so far.\n\nI am customer with HDFC with last 7 years with holding a saving account, credit card, policy, car loan and many services.\n\ninfact, in records I am a classic customer with bank wherein, my personal relationship mana...Read More'},
 {'Review_Title': 'Exploiting retail customers to please big lenders',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-upqpllslpsn',
  'Review': 'Relationship Managers are of no help. They are there to sell you the schemes which benefit the bank at your cost. They have no idea about the products that they are selling, nor do they have any idea about your customer profile or in one case even your account details\n\nThere is no direct conta...Read More'},
 {'Review_Title': 'Extremely poorly managed bank and total chaos',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-momrsnpomqn',
  'Review': 'I have\xa0\xa0very bad experience recently with their Opp Mumbai University branch, I am senior citizen and when approached for placing some fixed deposits which I wanted to shift from PS banks due to their poor performance and I was asked to open saving account and I did open two saving account one f...Read More'},
 {'Review_Title': 'Very bad management team',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-mrmsosmmuoo',
  'Review': 'One of the worst bank...team not work properly....unsikked team members....also just sitting in under air conditioning and talking....I wish I could take 0 star this hdfc bank. Me and my all family soon leave this bank and go the another bank....to much unacceptable delay work....\n\nShame on yo...Read More'},
 {'Review_Title': 'Hdfc home loan department',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-morquqsotoo',
  'Review': 'Since a week I am trying to contact hdfc bank as well as drafted mail to all hdfc sight regarding ROI revision issue but shame on none of them replied. Instead every day they call me to offer a loan. Once u r existing customer ur value becomes zero.\n\nPathetic service experience.\n\n...Read More'},
 {'Review_Title': 'Worthless Bank',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-mlqsqpquooo',
  'Review': 'HDFC is one of the worthless bank which I have ever come across in my lifetime.HDFC is having limited free transactions monthly, after that they will charge a fee for every transaction.The customer employee ratio of HDFC bank is also very low.They charges high amount of interest rate loan with n...Read More'},
 {'Review_Title': 'This bankk is more dangerous thennn corona viruss',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-osrrorqqloo',
  'Review': 'Everyone who is viewing my review that if you were planning a new account in hdfc bank then you will be thinking once and today I will share my story in which I suffered very muchhhhhh\n\nWhen I installed amazon then the app suggested me to open an account and start net banking so I searched on ...Read More'},
 {'Review_Title': 'Bad services for everything',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-usltrnroqno',
  'Review': 'A employee of HDFC bank call me and abuses me a lot to pay my credit card EMI. She knows my all details how much of EMI and total EMI is there. When I complaint in HDFC bank for that they mail me that there is no such call from our side. Give us voice recording as a proof. And next day they redu...Read More'},
 {'Review_Title': 'Customer care experience',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-tunmutqltmo',
  'Review': 'This is the worst ever experience in my life. I missed my debit card and they are asking me to wait for 30 days as my old address is reflecting in their record. How can a person survive without money?! Even I tried to change my new temporary address and it asks for proof. What about working peop...Read More'},
 {'Review_Title': 'Highly disappointed with HDFC Bank..',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-noortlltqmo',
  'Review': 'I have a salary account with HDFC Bank. I first selected HDFC Bank because they provide all types of facilities in the kit. Such as a Master card, Net banking, etc. However after few days of opening the account, I received a call from the credit card department. They suggested me to apply for a ...Read More'},
 {'Review_Title': 'Home loan Balance Transfer',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-msosmpnnqmo',
  'Review': 'Dont trust HDFC sales persons. They commit something after the processing of application and processing fee deduction, they will come up with hidden charges to disburse the loan. HDFC processing team will not take any responsiblity even they are front office customer service. If we raise concern...Read More'},
 {'Review_Title': 'HDFC Bank Worst Service',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-urrtrrttpmo',
  'Review': 'I would like to inform about the worst service that HDFC bank provides.\n\nI have an HDFC credit card and I wanted to change the address for it, so I requested for the address change via my registered email ID and the reply was to fill an address change form and drop it in the HDFC bank.\n\nI we...Read More'},
 {'Review_Title': 'Not satisfied with the service at all.',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-prsnssnmqlo',
  'Review': 'Hi guys I just want to say some about hdfc bank as I am not satisfied with this service. Because whenever I am withdrawing my amount from atm so at every transaction they people are debiting a higher amount from my account which is very very bad and unsatisfied service so I will give only 1 star...Read More'},
 {'Review_Title': 'WORST NETBANKING SITE EVER!!',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-mlppmstomlo',
  'Review': "I am really ruing the fact that hdfc became my new and primary bank due to a change in jobs. I was with ICICI for 9 years and net banking was so awesome. Compared to that, this is like some shoddy site by rookies. It's the worst. What it covers up as superb security is nothing but inefficient ba...Read More"},
 {'Review_Title': 'For bank',
  'URL': 'https://www.mouthshut.com/review/HDFC-Bank-review-qmrnosqnmlo',
  'Review': "It was my worse experience with hdfc bank. If they compared the area in Rajkot which is good and which is bad than they never open the bank in Rajkot. Because they divided the area in local and modern area and they never give the facilities to local area. All people are not always same. And it's...Read More"}]