Web scraping news page with a "load more"

Question

I'm trying to scrape this news website "https://inshorts.com/en/read/national" and i'm just getting the results for just the displayed articles, i need all the articles on the website which contain the word (eg."COVID-19"), and don't have to use the "load more" button.

Here's my code which gives the current articles:

import requests
from bs4 import BeautifulSoup
import pandas as pd
dummy_url="https://inshorts.com/en/read/badminton"
data_dummy=requests.get(dummy_url)
soup=BeautifulSoup(data_dummy.content,'html.parser')


urls=["https://inshorts.com/en/read/national"]
news_data_content,news_data_title,news_data_category,news_data_time=[],[],[],[]
for url in urls:
  category=url.split('/')[-1]
  data=requests.get(url)
  soup=BeautifulSoup(data.content,'html.parser')
  news_title=[]
  news_content=[]
  news_category=[]
  news_time = []
  for headline,article,time in zip(soup.find_all('div', class_=["news-card-title news-right-box"]),
                            soup.find_all('div',class_=["news-card-content news-right-box"]),
                            soup.find_all('div', class_ = ["news-card-author-time news-card-author-time-in-title"])):
    
    news_title.append(headline.find('span',attrs={'itemprop':"headline"}).string)
    news_content.append(article.find('div',attrs={'itemprop':"articleBody"}).string)
    news_time.append(time.find('span', clas=["date"]))

    news_category.append(category)
  news_data_title.extend(news_title)
  news_data_content.extend(news_content)
  news_data_category.extend(news_category)  
  news_data_time.extend(news_time)

df1=pd.DataFrame(news_data_title,columns=["Title"])
df2=pd.DataFrame(news_data_content,columns=["Content"])
df3=pd.DataFrame(news_data_category,columns=["Category"])
df4=pd.DataFrame(news_data_time, columns=["time"])
df=pd.concat([df1,df2,df3,df4],axis=1)


def name():
  a = input("File Name: ")
  return a
b = name()
df.to_csv(b + ".csv")

Answer 1

You can use this example how to simulate the clicking on Load More button:

import re
import requests
from bs4 import BeautifulSoup


url = "https://inshorts.com/en/read/national"
api_url = "https://inshorts.com/en/ajax/more_news"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"
}

# load first page:
html_doc = requests.get(url, headers=headers).text
min_news_id = re.search(r'min_news_id = "([^"]+)"', html_doc).group(1)

pages = 10  # <-- here I limit number of pages to 10
while pages:
    soup = BeautifulSoup(html_doc, "html.parser")

    # search the soup for your articles here
    # ...

    # here I just print the headlines:
    for headline in soup.select('[itemprop="headline"]'):
        print(headline.text)

    # load next batch of articles:
    data = requests.post(api_url, data={"news_offset": min_news_id}).json()
    html_doc = data["html"]
    min_news_id = data["min_news_id"]

    pages -= 1

Prints news headlines of first 10 pages:

...

Moeen has done some wonderful things in Test cricket: Root
There should be an evolution in player-media relationship: Federer
Swiggy in talks to raise over $500 mn at $10 bn valuation: Reports
Tesla investors urged to reject Murdoch, Kimbal Musk's re-election
Doctor dies on Pune-Mumbai Expressway when rolls of paper fall on his car
2 mothers name newborn girls after Cyclone Gulab in Odisha 
100 US citizens, permanent residents waiting to leave Afghanistan
Iran's nuclear programme has crossed all red lines: Israeli PM

Answer 2

@AndrejKesely can you help me with this page https://www.microplay.cl/productos/juguetes?categorias=pop , i need show to the screen the name of all Funkos. (page with 'load more' ) thanks in advance.