我需要从使用框架的网站抓取数据

Question

Please, I want you to give me some orientation about a project, cause I'm lost and really dunno where to start.

I'm pretty newbie at Python, but I've already did a web scraping script to get some information from some websites, using lxml and xpath to get data trough the HTML DOM.

But now, the client presented to me a challenge...

This website is using frames where I have to get data =( And I don't know how to handle with that...

And to complicate even more, the site requires login :(

If someone could help me with some information, like where do I have to start?

Is it possible to get data from a website that shows data into frames?

Here's the web address: https://www.bulkshared.com/online-ordering

I want to point the script to the "Pantry" section, but the url don't shows the path =(

Do you recommend me which kind of script? I want to use Python, but do I have to use BS? Xpath? Selenium?

Could someone donate a little portion of your time to try to help me?

Thank you very much for your time, guys!

Answer 1

import requests
from bs4 import BeautifulSoup
import re
import csv


def Login(url):
    with requests.Session() as req:
        r = req.get(url)
        soup = BeautifulSoup(r.content, 'html.parser')
        script = soup.find("script", type="text/javascript").text
        collectionId = re.search("collectionId\":\"(.*?)\"", script).group(1)
        metaSiteId = re.search("metaSiteId\":\"(.*?)\"", script).group(1)
        svSession = re.search("svSession\":\"(.*?)\"", script).group(1)
        data = {
            'email': 'test@test.com',
            'password': 'test123',
            'collectionId': collectionId,
            'metaSiteId': metaSiteId,
            'appUrl': 'https://www.bulkshared.com/online-ordering',
            'svSession': svSession
        }
        r = req.post(
            "https://www.bulkshared.com/_api/wix-sm-webapp/member/login", data=data)
        r = req.get(
            "https://api.wixrestaurants.com/v2/organizations/5716166580714419/full").json()
        return r


def Sorter():
    data = Login("https://www.bulkshared.com/")
    with open("result.csv", 'w', newline="", encoding="UTF-8") as f:
        writer = csv.writer(f)
        writer.writerow(["Name", "Price"])
        for item in data["menu"]["items"]:
            title = item["title"]["en_AU"]
            try:
                price = item["price"]
            except:
                price = "N/A"
            try:
                description = item["description"]["en_AU"].strip()
            except:
                description = "N/A"
            writer.writerow([title, description, price])


Sorter()

Note: after I've written the code, i discovered that the API is completely public and doesn't require passing any login session info.

So you can call it directly.

import requests
import json

r = requests.get(
    "https://api.wixrestaurants.com/v2/organizations/5716166580714419/full").json()

print(json.dumps(r, indent=4))