从表中获取数据的漂亮汤

Question

I had recently asked for help using beautiful soup to grab forex prices from a site. the data was hidden in the span. I was lucky enough to get help from two people who were amazing and helped me work through it. I have since found a different site that i want to scrape from, this time there is no span the text is in tr and td from the table.

https://www.wsj.com/market-data/quotes/fx/AUDNZD/historical-prices

is the website.. as you can see the high and low prices go back i believe 30 days on this table

i would like to grab the whole table so i can use the data as needed for different calculations

when i attempt to grab the data its still just coming back as an empty list.. and i have tried alot of different places to grab it from.

Can someone not only help me get what i want but explain what im doing wrong so i can learn to use the beautiful soup for myself so i dont have to keep asking for help.

last time i grabbed from span it saved it in a list of lists that i was able to use and save as variables for differnt days and then do calculations with it. this is what i am attempting to do again.

'''import requests from bs4 import BeautifulSoup import re

result = []

URL = "https://www.wsj.com/market-data/quotes/fx/AUDNZD/historical-prices" page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

table = soup.select('cr_dataTable') print(table)'''

i did not save all my attempts at different ways i tried.. i literally got down to this super basic attempt to just try to get a response back from somewhere that im grabbing so i could then continue into breaking it down to just the text.. everything i put in that soup.select() came back empty list.. so i kinda just got to a point where i decided i must not be doing any of this right. the soup is grabbing the html though. my find_all and find() and soup.select.. nothing seemed to work or get a repsonse back.
please advise on how i am going about this wrong.. this simple code here should come back with lots of data for all the code in the table correct.. then i can go through it to grab text and grab what i want??

'''import requests from bs4 import BeautifulSoup import re

result = []

URL = "https://www.wsj.com/market-data/quotes/fx/AUDNZD/historical-prices" page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

table = soup.find('table', class_='cr_dataTable') print(table)'''

comes back none!

Answer 1

You hadn't added headers thus the request was fetching output for robots.

Full Code import requests from bs4 import BeautifulSoup import json import os

result = []
headers = {
    'user-agent':
    'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Mobile Safari/537.36',
}
r = URL = "https://www.wsj.com/market-data/quotes/fx/AUDNZD/historical-prices"
page = requests.get(URL, headers=headers)
soup = BeautifulSoup(page.content, "html.parser")
div = soup.find('div', {"id": "historical_data_table"})
table = div.find('table', {"class": "cr_dataTable"})
for i in table.findAll("tr"):
    row = i
    row = row.findAll("td")
    DATE = row[0].text
    OPEN = row[1].text
    HIGH = row[2].text
    LOW = row[3].text
    CLOSE = row[4].text
    output = {"DATE": DATE, "OPEN": OPEN,
            "HIGH": HIGH, "LOW": LOW, "CLOSE": CLOSE}
    result.append(output)
if (os.path.exists("Data.json") == False):
    f = open("Data.json", "w")
    json.dump(result, f, indent=4)
else:
    with open('Data.json', 'w') as f:
        json.dump(result, f, indent=4)

Output

[
    {
        "DATE": "12/02/22",
        "OPEN": "1.0691",
        "HIGH": "1.0709",
        "LOW": "1.0568",
        "CLOSE": "1.0602"
    },
    {
        "DATE": "12/01/22",
        "OPEN": "1.0768",
        "HIGH": "1.0792",
        "LOW": "1.0669",
        "CLOSE": "1.0692"
    },
    {
        "DATE": "11/30/22",
        "OPEN": "1.0787",
        "HIGH": "1.0813",
        "LOW": "1.0737",
        "CLOSE": "1.0783"
    },
    {
        "DATE": "11/29/22",
        "OPEN": "1.0794",
        "HIGH": "1.0820",
        "LOW": "1.0773",
        "CLOSE": "1.0788"
    },
    {
        "DATE": "11/28/22",
        "OPEN": "1.0807",
        "HIGH": "1.0815",
        "LOW": "1.0752",
        "CLOSE": "1.0792"
    },
    {
        "DATE": "11/25/22",
        "OPEN": "1.0805",
        "HIGH": "1.0822",
        "LOW": "1.0782",
        "CLOSE": "1.0804"
    },
    {
        "DATE": "11/24/22",
        "OPEN": "1.0787",
        "HIGH": "1.0819",
        "LOW": "1.0765",
        "CLOSE": "1.0797"
    },
    {
        "DATE": "11/23/22",
        "OPEN": "1.0801",
        "HIGH": "1.0837",
        "LOW": "1.0747",
        "CLOSE": "1.0781"
    },
    {
        "DATE": "11/22/22",
        "OPEN": "1.0826",
        "HIGH": "1.0838",
        "LOW": "1.0781",
        "CLOSE": "1.0804"
    },
    {
        "DATE": "11/21/22",
        "OPEN": "1.0891",
        "HIGH": "1.0891",
        "LOW": "1.0799",
        "CLOSE": "1.0828"
    },
    {
        "DATE": "11/18/22",
        "OPEN": "1.0915",
        "HIGH": "1.0934",
        "LOW": "1.0833",
        "CLOSE": "1.0849"
    },
    {
        "DATE": "11/17/22",
        "OPEN": "1.0964",
        "HIGH": "1.0981",
        "LOW": "1.0912",
        "CLOSE": "1.0917"
    },
    {
        "DATE": "11/16/22",
        "OPEN": "1.0971",
        "HIGH": "1.0997",
        "LOW": "1.0941",
        "CLOSE": "1.0963"
    },
    {
        "DATE": "11/15/22",
        "OPEN": "1.0995",
        "HIGH": "1.1002",
        "LOW": "1.0946",
        "CLOSE": "1.0975"
    },
    {
        "DATE": "11/14/22",
        "OPEN": "1.0957",
        "HIGH": "1.1015",
        "LOW": "1.0953",
        "CLOSE": "1.0994"
    },
    {
        "DATE": "11/11/22",
        "OPEN": "1.0987",
        "HIGH": "1.1046",
        "LOW": "1.0949",
        "CLOSE": "1.0965"
    },
    {
        "DATE": "11/10/22",
        "OPEN": "1.0927",
        "HIGH": "1.0992",
        "LOW": "1.0913",
        "CLOSE": "1.0986"
    },
    {
        "DATE": "11/09/22",
        "OPEN": "1.0927",
        "HIGH": "1.0975",
        "LOW": "1.0901",
        "CLOSE": "1.0929"
    },
    {
        "DATE": "11/08/22",
        "OPEN": "1.0908",
        "HIGH": "1.0928",
        "LOW": "1.0882",
        "CLOSE": "1.0919"
    },
    {
        "DATE": "11/07/22",
        "OPEN": "1.0863",
        "HIGH": "1.0977",
        "LOW": "1.0863",
        "CLOSE": "1.0910"
    },
    {
        "DATE": "11/04/22",
        "OPEN": "1.0896",
        "HIGH": "1.0960",
        "LOW": "1.0877",
        "CLOSE": "1.0909"
    },
    {
        "DATE": "11/03/22",
        "OPEN": "1.0914",
        "HIGH": "1.0937",
        "LOW": "1.0883",
        "CLOSE": "1.0898"
    },
    {
        "DATE": "11/02/22",
        "OPEN": "1.0945",
        "HIGH": "1.0957",
        "LOW": "1.0902",
        "CLOSE": "1.0913"
    },
    {
        "DATE": "11/01/22",
        "OPEN": "1.1003",
        "HIGH": "1.1033",
        "LOW": "1.0930",
        "CLOSE": "1.0944"
    },
    {
        "DATE": "10/31/22",
        "OPEN": "1.1031",
        "HIGH": "1.1348",
        "LOW": "1.0989",
        "CLOSE": "1.1004"
    },
    {
        "DATE": "10/28/22",
        "OPEN": "1.1070",
        "HIGH": "1.1084",
        "LOW": "1.1012",
        "CLOSE": "1.1032"
    },
    {
        "DATE": "10/27/22",
        "OPEN": "1.1140",
        "HIGH": "1.1154",
        "LOW": "1.1058",
        "CLOSE": "1.1072"
    },
    {
        "DATE": "10/26/22",
        "OPEN": "1.1130",
        "HIGH": "1.1176",
        "LOW": "1.1092",
        "CLOSE": "1.1133"
    },
    {
        "DATE": "10/25/22",
        "OPEN": "1.1089",
        "HIGH": "1.1122",
        "LOW": "1.1065",
        "CLOSE": "1.1111"
    },
    {
        "DATE": "10/24/22",
        "OPEN": "1.1124",
        "HIGH": "1.1124",
        "LOW": "1.1020",
        "CLOSE": "1.1085"
    },
    {
        "DATE": "10/21/22",
        "OPEN": "1.1063",
        "HIGH": "1.1102",
        "LOW": "1.1044",
        "CLOSE": "1.1077"
    },
    {
        "DATE": "10/20/22",
        "OPEN": "1.1056",
        "HIGH": "1.1094",
        "LOW": "1.1023",
        "CLOSE": "1.1062"
    },
    {
        "DATE": "10/19/22",
        "OPEN": "1.1100",
        "HIGH": "1.1107",
        "LOW": "1.1052",
        "CLOSE": "1.1055"
    },
    {
        "DATE": "10/18/22",
        "OPEN": "1.1151",
        "HIGH": "1.1210",
        "LOW": "1.1071",
        "CLOSE": "1.1101"
    },
    {
        "DATE": "10/17/22",
        "OPEN": "1.1138",
        "HIGH": "1.1193",
        "LOW": "1.1137",
        "CLOSE": "1.1161"
    },
    {
        "DATE": "10/14/22",
        "OPEN": "1.1176",
        "HIGH": "1.1191",
        "LOW": "1.1121",
        "CLOSE": "1.1151"
    },
    {
        "DATE": "10/13/22",
        "OPEN": "1.1192",
        "HIGH": "1.1215",
        "LOW": "1.1157",
        "CLOSE": "1.1163"
    },
    {
        "DATE": "10/12/22",
        "OPEN": "1.1235",
        "HIGH": "1.1244",
        "LOW": "1.1172",
        "CLOSE": "1.1188"
    },
    {
        "DATE": "10/11/22",
        "OPEN": "1.1318",
        "HIGH": "1.1328",
        "LOW": "1.1195",
        "CLOSE": "1.1237"
    },
    {
        "DATE": "10/10/22",
        "OPEN": "1.1367",
        "HIGH": "1.1370",
        "LOW": "1.1266",
        "CLOSE": "1.1317"
    },
    {
        "DATE": "10/07/22",
        "OPEN": "1.1322",
        "HIGH": "1.1376",
        "LOW": "1.1301",
        "CLOSE": "1.1358"
    },
    {
        "DATE": "10/06/22",
        "OPEN": "1.1309",
        "HIGH": "1.1355",
        "LOW": "1.1244",
        "CLOSE": "1.1327"
    },
    {
        "DATE": "10/05/22",
        "OPEN": "1.1348",
        "HIGH": "1.1381",
        "LOW": "1.1242",
        "CLOSE": "1.1308"
    },
    {
        "DATE": "10/04/22",
        "OPEN": "1.1386",
        "HIGH": "1.1426",
        "LOW": "1.1306",
        "CLOSE": "1.1349"
    },
    {
        "DATE": "10/03/22",
        "OPEN": "1.1460",
        "HIGH": "1.1460",
        "LOW": "1.1362",
        "CLOSE": "1.1388"
    },
    {
        "DATE": "09/30/22",
        "OPEN": "1.1387",
        "HIGH": "1.1444",
        "LOW": "1.1320",
        "CLOSE": "1.1439"
    },
    {
        "DATE": "09/29/22",
        "OPEN": "1.1382",
        "HIGH": "1.1417",
        "LOW": "1.1346",
        "CLOSE": "1.1350"
    },
    {
        "DATE": "09/28/22",
        "OPEN": "1.1417",
        "HIGH": "1.1495",
        "LOW": "1.1290",
        "CLOSE": "1.1385"
    },
    {
        "DATE": "09/27/22",
        "OPEN": "1.1453",
        "HIGH": "1.1466",
        "LOW": "1.1370",
        "CLOSE": "1.1419"
    },
    {
        "DATE": "09/26/22",
        "OPEN": "1.1365",
        "HIGH": "1.1465",
        "LOW": "1.1328",
        "CLOSE": "1.1454"
    },
    {
        "DATE": "09/23/22",
        "OPEN": "1.1365",
        "HIGH": "1.1378",
        "LOW": "1.1323",
        "CLOSE": "1.1373"
    },
    {
        "DATE": "09/22/22",
        "OPEN": "1.1329",
        "HIGH": "1.1373",
        "LOW": "1.1303",
        "CLOSE": "1.1366"
    },
    {
        "DATE": "09/21/22",
        "OPEN": "1.1342",
        "HIGH": "1.1363",
        "LOW": "1.1315",
        "CLOSE": "1.1334"
    },
    {
        "DATE": "09/20/22",
        "OPEN": "1.1284",
        "HIGH": "1.1365",
        "LOW": "1.1273",
        "CLOSE": "1.1347"
    },
    {
        "DATE": "09/19/22",
        "OPEN": "1.1221",
        "HIGH": "1.1295",
        "LOW": "1.1206",
        "CLOSE": "1.1289"
    },
    {
        "DATE": "09/16/22",
        "OPEN": "1.1232",
        "HIGH": "1.1256",
        "LOW": "1.1197",
        "CLOSE": "1.1218"
    },
    {
        "DATE": "09/15/22",
        "OPEN": "1.1247",
        "HIGH": "1.1261",
        "LOW": "1.1212",
        "CLOSE": "1.1233"
    },
    {
        "DATE": "09/14/22",
        "OPEN": "1.1255",
        "HIGH": "1.1255",
        "LOW": "1.1201",
        "CLOSE": "1.1239"
    },
    {
        "DATE": "09/13/22",
        "OPEN": "1.1218",
        "HIGH": "1.1259",
        "LOW": "1.1194",
        "CLOSE": "1.1223"
    },
    {
        "DATE": "09/12/22",
        "OPEN": "1.1186",
        "HIGH": "1.1240",
        "LOW": "1.1181",
        "CLOSE": "1.1225"
    },
    {
        "DATE": "09/09/22",
        "OPEN": "1.1156",
        "HIGH": "1.1215",
        "LOW": "1.1139",
        "CLOSE": "1.1212"
    },
    {
        "DATE": "09/08/22",
        "OPEN": "1.1142",
        "HIGH": "1.1157",
        "LOW": "1.1115",
        "CLOSE": "1.1151"
    },
    {
        "DATE": "09/07/22",
        "OPEN": "1.1153",
        "HIGH": "1.1181",
        "LOW": "1.1134",
        "CLOSE": "1.1141"
    },
    {
        "DATE": "09/06/22",
        "OPEN": "1.1150",
        "HIGH": "1.1173",
        "LOW": "1.1127",
        "CLOSE": "1.1152"
    },
    {
        "DATE": "09/05/22",
        "OPEN": "1.1113",
        "HIGH": "1.1167",
        "LOW": "1.1113",
        "CLOSE": "1.1153"
    }
]