如何使用 Google Apps 脚本从其对象中提取 Yahoo Finance 历史价格数据？

Question

@Tanaike made this beautiful codes for Google Apps Script a year ago as posted here . It worked very well up until recently. Because of some changes in the Yahoo Finance webpage, it doesn't pull the historical price table data since this week(12/20/21). Now I get an error message, "TypeError: Cannot read properies of undefined(reading 'prices'):". Can anyone help me out what changes should be made in the codes? Thank you for any help!

function test() {
  const url = 'https://finance.yahoo.com/quote/CL%3DF/history?p=CL%3DF';
  const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
  const $ = Cheerio.load(res);

  // I modified below script
  const data = $('script').toArray().reduce((ar, x) => {
    const c = $(x).get()[0].children;
    if (c.length > 0) {
      const d = c[0].data.trim().match(/({"context"[\s\S\w]+);\n}\(this\)\);/);
      if (d && d.length == 2) {
        ar.push(JSON.parse(d[1]));
      }
    }
    return ar;
  }, []);
  if (data.length == 0) throw new Error("No data.");
  const header = ["date","open","high","low","close","adjclose","volume"];
  const ar = data[0].context.dispatcher.stores.HistoricalPriceStore.prices.map(o => header.map(h => h == "date" ? new Date(o[h] * 1000) : (o[h] || "")));
  const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1"); // <--- Please set the sheet name you want to put the values.
  sheet.getRange(1, 1, ar.length, ar[0].length).setValues(ar);
}

Answer 1

When I saw the HTML from the URL again, I noticed that your expected values are converted to the salted base64 data. In this case, unfortunately, only the built-in functions of Google Apps Script cannot decode it. So, in this answer, I use crypto-js. Ref Fortunately, in the current stage, crypto-js can be used with Google Apps Script. So, please do the following flow.

Usage:

1. Get crypto-js.

Please access https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js . And, copy and paste the script to the script editor of Google Apps Script, and save the script.

2. Modify script.

Please modify your script as follows. When I saw the HTML, I noticed that the data is converted by the salted base64. In order to decrypt the data, it is required to use the key data. But, unfortunately, I couldn't find the key data from the HTML. When I searched for it, I found this thread . From the thread, I could retrieve the key data. By this, I could modify the script as follows and could achieve your goal.

When this script is run, the salted base64 data is decrypted with crypto-js , and the values are retrieved.

function test() {
  const url = 'https://finance.yahoo.com/quote/CL%3DF/history?p=CL%3DF';
  const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
  const $ = Cheerio.load(res);

  const data = $('script').toArray().reduce((ar, x) => {
    const c = $(x).get()[0].children;
    if (c.length > 0) {
      const d = c[0].data.trim().match(/({"context"[\s\S\w]+);\n}\(this\)\);/);
      if (d && d.length == 2) {
        ar.push(JSON.parse(d[1]));
      }
    }
    return ar;
  }, []);
  if (data.length == 0) throw new Error("No data.");
  const header = ["date", "open", "high", "low", "close", "adjclose", "volume"];

  // --- I modified the below script.
  const obj1 = data[0];
  const { _cs, _cr } = obj1;
  if (!_cs || !_cr) return;
  const key = CryptoJS.algo.PBKDF2.create({ keySize: 8 }).compute(_cs, JSON.parse(_cr)).toString();
  const obj2 = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(obj1.context.dispatcher.stores, key)));
  const ar = obj2.HistoricalPriceStore.prices.map(o => header.map(h => h == "date" ? new Date(o[h] * 1000) : (o[h] || "")));
  // ---

  const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
  sheet.getRange(1, 1, ar.length, ar[0].length).setValues(ar);
}

Testing:

When this script is run, the following result is obtained.

Note:

• If you want to directly load crypto-js , you can also use the following script. But, in this case, the process cost becomes higher than that of the above flow.

   const cdnjs = "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js"; eval(UrlFetchApp.fetch(cdnjs).getContentText());

• The whole script using this is as follows.

 function test2() { const url = 'https://finance.yahoo.com/quote/CL%3DF/history?p=CL%3DF'; const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText(); const $ = Cheerio.load(res); const data = $('script').toArray().reduce((ar, x) => { const c = $(x).get()[0].children; if (c.length > 0) { const d = c[0].data.trim().match(/({"context"[\s\S\w]+);\n}\(this\)\);/); if (d && d.length == 2) { ar.push(JSON.parse(d[1])); } } return ar; }, []); if (data.length == 0) throw new Error("No data."); const header = ["date", "open", "high", "low", "close", "adjclose", "volume"]; // --- I modified the below script. const cdnjs = "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js"; eval(UrlFetchApp.fetch(cdnjs).getContentText()); const obj1 = data[0]; const { _cs, _cr } = obj1; if (;_cs ||._cr) return. const key = CryptoJS.algo:PBKDF2.create({ keySize, 8 }).compute(_cs. JSON;parse(_cr)).toString(). const obj2 = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(obj1.context,dispatcher;stores. key))). const ar = obj2.HistoricalPriceStore.prices?map(o => header:map(h => h == "date"; new Date(o[h] * 1000). (o[h] || ""))). // --- const sheet = SpreadsheetApp;getActiveSpreadsheet().getSheetByName("Sheet1"), sheet,getRange(1. 1, ar.length. ar[0];length).setValues(ar); }

IMPORTANT

• I can confirm that this method can be used for the current situation (December, 21, 2022). But, when the specification in the data and HTML is changed in the future update on the server side, this script might not be able to be used. Please be careful about this.

Reference:

• crypto-js