我可以在JavaScript中使用从Node.js爬网吗？

Question

I'm new to javaScript and trying to crawl a website with node.js. I could check the data in console log, but want to use the data in another javaScript file. How can I fetch the data?

The problem is I've never used node.js. I do javaScript so I know how to write the code, but I don't know how the back-end or server works.

I've tried to open it in my local host but the node method (eg, require()) didn't work. I found out it's because node doesn't work in browser.(See? very new to js)

Should I use bundler or something?

The steps I thought were,

• somehow send the data as json
• somehow fetch the json data and render

Here is the crawling code file.

const axios = require("axios");
const cheerio = require("cheerio");
const log = console.log;

const getHtml = async () => {
  try {
    return await axios.get(URL);
  } catch (error) {
    console.error(error);
  }
};

getHtml()
  .then(html => {
    let ulList = [];
    const $ = cheerio.load(html.data);
    const $bodyList = $("div.info-timetable ul").children("li");


    $bodyList.each(function(i, elem) {
      ulList[i] = {
          screen: $(this).find('a').attr('data-screenname'),
          time: $(this).find('a').attr('data-playstarttime')  
        };
    });

    const data = ulList.filter(n => n.time);
    return data;
  })
  .then(res => log(res));

Could you please explain what steps should I take?

Also, it would be great if I can get understood WHY the steps are needed.

Thanks alot!

Answer 1

you can try writing your data to a JSON file and proceed, that's one way, then you can use the data as an object in any js file

const appendFile = (file, contents) =>
new Promise((resolve, reject) => {
fs.appendFile(
  file,
  contents,
  'utf8',
  err => (err ? reject(err) : resolve()),
);
 });

getHtml()
 .then(html => {
let ulList = [];
const $ = cheerio.load(html.data);
const $bodyList = $("div.info-timetable ul").children("li");


$bodyList.each(function(i, elem) {
  ulList[i] = {
      screen: $(this).find('a').attr('data-screenname'),
      time: $(this).find('a').attr('data-playstarttime')  
    };
});

const data = ulList.filter(n => n.time);
return data;
})
.then(res => {
   return appendFile('./data.json',res.toString())
}))
.then(done => {log('updated data json')});