I am attempting to write a webScrape to a text file and it just isn't working. All the text file says is object over and over again. Any help will be much appreciated, i am relatively new to this type of programming and i'm coming up blank!
EDIT: I now have the text file working but i'm wondering how i can to a line break after each line of data is added.
const axios = require('axios');
const cheerio = require('cheerio');
const camelCase = require('camelcase'); // added this for readale properties
const fs = require('fs')
// use async / await feature
async function scrape(url) {
// get html page
const { data } = await axios.get(url);
// convert html string to cheerio instance
const $ = cheerio.load(data);
// query all list items
return $('.tabular-data-panel > ul')
// convert cheerio collection to array for easier manipulation
.toArray()
// transform each item into proper key values
.map(list => $(list)
// query the label element
.find('.panel-row-title')
// convert to array for easier manipulation
.toArray()
// use reduce to create the object
.reduce((fields, labelElement) => {
// get the cheerio instance of the element
const $labelElement = $(labelElement);
// get the label of the field
const key = $labelElement.text().trim();
// get the value of the field
const value = $labelElement.next().text().trim();
// asign the key value into the reduced object
// note that we used camelCase() to make the property easy to read
fields[camelCase(key)] = value;
// return the object
return fields;
}, {})
);
}
async function main() {
const url = 'https://www.lseg.com/resources/1000-companies-inspire/2018-report-1000-companies-uk/search-1000-companies-uk-2018?results_per_page=100';
const companies = await scrape(url);
fs.writeFile('Output.txt', companies, (err) => {
if (err) throw err;
console.log('it/s done')
})
console.log(companies);
}
main();
the result of function scrape(url)
seems to be like an array. You can JSON.stringify
the result to make it into string. Or, you can also just use join()
// ...
const companies = await scrape(url).then(data => JSON.stringify(data))
OR
const companies = await scrape(url).then(data => data.join('\n'))
Firstly, you can find great instructions here: Javascript how to write data to file
The second try:
fs.writeFile('Output.txt', JSON.stringify(companies), (err) => {
if (err) throw err;
console.log('it/s done')
})
console.log(companies);
}
Because you can only write Strings to a text-file.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.