承诺并将数组中的元素添加到另一个数组中的对象中

Question

The assignment is a command line node application that scrapes some data off of a specific site and saves that data to a CSV file.

I'm using scrape-it to scrape the data and successfully getting all of the data I need, but I'm struggling to figure out how to add each URL (stored in urls) into their corresponding shirts object, which is an array of objects.

Here is what I have so far.

const scrapeIt = require("scrape-it");

const mainURL = "http://shirts4mike.com/";

scrapeIt(`${mainURL}shirts.php`, {
  pages: {
    listItem: ".products li",
    name: "pages",
    data: {
      url: {
        selector: "a",
        attr: "href"
      }
    }
  }
})
  .then(({ data }) => {
    const urls = data.pages.map(page => `${mainURL}${page.url}`);
    console.log(urls);
    const shirtCalls = urls.map(url =>
      scrapeIt(url, {
        name: {
          selector: ".shirt-picture img",
          attr: "alt"
        },
        image: {
          selector: ".shirt-picture img",
          attr: "src"
        },
        price: {
          selector: "span.price"
        }
      })
    );
    return Promise.all(shirtCalls);
  })
  .then(shirtResults => {
    const shirts = shirtResults.map(shirtResult => shirtResult.data);
    console.log(shirts);
  });

So the output that "shirts" gives me is,

[ { name: 'Logo Shirt, Red',
    image: 'img/shirts/shirt-101.jpg',
    price: '$18' },
  { name: 'Mike the Frog Shirt, Black',
    image: 'img/shirts/shirt-102.jpg',
    price: '$20' },
  { name: 'Mike the Frog Shirt, Blue',
    image: 'img/shirts/shirt-103.jpg',
    price: '$20' },
  { name: 'Logo Shirt, Green',
    image: 'img/shirts/shirt-104.jpg',
    price: '$18' },
  { name: 'Mike the Frog Shirt, Yellow',
    image: 'img/shirts/shirt-105.jpg',
    price: '$25' },
  { name: 'Logo Shirt, Gray',
    image: 'img/shirts/shirt-106.jpg',
    price: '$20' },
  { name: 'Logo Shirt, Teal',
    image: 'img/shirts/shirt-107.jpg',
    price: '$20' },
  { name: 'Mike the Frog Shirt, Orange',
    image: 'img/shirts/shirt-108.jpg',
    price: '$25' } ]

But what I am trying to get the final result to look like is....

[ { name: 'Logo Shirt, Red',
    image: 'img/shirts/shirt-101.jpg',
    price: '$18',
    url: 'http://shirts4mike.com/shirt.php?id=101' //which is at urls[0]
  },
  { name: 'Mike the Frog Shirt, Black',
    image: 'img/shirts/shirt-102.jpg',
    price: '$20',
    url: 'http://shirts4mike.com/shirt.php?id=102' //urls[1]
  }, //...etc etc

Hopefully that all makes sense, still very new to promises (and node) so I'm feeling a bit out of my depth. Thank you in advance!

Answer 1

Try something like this:

const scrapeIt = require("scrape-it");

const mainURL = "http://shirts4mike.com/";

scrapeIt(`${mainURL}shirts.php`, {
  pages: {
    listItem: ".products li",
    name: "pages",
    data: {
      url: {
        selector: "a",
        attr: "href"
      }
    }
  }
})
  .then(({ data }) => {
    const urls = data.pages.map(page => `${mainURL}${page.url}`);
    console.log(urls);
    return urls.map(async (url) => {
      let urlObj = await scrapeIt(url, {
        name: {
          selector: ".shirt-picture img",
          attr: "alt"
        },
        image: {
          selector: ".shirt-picture img",
          attr: "src"
        },
        price: {
          selector: "span.price"
        }
      });

     return {...urlObj.data, url};
   });
  })
  .then(shirtResults => {
    console.log(shirtResults);
  });

Answer 2

So I actually managed to get it working, thanks to a suggestion another user made (though I think they deleted their comment?). In the final .then(), I mapped over shirts, took the pageID from the image property, and interpolated the mainURL, the path, and finally the pageID in a template literal, and added that key/value into each object. Also used this as an opportunity to store the full image url in the image property.

  .then(shirtResults => {
    const shirts = shirtResults.map(shirtResult => shirtResult.data);
    shirts.map(shirt => {
      let pageID = shirt.image.replace(/\D/g, "");
      shirt.url = `${mainURL}shirt.php?id=${pageID}`;
      shirt.image = shirt.image.replace(/^/, `${mainURL}`);
    });
    console.log(shirts);
  });

Thanks for the help!