Javascript 是否可以生成 Cheerio 无法提取的 DOM html？

Question

I am trying to extract the price from this webpage: https://www.allbirds.com/products/mens-wool-runner-up-mizzles-natural-grey?size=13

I narrowed it down to these divs:

<div class="jsx-3947815802 Container">
<div class="jsx-526902087 Grid">
<div class="jsx-2943457050 Grid__cell Grid__cell--small-12 Grid__cell--medium-7 Grid__cell--large-up-8">...

The jsx-{random_number} for the class names is suspicious to me. They seem generated on the fly. The price I need is inside divs like these. However, these don't exist in the page source and or the cheerio object I am using during runtime. It just disappears.

How common is this technique? It seems like a pretty good way to stop web scrapers. How do I get around it?

Answer 1

If those classes are random, it might be annoying, but it's not a deal-breaker, because the other classes look to be static.

For example, the element that includes the price looks something like:

<p class="jsx-3188494938 Paragraph PdpMasterProductDetails__paragraph">$135</p>

The PdpMasterProductDetails__paragraph does not change. So, you can retrieve the text by using that as a selector:

$('.PdpMasterProductDetails__paragraph').text()

You can also retrieve the price from a meta tag:

<meta property="og:price:amount" content="135">

which can be selected via the selector string:

meta[property="og:price:amount"]

Answer 2

How common is this technique?

Very.

Building websites as Single Page Applications with tools like React is very common.

It seems like a pretty good way to stop web scrapers.

It isn't.

How do I get around it?

Hit the web service the React code fetches the raw data from directly. It's easily discoverable via the Network tab in the browser's developer tools.