从 HTML 提取 href 元素无法使用 node.js
[英]Extract href from HTML Element not working using node.js
问题描述
I'm trying to scrape data from that website: https://www.gelbeseiten.de/Suche/Fotografen/Berlin and for some reason I can't get a specific element while the others just work fine.
我正在尝试从该网站上抓取数据： https://www.gelbeseiten.de/Suche/Fotografen/Berlin ，由于某种原因，我无法获得特定元素，而其他元素则可以正常工作。 I use node.js and puppeteer.我使用 node.js 和 puppeteer。 I need the URL of the element with the class .contains-icon-homepage and I get an error message targeting it.我需要元素的 URL 和 class .contains-icon-homepage并且我收到针对它的错误消息。 If I target the .contains-icon-aktualisieren element above, then it works.如果我以上面的.contains-icon-aktualisieren元素为目标，那么它可以工作。
The error message:错误信息：
UnhandledPromiseRejectionWarning: Error: Evaluation failed: TypeError: Cannot read property 'getAttribute' of null UnhandledPromiseRejectionWarning：错误：评估失败：TypeError：无法读取 null 的属性“getAttribute”
The HTML: HTML：
<article class="mod mod-Treffer" data-teilnehmerid="1057046049">
<a href="https://www.gelbeseiten.de/gsbiz/64195b36-063f-4401-8536-0d1c71a76326" data-realid="64195b36-063f-4401-8536-0d1c71a76326" data-tnid="1057046049" target="_self"
>
<div class="mod-hervorhebung">
</div>
<picture class="trefferlisten_logo">
<source media="(min-width: 768px)" srcset="https://ies.v4all.de/0122/GS/0001/7/5017/30015017_310x190.png" />
<img alt="" data-lazy-src="https://ies.v4all.de/0122/GS/0001/7/5017/30015017_310x190.png"/>
</picture>
<h2 data-wipe-name="Titel">Rudolph Silke</h2>
<p class="d-inline-block mod-Treffer--besteBranche">
Fotografen und Fotostudios
</p>
<address class="mod mod-AdresseKompakt">
<p data-wipe-name="Adresse">
Samariterstr. 33,
<span class="nobr">
10247
Berlin
</span>
(Friedrichshain)
<span class="mod-AdresseKompakt__entfernung" title="Entfernung ab Suchmittelpunkt">6 km</span>
</p>
<p class="mod-AdresseKompakt__phoneNumber" data-hochgestellt-position="end" data-wipe-name="Kontaktdaten">(030) 4 26 66 86</p>
</address>
<div class="oeffnungszeit_kompakt__zustandsinfo--geschlossen">
<span>Geschlossen</span>,
<span class="nobr">öffnet Samstag um 10:00</span>
</div>
</a>
<div class="aktionsleiste_kompakt">
<div class="mod-gsSlider mod-gsSlider--noneOnWhite">
<span
class="mod-gsSlider__arrow mod-gsSlider__arrow--arrow" data-direction="left" data-show="false" data-wipe="{"listener":"click","name":"Trefferliste: Aktionleiste-button-links"}"></span>
<span
class="mod-gsSlider__arrow mod-gsSlider__arrow--arrow" data-direction="right" data-show="false" data-wipe="{"listener":"click","name":"Trefferliste: Aktionleiste-button-rechts"}"></span>
<div class="mod-gsSlider__slider">
<a
class="contains-icon-aktualisieren gs-btn"
rel="noopener"
href="https://www.gelbeseiten.de/gsbiz/64195b36-063f-4401-8536-0d1c71a76326#aktuelleinformationen"
data-wipe="{"listener": "mouseup", "name": "Trefferliste Actionbutton Aktualisieren", "id": "1057046049", "synchron": false}" data-isNeededPromise="false" data-cookieinfo="64195b36-063f-4401-8536-0d1c71a76326=1057046049"
>Aktualisieren</a>
<a
class="contains-icon-homepage gs-btn"
target="_blank"
rel=" noopener"
href="http://www.fotoherz.de"
data-wipe="{"listener":"click", "name":"Trefferliste Webseite-Button", "id":"1057046049"}" data-isNeededPromise="false"
>Webseite</a>
<a
class="contains-icon-email gs-btn"
href="mailto:kontakt@silke-rudolph.de?subject=Anfrage%20%C3%BCber%20Gelbe%20Seiten"
data-wipe="{"listener":"click", "name":"Trefferliste Email-Button", "id":"1057046049"}" data-isNeededPromise="false"
>E-Mail</a>
<span
class="contains-icon-route_finden gs-btn"
data-wipe="{"listener":"click", "name":"Trefferliste Navigation-Button", "id":"1057046049"}" data-parameters="{"partner": "googlemaps", "searchquery": "Samariterstr%2033%2010247%20Berlin"}" data-target="_blank"
>Route</span>
<a
class="contains-icon-details gs-btn"
rel="noopener"
href="https://www.gelbeseiten.de/gsbiz/64195b36-063f-4401-8536-0d1c71a76326"
data-wipe="{"listener": "mouseup", "name": "Trefferliste Actionbutton Mehr Details", "id": "1057046049", "synchron": false}" data-isNeededPromise="false" data-cookieinfo="64195b36-063f-4401-8536-0d1c71a76326=1057046049"
>Mehr Details</a>
<div class="mod-gsSlider__spacer"></div>
</div>
</div>
</div>
</article>
My JS code:我的 JS 代码：
const puppeteer = require("puppeteer");
async function getContacts(){
const browser = await puppeteer.launch({
headless: false,
defaultViewport: null
});
const page = await browser.newPage();
const url = "https://www.gelbeseiten.de/Suche/Fotografen/Berlin";
await page.goto(url);
await page.waitFor(".mod-Treffer");
const results = await page.$$eval(".mod-Treffer", rows => {
return rows.map(row => {
const properties = {};
const firma = row.querySelector(".mod-Treffer h2");
const tel = row.querySelector(".mod-AdresseKompakt__phoneNumber");
const webSite = row.querySelector(" .contains-icon-homepage");
properties.firma = firma.innerText;
properties.tel = tel.innerText;
properties.webSite = webSite.getAttribute("href");
return properties;
})
})
console.log(results)
}
getContacts();
1 个解决方案
解决方案1
0 2020-07-22 14:33:00
I have tested and some .mod-Treffer elements are empty .我已经测试了一些.mod-Treffer元素是空的。 Thus, when you want to query for .contains-icon-homepage , given that you query, no errors are thrown.因此，当您要查询.contains-icon-homepage时，如果您查询，则不会引发错误。 Nonetheless, when you try to access to an internal tag attr (ie .getAttribute(); ), and because the object is null , you get an exception.尽管如此，当您尝试访问内部标记 attr（即.getAttribute(); ）时，并且因为 object 是null ，您会遇到异常。
Solution : You should test for non undefined tags.解决方案：您应该测试undefined的标签。 Like:喜欢：
if (webSite !== undefined) {
properties.webSite = webSite.getAttribute("href");
}
Capture :捕获：
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.