[英]How to scrape date from iframe tag with with puppeteer
I try to scrape some data from an iframe/frame tag, but I got stuck with the code for puppeteer.I'm a novice so please bear with me.This is the link of the site .In there when I click in a name in the first frame I get some data in second frame on witch again I can click and get data in third frame.我试图从 iframe/frame 标签中抓取一些数据,但是我被 puppeteer 的代码卡住了。我是一个新手,所以请耐心等待。这是网站的链接。当我点击一个名字时在那里在第一帧中,我再次在女巫的第二帧中获取一些数据,我可以单击并在第三帧中获取数据。 In the code I try to loop true first frame to get all data for the second and third.在代码中,我尝试循环真正的第一帧以获取第二帧和第三帧的所有数据。
Thank you for any hints.谢谢你的任何提示。
I have run this command: document.querySelector("body > form > font > select > option") in console, but I can't find a way to run it in puppeteer.我已经在控制台中运行了这个命令:document.querySelector("body > form > font > select > option"),但我找不到在 puppeteer 中运行它的方法。
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('');
const iframeParagraph = await page.evaluate(() => {
const iframe = document.getElementsByName("stanga");
// grab iframe's document object
const iframeDoc = iframe.contentDocument || iframe.contentWindow.document;
const iframeP = iframeDoc.getElementsByName("fmtstatii");
return iframeP.innerHTML;
});
console.log(iframeParagraph);
await browser.close();
})();
or要么
const puppeteer = require('puppeteer');
let scrape = async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto('');
await page.click('document.querySelector("body > form > font > select")');
await page.waitFor(1000);
const result = await page.evaluate(() => {
let statie = document.querySelector('document.querySelector("body > form > font > select > option")').innerText;
return {
statie
}
});
browser.close();
return result;
};
scrape().then((value) => {
console.log(value); // Success!
});
This is the error that I get:这是我得到的错误:
[(node:13308) UnhandledPromiseRejectionWarning: Error: Evaluation failed: DOMException: Failed to execute 'querySelector' on 'Document': 'document.querySelector("body > form > font > select")' is not a
valid selector.
at __puppeteer_evaluation_script__:1:33
at ExecutionContext._evaluateInternal (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\ExecutionContext.js:122:13)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at ExecutionContext.<anonymous> (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\helper.js:111:15)
at ElementHandle.$ (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\JSHandle.js:395:50)
at ElementHandle.<anonymous> (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\helper.js:112:23)
at DOMWorld.$ (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\DOMWorld.js:121:34)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at Frame.<anonymous> (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\helper.js:111:15)
at Page.click (D:\Zero\ratt_scrap\node_modules\puppeteer\lib\Page.js:986:29)
at scrape (D:\Zero\ratt_scrap\scrape.js:23:16)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:13308) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:13308) \[DEP0018\] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.]
You have made several mistakes:你犯了几个错误:
You should interact with the Frame
instead of the Page
object.您应该与Frame
而不是Page
对象进行交互。
const frame = await page.frames().find(frame => frame.name() === 'stanga'); // Find the right frame.
The click()
method expects the selector <string>
, so you don't need to add the document.querySelector
inside the click()
method. click()
方法需要selector <string>
,因此您不需要在click()
方法中添加document.querySelector
。
await frame.click('body > form > font > select');
And to get all innerText
you have to iterate over the elements.要获得所有的innerText
您必须遍历元素。
Don't forget to add the await
.不要忘记添加await
。 You have missed for the close
method.你错过了close
方法。
await browser.close();
SOLUTION:解决方案:
const puppeteer = require('puppeteer');
let scrape = async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto('http://ratt.ro/txt');
const frame = await page.frames().find(frame => frame.name() === 'stanga');
await frame.click('body > form > font > select');
await page.waitFor(1000);
const optionsResult = await frame.$$eval('body > form > font > select > option', (options) => {
const result = options.map(option => option.innerText);
return result;
});
await browser.close();
return optionsResult;
};
scrape().then((value) => {
console.log(value); // Success!
});
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.