如何抓取 javascript 哈希鏈接內容？

Question

嗨，我在使用 Puppeter 進行網絡抓取方面有點新，我目前面臨下一個問題：

在我試圖提取信息的站點中，我有一個帶有典型 js 分頁的引導表，例如以下示例： https : //getbootstrap.com/docs/4.1/components/pagination/

當我用 Chrome 檢查器檢查頁面 html 時，我只能看到2 ，當我檢查鏈接位置時，我看到

https://webpage.com/works#

我怎么知道總共有多少頁？ 我如何點擊它們？ 我不明白如何訪問這種類型的分頁的每一頁。

謝謝！

Answer 1

沒有萬無一失的方法，但我按這個順序處理分頁，

等待目標元素出現
從目標收集數據
移除目標元素
點擊下一步按鈕
...循環直到沒有下一個按鈕或即使等待后內容也沒有加載

概念證明：

目標 HTML 代碼：

 <!-- Copied from: https://jsfiddle.net/solodev/yw7y4wez --> <!DOCTYPE html> <html> <head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <title>Pagination Example</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <meta name="robots" content="noindex, nofollow"> <meta name="googlebot" content="noindex, nofollow"> <meta name="viewport" content="width=device-width, initial-scale=1"> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script> <link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script> <script type="text/javascript" src="https://www.solodev.com/assets/pagination/jquery.twbsPagination.js"></script> <style type="text/css"> .container { margin-top: 20px; } .page { display: none; } .page-active { display: block; } </style> <script type="text/javascript"> window.onload = function() { $('#pagination-demo').twbsPagination({ totalPages: 5, // the current page that show on start startPage: 1, // maximum visible pages visiblePages: 5, initiateStartPageClick: true, // template for pagination links href: false, // variable name in href template for page number hrefVariable: '{{number}}', // Text labels first: 'First', prev: 'Previous', next: 'Next', last: 'Last', // carousel-style pagination loop: false, // callback function onPageClick: function(event, page) { $('.page-active').removeClass('page-active'); $('#page' + page).addClass('page-active'); }, // pagination Classes paginationClass: 'pagination', nextClass: 'next', prevClass: 'prev', lastClass: 'last', firstClass: 'first', pageClass: 'page', activeClass: 'active', disabledClass: 'disabled' }); } </script> </head> <body> <div class="container"> <div class="jumbotron page" id="page1"> <div class="container"> <h1 class="display-3">Adding Pagination to your Website</h1> <p class="lead">In this article we teach you how to add pagination, an excellent way to navigate large amounts of content, to your website using a jQuery Bootstrap Plugin.</p> <p><a class="btn btn-lg btn-success" href="https://www.solodev.com/blog/web-design/adding-pagination-to-your-website.stml" role="button">Learn More</a></p> </div> </div> <div class="jumbotron page" id="page2"> <h1 class="display-3">Not Another Jumbotron</h1> <p class="lead">Cras justo odio, dapibus ac facilisis in, egestas eget quam. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> <p><a class="btn btn-lg btn-success" href="#" role="button">Sign up today</a></p> </div> <div class="jumbotron page" id="page3"> <h1 class="display-3">Data. Data. Data.</h1> <p>This example is a quick exercise to illustrate how the default responsive navbar works. It's placed within a <code>.container</code> to limit its width and will scroll with the rest of the page's content. </p> <p> <a class="btn btn-lg btn-primary" href="../../components/navbar" role="button">View navbar docs »</a> </p> </div> <div class="jumbotron page" id="page4"> <h1 style="-webkit-user-select: auto;">Buy Now!</h1> <p class="lead" style="-webkit-user-select: auto;">Cras justo odio, dapibus ac facilisis in, egestas eget quam. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet.</p> <p style="-webkit-user-select: auto;"><a class="btn btn-lg btn-success" href="#" role="button" style="-webkit-user-select: auto;">Get started today</a></p> </div> <div class="jumbotron page" id="page5"> <h1 class="cover-heading">Cover your page.</h1> <p class="lead">Cover is a one-page template for building simple and beautiful home pages. Download, edit the text, and add your own fullscreen background photo to make it your own.</p> <p class="lead"> <a href="#" class="btn btn-lg btn-primary">Learn more</a> </p> </div> <ul id="pagination-demo" class="pagination-lg pull-right"></ul> </div> <script> // tell the embed parent frame the height of the content if (window.parent && window.parent.parent) { window.parent.parent.postMessage(["resultsFrame", { height: document.body.getBoundingClientRect().height, slug: "yw7y4wez" }], "*") } </script> </body> </html>

這是代碼的示例工作版本，

const puppeteer = require('puppeteer');

async function runScraper() {
  let browser = {};
  let page = {};
  const url = 'http://localhost:8080';

  // open the page and wait
  async function navigate() {
    browser = await puppeteer.launch({ headless: false });
    page = await browser.newPage();
    await page.goto(url);
  }

  async function scrapeData() {
    const headerSel = 'h1';
    // wait for element
    await page.waitFor(headerSel);
    return page.evaluate((selector) => {
      const target = document.querySelector(selector);

      // get the data
      const text = target.innerText;

      // remove element so the waiting function works
      target.remove();
      return text;
    }, headerSel);
  }

  // this is a sample concept of pagination
  // it will vary from page to page because not all site have same type of pagination

  async function paginate() {
    // manually check if the next button is available or not
    const nextBtnDisabled = !!(await page.$('.next.disabled'));
    if (!nextBtnDisabled) {
      // since it's not disable, click it
      await page.evaluate(() => document.querySelector('.next').click());

      // just some random waiting function
      await page.waitFor(100);
      return true;
    }
    console.log({ nextBtnDisabled });
  }

  /**
   * Scraping Logic
   */
  await navigate();

  // Scrape 5 pages
  for (const pageNum of [...Array(5).keys()]) {
    const title = await scrapeData();
    console.log(pageNum + 1, title);
    await paginate();
  }
}

runScraper();

結果：

Server running at 8080
1 'Adding Pagination to your Website'
2 'Not Another Jumbotron'
3 'Data. Data. Data.'
4 'Buy Now!'
5 'Cover your page.'
{ nextBtnDisabled: true }

我沒有分享服務器代碼，它基本上是上面的 html 片段。

Answer 2

使用屬性footerTemplate和displayHeaderFooter顯示最初使用puppeteer API 的頁面

await page.pdf({
  path: 'hacks.pdf',
  format: 'A4',
  displayHeaderFooter: true,
  footerTemplate: '<div><div class='pageNumber'></div> <div>/</div><div class='totalPages'></div></div>'
});

https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pagepdfoptions

footerTemplate打印頁腳的 HTML 模板。

// 應該是有效的 HTML 標記，具有以下用於將打印值注入其中的CSS 類：

// -日期格式的打印日期

// -標題文檔標題

// - url文檔位置

// - pageNumber當前頁碼

// - totalPages文檔中的總頁數

如何抓取 javascript 哈希鏈接內容？

問題描述

2 個解決方案

解決方案1
0 2018-10-19 04:45:05

概念證明：

解決方案2
0 2020-01-27 09:23:42

如何抓取 javascript 哈希鏈接內容？

問題描述

2 個解決方案

解決方案1 0 2018-10-19 04:45:05

概念證明：

解決方案2 0 2020-01-27 09:23:42

解決方案1
0 2018-10-19 04:45:05

解決方案2
0 2020-01-27 09:23:42