使用Javascript获取网页文本

Question

BeautifulSoup, a python library, has a function called get_text() that can take a parsed HTML page, such as this: https://pastebin.com/DJwA3S5P

and extract all of the text from it, thus turning it into this: https://pastebin.com/qMqrj8RS

Here's another example of what the function can do:

If given the following:

<span id="sm_flash_225" onclick="sm_flash_process('bail', this,1)" onmouseover="sm_flash_add('bail', this, 1);" onmouseout="sm_flash_remove('bail', this, 1);">bail</span> 

BeautifulSoup's get_text() function will simply turn it into: bail

In other words, it takes <span id ="some_id" more random stuff...>text</span> and turns into into text .

I have the HTML file of a website that is contained as one large formatted string. I would like to write the Javascript equivalent of BeautifulSoup's get_text() in order to only get the text of the webpage. I'm fine with using any third party library etc., I don't want to re-invent the wheel. However, it's worth noting that I'm writing this in the context of a Chrome/Firefox web extension, so I don't believe I can use every single 3rd party library.

I acquired the HTML file with the following code:

fetch(url)
.then((resp) => resp.text())
.then(function (data) { 
    //get the text of the webpage by 
    //mimicking Beautiful Soup's get_text() function        
})

Answer 1

try this:

fetch("test.html")
  .then(data => data.text())
  .then(text => {
    let div = document.createElement("div");
    div.innerHTML = text;
    console.log(div.textContent);
  });

Answer 2

Safer not to insert live HTML (and JS) from another site onto your own. Use DOMParser instead:

 fetch("https://cors-anywhere.herokuapp.com/stackoverflow.com", ) .then(response => response.text()) .then(responseText => { const responseDocument = (new DOMParser()).parseFromString(responseText, 'text/html'); console.log(responseDocument.head.textContent); console.log(responseDocument.body.textContent); });