I have the below html snippet. I want to web scraping the page to get the topics and subtopics and store it in objects.
the desired result is something:
{
'topic': 'Java Basics',
'subtopics':['Define the scope of variables', 'Define the structure of a Java class', ...]
}
I trying to make it work with Jsdom for Node.js and JQuery:
var jsdom = require('jsdom');
var fs = require("fs");
var topicos = fs.readFileSync("topic.html", "utf-8");
jsdom.env(topicos, ["http://code.jquery.com/jquery.js"], function (error, window) {
var $ = window.$;
var length = $('div ~ ').each(function () {
//???
var topic = $(this);
var text = topic.text();
console.log(text.trim())
});
})
but due to my lack of experience in jQuery, I am not able to organize the hierarchy properly.
Html snippet:
<div>
<strong>Java Basics </strong></div>
<ul>
<li>
Define the scope of variables </li>
<li>
Define the structure of a Java class
</li>
<li>
Create executable Java applications with a main method; run a Java program from the command line; including
console output.
</li>
<li>
Import other Java packages to make them accessible in your code
</li>
<li>
Compare and contrast the features and components of Java such as:
platform independence, object orientation, encapsulation, etc.
</li>
</ul>
<div>
<strong>Working With Java Data Types </strong></div>
<ul>
<li>
Declare and initialize variables (including casting of primitive data types)
</li>
<li>
Differentiate between object reference variables and primitive variables
</li>
<li>
Know how to read or write to object fields
</li>
<li>
Explain an Object's Lifecycle (creation, "dereference by reassignment" and garbage collection)
</li>
<li>
Develop code that uses wrapper classes such as Boolean, Double, and Integer. </li>
</ul>
...
Here is working snippet fiddle
var topicos = [];
jQuery('div').each(function(){
var data = {};
var jThis = jQuery(this);
data.topic = jThis.find('strong').text();
data.subtopics = [];
jThis.next('ul').find('li').each(function(){
var jThis = jQuery(this);
data.subtopics.push(jThis.text());
});
topicos.push(data);
});
console.log(topicos);
But I would highly recommend to add classes to your markup and use them as selectors instead of tag-names:
<div class="js-topic-data">
<div>
<strong class="js-topic">Java Basics </strong>
</div>
<ul>
<li class="js-sub-topic">
Define the scope of variables </li>
<li>
</ul>
</div>
Then you could do something like:
jQuery('.js-topic-data').each(function(){
var data = {};
var jThis = jQuery(this);
data.topic = jThis.find('.js-topic').text();
data.subtopics = [];
jThis.next('.js-sub-topic').each(function(){
var jThis = jQuery(this);
data.subtopics.push(jThis.text());
});
topicos.push(data);
});
which is much more robust for markup changes etc
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.