使用jsoup获取URL的子链接

Question

Consider a URl www.example.com it may have plenty numbers of links ,some may be internal and other may be external.I want to get a list of all the sub links ,not even the sub-sub links but only sub link. EG if there are four links as follows

1)www.example.com/images/main
2)www.example.com/data
3)www.example.com/users
4)www.example.com/admin/data

Then out of the four only 2 and 3 are of use as they are sub links not the sub-sub and so on links .Is there a way to achieve it through j-soup..If this could not be achieved through j-soup then one can introduce me with some other java API. Also note that it should be a link of the parent Url which is initially sent(ie www.example.com)

Answer 1

If i can understand a sub-link can contain one slash you can attempt with this with counting the number of slashes for example :

List<String> list = new ArrayList<>();
list.add("www.example.com/images/main");
list.add("www.example.com/data");
list.add("www.example.com/users");
list.add("www.example.com/admin/data");

---

for(String link : list){
    if((link.length() - link.replaceAll("[/]", "").length()) == 1){
        System.out.println(link);
    }
}

link.length() : count the number of characters
link.replaceAll("[/]", "").length() : count the number of slashes

If the difference equal to one then right link else no.

---

EDIT

How will i scan the whole website for sub links?

The answer for this with the robots.txt file or Robots exclusion standard , so in this it define all the sub-links of the web site for example https://stackoverflow.com/robots.txt , so the idea is, to read this file and you can extract the sub-links from this web-site here is a piece of code that can help you :

public static void main(String[] args) throws Exception {

    //Your web site
    String website = "http://stackoverflow.com";
    //We will read the URL https://stackoverflow.com/robots.txt
    URL url = new URL(website + "/robots.txt");

    //List of your sub-links
    List<String> list;

    //Read the file with BufferedReader
    try (BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()))) {
        String subLink;
        list = new ArrayList<>();

        //Loop throw your file
        while ((subLink = in.readLine()) != null) {

            //Check if the sub-link is match with this regex, if yes then add it to your list
            if (subLink.matches("Disallow: \\/\\w+\\/")) {
                list.add(website + "/" + subLink.replace("Disallow: /", ""));
            }else{
                System.out.println("not match");
            }
        }
    }

    //Print your result
    System.out.println(list);
}

This will show you :

[ https://stackoverflow.com/posts/ , https://stackoverflow.com/posts ?, https://stackoverflow.com/search/ , https://stackoverflow.com/search ?, https://stackoverflow.com/feeds/ , https://stackoverflow.com/feeds ?, https://stackoverflow.com/unanswered/ , https://stackoverflow.com/unanswered ?, https://stackoverflow.com/u/ , https://stackoverflow.com/messages/ , https://stackoverflow.com/ajax/ , https://stackoverflow.com/plugins/]

Here is a Demo about the regex that i use .

Hope this can help you.

Answer 2

To scan the links on the web page you can use JSoup library.

import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

class read_data {

    public static void main(String[] args) {
        try {
            Document doc = Jsoup.connect("**your_url**").get();
            Elements links = doc.select("a");
            List<String> list = new ArrayList<>();
            for (Element link : links) {
                list.add(link.attr("abs:href"));
            }
        } catch (IOException ex) {

        }
    }
}

list can be used as suggested in the previous answer.

---

The code for reading all the links on a website is given below. I have used http://stackoverflow.com/ for illustration. I would recommend you to go through company's terms of use before scarping it's website.

import java.io.IOException;
import java.util.HashSet;
import java.util.Set;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;

public class readAllLinks {

    public static Set<String> uniqueURL = new HashSet<String>();
    public static String my_site;

    public static void main(String[] args) {

        readAllLinks obj = new readAllLinks();
        my_site = "stackoverflow.com";
        obj.get_links("http://stackoverflow.com/");
    }

    private void get_links(String url) {
        try {
            Document doc = Jsoup.connect(url).get();
            Elements links = doc.select("a");
            links.stream().map((link) -> link.attr("abs:href")).forEachOrdered((this_url) -> {
                boolean add = uniqueURL.add(this_url);
                if (add && this_url.contains(my_site)) {
                    System.out.println(this_url);
                    get_links(this_url);
                }
            });

        } catch (IOException ex) {

        }

    }
}

You will get list of all the links in uniqueURL field.