需要从网页上抓取网址

Question

I need to scrape a url from a website which is located within some javascript code.

<script type="text/javascript">
    (function() {
        // somewhere..
        $.get("http://someurl.com?q=34343&b=343434&c=343434")...
    });
</script>

I know that the url starts with http://someurl.com?q= and it needs to have at least a second query parameter ( &b= ) inside, but the rest of the content is unknown.

I initially tried with jsoup , however it's not really suitable for that task. Manually fetching the page and then applying a regex pattern on it is also not a preferable option since the page is huge. What could I do to get the url quick and safe?

Answer 1

You can use this regex

/\$\.get\("(http:\/\/someurl\.com\?q=[\w.\-%#\/]*&b=[\w.\-%&=\/]*)"\)/g

This regex will search directly for this string:

$.get("http://someurl.com?q=

It will then allow any number of URL valid characters to occur as the value of q.

It will then look to match

&b=

and then again any number of valid characters followed by the opposing quotation marks. I tested it with

MATCH - $.get("http://someurl.com?q=34343&b=343434&c=343434")
MATCH - $.get("http://someurl.com?q=34343&b=13a43&k=343434&c2=something")
FAIL  - $.get("http://someurl.com?q=34343&c=343434&b=343434")
FAIL  - $.get("http://someurl.com?a=34343&b=343434=343434")

If you only want to return the first result you can remove the global identifier from the end

/\$\.get\("(http:\/\/someurl\.com\?q=[\w.\-%#\/]*&b=[\w.\-%&=\/]*)"\)/