Hi, I'm Trying To scrape href from Anchor tag Using JQuery.ajax

Question

But I Only get the Only URLs which are rendered on HTML, not All because there is a Slider eg 25/145, I want to Get all the Urls, any solutions

var result = [];
var index = 0;
jQuery.ajax({
  async: false,
  type: 'POST',
  url: "https://www.usajobs.gov/Search/Results?a=TD03&p=1",
  headers: {
    'contentType': 'application/x-www-form-urlencoded; charset=UTF-8'
  },
  crossDomain: true,
  success: function(response) {
    var urls = $("#usajobs-search-results").find('a').each(function() {
      let url = $(this).attr('href');
      console.log(`https://www.usajobs.gov/${url}`);
    });
    for (var i = 0; i < urls.length; i++) {
      result[index] = urls[i].href;
      index++;
    }
  },
});

Answer 1

1st do you have permission to scrape?

If so

Seems you need to change the X in

https://www.usajobs.gov/Search/Results?a=TD03&p=X 

in the success

from 1 until 404

and use a setTimeout to call again to not hammer them Also

 var urls = $("#usajobs-search-results a').map(function() {
   return this.href; }).get()
 })