[英]Using Java Selenium to scrape a data across multiple pages when the url doesn't change
I have been trying to write a simple code to scrape some basic stats from www.whoscored.com.
我一直在尝试编写一个简单的代码,以从www.whoscored.com抓取一些基本统计信息。 My code loops through first page links and scrape data just fine but when I switch with driver.findElement(By.xpath("")).click();
我的代码遍历首页链接并抓取数据很好,但是当我使用driver.findElement(By.xpath(“”))。click();切换时, to next table, it collects data from first game BUT then keeps looping/return AGAIN on the first table/page...
到下一张桌子,它从第一个游戏BUT收集数据,然后继续在第一个桌子/页面上循环/返回...
Here is the code: 这是代码:
package Whoscored;
import java.io.File;
public class Datascrape {
public static void main(String[] args) throws InterruptedException,
IOException {
WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS)
// get the URL
driver.get("https://www.whoscored.com/Regions/81/Tournaments/6/Seasons/6393/Germany-Bundesliga-II");
Thread.sleep(1000);
driver.manage().window().maximize();
// first loop to identify matches-list over and over again
for (int w=0;w<=100;w++){
List<WebElement> lista1 =
driver.findElements(By.cssSelector(".match-link.match-report.rc"));
// second loop for each table
for (int j=0;j<=lista1.size(); j++){
lista1.get(j).click();
driver.manage().window().maximize();
// go deeper (two clicks), grab stats and return to original position/table
// clicks
driver.findElement(By.xpath(".//*[@id='sub-
navigation']/ul/li[4]/a")).click();
driver.findElement(By.xpath(".//*[@id='live-match-
options']/li[3]")).click();
// grab stats/text
String home6 = driver.findElement(By.xpath(".//*
[@id='chalkboard']/div[2]/div[1]/div[3]/div[2]/span[1]")).getText();
String away6 = driver.findElement(By.xpath(".//*
[@id='chalkboard']/div[2]/div[1]/div[3]/div[2]/span[2]")).getText();
driver.navigate().back();
Thread.sleep(5000);
driver.navigate().back();
Thread.sleep(5000);
lista1= driver.findElements(By.cssSelector(".match-link.match-
report.rc"));
}
// after it finishes with first table go to second table with click
driver.findElement(By.xpath(".//*[@id='date-
controller']/a[1]/span")).click();
Thread.sleep(5000);
}
}
}
You have to make sure that data has been loaded before reading the document. 阅读文档之前,您必须确保已加载数据。 eg your call
例如你的电话
driver.findElement(By.xpath(".//*[@id='date-controller']a[1]/span")).click();
is fine, but you also need to make sure that the next sibling 很好,但是您还需要确保下一个兄弟姐妹
driver.findElement(By.xpath(".//*[@id='date-controller']a[2]/span"))
has changed its value for example the initial span contains "23 - 29 Jan 2017" and onclick it should change to "19 - 25 Dec 2016" 已更改其值,例如,初始范围包含“ 2017年1月23日至29日”,而onclick应更改为“ 2016年12月19日至25日”
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.