在Silverlight 4页上进行网页抓取

Question

There is a web page with a silver light application embedded and I want to scrape it. I wonder if there is a web scraping solution or browser automation solution (or any trick/hack) that supports silverlight in either C#, Java or Python.

I am currently trying silvernium but it is quite outdated project and doesn't seem to work properly with silverlight 4.

Here is some of the html information of the SL object in the page:

<object data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="100%" height="100%">

<param name="source" value="PATH/WebSilverlight.xap"/>
<param name="onerror" value="onSilverlightError"/>
<param name="background" value="white"/>
<param name="minRuntimeVersion" value="4.0.50524.0"/>
<param name="autoUpgrade" value="true"/>
<param name="windowless" value="true"/>
<param name="enableautozoom" value="true"/>
...

Answer 1

I have worked successfully with Telerik testing framework for an

automation solution that supports silverlight in C#

It's free and once you get to used to it - very easy, because of the rich API and the cross-browser compatibility. Maybe the trickiest part is to initially config the tests.

Simple example:

Settings mySettings = new Settings();
mySettings.Web.DefaultBrowser = BrowserType.InternetExplorer;
Manager myManager = new Manager(mySettings);
myManager.Start();    
myManager.LaunchNewBrowser();

myManager.ActiveBrowser.NavigateTo("http://www.example.com");

Element mybtn = myManager.ActiveBrowser.Find.ByTagIndex("input", 3);
myManager.ActiveBrowser.Actions.Click(mybtn);

myManager.Dispose();

A good addition to it is the Windows Inspect tool . It'll enable you to select any UI element and view the element's accessibility data in some tricky cases.

Update:

I've searched some helpful documentation links that I've used back in the days. Look at Getting started with Silverlight UI Automation and Locating elements .

Answer 2

In the end, I implemented a workaround using the computer vision based technology SikuliX and getting a printed PDF out of the Silverlight web app like normal normal user would to get the information. Here is an script that shows how to run this along with Selenium.

Another alternative is to hack the requests and keep the session alive while navigating to the information you need using either Scrapy , abot , crawler4j or any other similar technology.