getting data from website using C#/vb.net

Question

I have an XML with following structure:

<table name="tblsiccodes">
    <row>
        <sSICCode>0888</sSICCode>
    </row>
    <row>
        <sSICCode>0900</sSICCode>
    </row>
    <row>
        <sSICCode>1000</sSICCode>
    </row>
    <row>
        <sSICCode>1040</sSICCode>
    </row>
    <row>
        <sSICCode>1044</sSICCode>
    </row>
    <row>
        <sSICCode>1090</sSICCode>
    </row>
    <row>
        <sSICCode>1220</sSICCode>
    </row>
    <row>
        <sSICCode>1221</sSICCode>
    </row>
</table>

and I need to pick the description for each of SIC code from this site

http://www.epa.gov/enviro/html/sic_lkup.html

or this site

http://www.sec.gov/info/edgar/siccodes.htm

What will be easiest way using vb.net to get description for each SIC from the website? I can not modify this XML file. Do I need to create a collection with SIC code and description manually copied to it and then get description of SIC code in XMl by matching SIC code with collection ? Final collection will be used to populate a repeater with each item/row having SIC code and Industry Name.

Do I need to do screen scrapping ? I don't think that these websites have some API/service using which I can get industry names

I can not use HTMl agility pack or any other third party library except Jquery.

Please suggest.

Answer 1

Looking at the source of the first page, it contains the following select:

<select>
   <option value=''> NO SIC CODE SELECTED </option>
   <option value='3291'> ABRASIVE PRODUCTS (SIC Code: 3291) </option>
   (...)
</select>

and so it goes on.
You can save all the options of the select as another xml file, and then use XDocument, or any other Xml loading features of .net (be that c# or vb.net) to traverse ur document, and find each SIC code by the value of the value attribute in the xml containting all the optoins.
Of course, this might be not the fastest way, but the easiest to implement.