Formatting errors in my LINQ to XML query

Question

I'm trying to work the kinks out of this LINQ to XML query of stock market data returned via yql from yahoo finance. Several of the fields come in either with misformed data or null values. My current error is:

Input string was not in a correct format.

I've added a check nulls for every element I'm interested in, but the error persists.

If it helps, the highlighted code section that appears to throw the exception is the the block containing the "new" initialization of my data point object. If the XML passed the checks in the query, it would throw an error when nulls are passed to the constructor when initializing the DailyPricingVolDP .

Func<string, DateTime?> tryToGetDate =
    value =>
    {
        DateTime dateValue;
        return DateTime.TryParse(value, out dateValue)
            ? (DateTime?)dateValue
            : null;
    };

XDocument doc = XDocument.Load(addressString);
XElement results = doc.Root.Element("results");

CultureInfo enUS = new CultureInfo("en-US");
DateTime targetDate = new DateTime(2012, 1, 25);

var makeInfo =
    from s in doc.Descendants("quote")
    where s.Element("LastTradeDate") != null
       && s.Attribute("symbol") != null
    let dateStr = s.Element("LastTradeDate").Value
    let dateValue = tryToGetDate(dateStr)
    where dateValue != null && (DateTime)dateValue == targetDate
       && s.Element("Open") != null && s.Element("DaysHigh") != null
       && s.Element("DaysLow") != null && s.Element("LastTradePriceOnly") != null
       && s.Element("Volume") != null
    select new DailyPricingVolDP(
        (string)s.Attribute("symbol"),
        (DateTime)s.Element("LastTradeDate"),
        (double)s.Element("Open"),
        (double)s.Element("DaysHigh"),
        (double)s.Element("DaysLow"),
        (double)s.Element("LastTradePriceOnly"),
        (long)s.Element("Volume"));

<quote symbol="AFFM">
  <Ask>0.89</Ask>
  <AverageDailyVolume>6523</AverageDailyVolume>
  <Bid>0.67</Bid>
  <AskRealtime>0.89</AskRealtime>
  <BidRealtime>0.67</BidRealtime>
  <BookValue>4.804</BookValue>
  <Change_PercentChange>0.00 - 0.00%</Change_PercentChange>
  <Change>0.00</Change>
  <Commission />
  <ChangeRealtime>0.00</ChangeRealtime>
  <AfterHoursChangeRealtime>N/A - N/A</AfterHoursChangeRealtime>
  <DividendShare>0.00</DividendShare>
  <LastTradeDate>1/25/2012</LastTradeDate>
  <TradeDate />
  <EarningsShare>-4.491</EarningsShare>
  <ErrorIndicationreturnedforsymbolchangedinvalid />
  <EPSEstimateCurrentYear>0.00</EPSEstimateCurrentYear>
  <EPSEstimateNextYear>0.00</EPSEstimateNextYear>
  <EPSEstimateNextQuarter>0.00</EPSEstimateNextQuarter>
  <DaysLow />
  <DaysHigh />
  <YearLow>0.42</YearLow>
  <YearHigh>2.95</YearHigh>
  <HoldingsGainPercent>- - -</HoldingsGainPercent>
  <AnnualizedGain />
  <HoldingsGain />
  <HoldingsGainPercentRealtime>N/A - N/A</HoldingsGainPercentRealtime>
  <HoldingsGainRealtime />
  <MoreInfo>cnprmiIed</MoreInfo>
  <OrderBookRealtime />
  <MarketCapitalization>9.2M</MarketCapitalization>
  <MarketCapRealtime />
  <EBITDA>-34.5M</EBITDA>
  <ChangeFromYearLow>+0.18</ChangeFromYearLow>
  <PercentChangeFromYearLow>+42.86%</PercentChangeFromYearLow>
  <LastTradeRealtimeWithTime>N/A - &lt;b&gt;0.60&lt;/b&gt;</LastTradeRealtimeWithTime>
  <ChangePercentRealtime>N/A - 0.00%</ChangePercentRealtime>
  <ChangeFromYearHigh>-2.35</ChangeFromYearHigh>
  <PercebtChangeFromYearHigh>-79.66%</PercebtChangeFromYearHigh>
  <LastTradeWithTime>Jan 25 - &lt;b&gt;0.60&lt;/b&gt;</LastTradeWithTime>
  <LastTradePriceOnly>0.60</LastTradePriceOnly>
  <HighLimit />
  <LowLimit />
  <DaysRange>N/A - N/A</DaysRange>
  <DaysRangeRealtime>N/A - N/A</DaysRangeRealtime>
  <FiftydayMovingAverage>0.5639</FiftydayMovingAverage>
  <TwoHundreddayMovingAverage>1.4051</TwoHundreddayMovingAverage>
  <ChangeFromTwoHundreddayMovingAverage>-0.8051</ChangeFromTwoHundreddayMovingAverage>
  <PercentChangeFromTwoHundreddayMovingAverage>-57.30%</PercentChangeFromTwoHundreddayMovingAverage>
  <ChangeFromFiftydayMovingAverage>+0.0361</ChangeFromFiftydayMovingAverage>
  <PercentChangeFromFiftydayMovingAverage>+6.39%</PercentChangeFromFiftydayMovingAverage>
  <Name>Affirmative Insur</Name>
  <Notes />
  <Open />
  <PreviousClose>0.60</PreviousClose>
  <PricePaid />
  <ChangeinPercent>0.00%</ChangeinPercent>
  <PriceSales>0.03</PriceSales>
  <PriceBook>0.12</PriceBook>
  <ExDividendDate>12-Dec-08</ExDividendDate>
  <PERatio />
  <DividendPayDate>31-Dec-08</DividendPayDate>
  <PERatioRealtime />
  <PEGRatio />
  <PriceEPSEstimateCurrentYear />
  <PriceEPSEstimateNextYear />
  <Symbol>AFFM</Symbol>
  <SharesOwned />
  <ShortRatio>0.20</ShortRatio>
  <LastTradeTime>1:34pm</LastTradeTime>
  <TickerTrend>&amp;nbsp;======&amp;nbsp;</TickerTrend>
  <OneyrTargetPrice>6.00</OneyrTargetPrice>
  <Volume>200</Volume>
  <HoldingsValue />
  <HoldingsValueRealtime />
  <YearRange>0.42 - 2.95</YearRange>
  <DaysValueChange>- - 0.00%</DaysValueChange>
  <DaysValueChangeRealtime>N/A - N/A</DaysValueChangeRealtime>
  <StockExchange>NasdaqNM</StockExchange>
  <DividendYield />
  <PercentChange>0.00%</PercentChange>
</quote>

Answer 1

This

s.Element("Open") != null

does not do what you think it does. If there is an element Open , even looking the same as in your example ( <Open /> ) - that condition is true. Element Open is present, and it is not null. It just doesn't have value (or to be precise, it has value of empty string , "" - which is causing your format exceptions).

In your where clause you should change conditions to

s.Element("Volume").Value != ""

in order to prevent attempts of parsing empty-valued elements.

---

Quick example. What does this tiny program prints?

var xDoc = XDocument.Parse("<quote><Open /></quote>");
var open = xDoc.Descendants("quote").First().Element("Open");
Console.WriteLine(open == null);
Console.WriteLine(open);

That's right. False followed by <Open /> .

Answer 2

Those null checks you are doing in your query are pointless, (just about all) those elements are present in the XML, but many of them are empty. You are attempting to parse empty strings and that fails giving you the error. You should give us a sample XML that is representative of what you actually have (ie, has values in them). If this is what you have, you are performing the wrong kind of checks.

I've restructured the query like this and I've marked where the exception is occurring:

var query =
    from quote in doc.Descendants("quote")
    let lastTradeDate = (DateTime)quote.Element("LastTradeDate")
    where lastTradeDate == targetDate
    let symbol = (string)quote.Attribute("symbol")
    let open = (double)quote.Element("Open") // EXCEPTION: empty string to a double?
    let daysHigh = (double)quote.Element("DaysHigh") // DITTO
    let daysLow = (double)quote.Element("DaysLow") // DITTO
    let lastTradePriceOnly = (double)quote.Element("LastTradePriceOnly")
    let volume = (long)quote.Element("Volume") // no Volume element exists, null check might be appropriate
    select new DailyPricingVolDP(symbol, lastTradeDate, open, daysHigh, daysLow, lastTradePriceOnly, volume);