What is the tidiest way to collect lots of data in .Net

Question

I am writing a program which collects lots of individual pieces of data from a MySQL database and serializes this data into an XML document.

My XML document has five basic groups, each of which contains 3 subgroups and I am collecting about 100 pieces of data in total.

The content itself is order details on a shopping website. I first run a query to work out and return the next Order ID which needs to be sent to the client, and then use this Order ID to pull back each piece of data (Address line 1, Address Line 2, Items purchased etc) from various tables in a relational database.

No matter which way I go about this it's messy. I end up with massive methods, with one variable defined for each piece of data I need, and each variable is equal to the return value of a method I have which runs a query against my MySQL database. Some of the data needs extra logic applied (eg some data needs cleaned, some dates need formatted a certain way for the XML).

A few points

• I didn't create the database structure or the shopping website, and I can't change it
• I have a working program which is currently running. This is a program I wrote months ago which I am going back to improve

Some example code is below:

string valueOne = myMethod("SELECT `valueOne` FROM `tableOne` WHERE `OrderID` = '12345';");
string valueTwo = myMethod("SELECT `valueTwo` FROM `tableOne` WHERE `OrderID` = '12345';");
string valueThree = myMethod("SELECT `valueThree` FROM `tableTwo` WHERE `OrderID` = '12345';");
int valueFour = Convert.ToInt32(myMethod("SELECT `valueFour` FROM `tableThree` WHERE `OrderID` = '12345';"));
string valueFive = myMethod("SELECT `valueFive` FROM `tableThree` WHERE `OrderID` = '12345';");

if(valueFive == "FooBar")
{
    //Do Stuff
}

string valueSix = myMethod("SELECT `valueSix` FROM `tableThree` WHERE `OrderID` = '12345';");
DateTime valueSeven = DateTime.Parse(myMethod("SELECT `valueSeven` FROM `tableFour` WHERE `OrderID` = '12345';"));
string valueEight = myMethod("SELECT `valueEight` FROM `tableFive` WHERE `OrderID` = '12345';");
string valueNine = String.Format("QWERTY - {0} - YTREWQ", myMethod("SELECT `valueNine` FROM `tableSix` WHERE `OrderID` = '12345';"));
string valueTen = myMethod("SELECT `valueTen` FROM `tableSeven` WHERE `OrderID` = '12345';");

        MyClass fooBar = new MyClass()
        {
            valueOne = valueOne,
            valueTwo = valueTwo,
            valueThree = valueThree,
            valueFour = valueFour,
            valueFive = valueFive,
            valueSix = valueSix,
            valueSeven = valueSeven,
            mySecondClass = new MySecondClass()
            {
                valueEight = valueEight,
                valueNine = valueNine,
                myThirdClass = new MyThirdClass() { valueTen = valueTen }
            }
        };

SerializeToXML<MyClass>(fooBar);

Imagine that, but with a lot more data. Yes it works, but it is untidy, difficult to maintain and, more generally, isn't very good.

So my question is, what is the proper way to collect large amounts of data in a .Net application?

Answer 1

Right off the bat it looks like you can combine a LOT of queries into a single one. For example, you have the following:

string valueOne = myMethod("SELECT `valueOne` FROM `tableOne` WHERE `OrderID` = '12345';");
string valueTwo = myMethod("SELECT `valueTwo` FROM `tableOne` WHERE `OrderID` = '12345';");
string valueThree = myMethod("SELECT `valueThree` FROM `tableTwo` WHERE `OrderID` = '12345';");
int valueFour = Convert.ToInt32(myMethod("SELECT `valueFour` FROM `tableThree` WHERE `OrderID` = '12345';"));
string valueFive = myMethod("SELECT `valueFive` FROM `tableThree` WHERE `OrderID` = '12345';");

I would get rid of those individual queries and use a single one:

SELECT one.valueOne, one.valueTwo, two.valueThree, three.valueFour, three.valueFive
  FROM tableOne one
    INNER JOIN tableTwo two on (two.OrderId = one.OrderId)
    INNER JOIN tableThree three on (three.OrderId = one.OrderId)
  WHERE one.OrderId = '12345';

Quite frankly, it looks like you can do the same thing to the others.

Also, I'm not entirely sure what myMethod does, but it sure looks like it's simply returning a scalar value on a query. Get rid of that and replace it with something that will give you a DataTable or object collection back. That way you can pull all the data you need in one go.

NOTE: the Inner Joins work if you expect every one of those tables to have values based on the OrderId. If they don't, then do an outer join starting with the table that will always have an existing record. For example:

SELECT one.valueOne, one.valueTwo, two.valueThree, three.valueFour, three.valueFive
      FROM orders o
        LEFT OUTER JOIN tableOne one on (one.OrderId = o.OrderId)
        LEFT OUTER JOIN tableTwo two on (two.OrderId = o.OrderId)
        LEFT OUTER JOIN tableThree three on (three.OrderId = o.OrderId)
      WHERE o.OrderId = '12345';

Answer 2

You are really under-utilizing object oriented programming. From your example it's hard to infer the relationships between your tables, but Chris Lively did a great job inferring the relationships between your tables, and you described it as an order details from a shopping website. Certainly that implies some consistent relationships.

I would approach the problem this way:

• Develop the most efficient consolidated queries to get the data you need.
• For each query, create a class. Structure your classes to match your data ("five basic groups, each of which contains 3 subgroups").
• Follow good OOP principles and hide the complexity required by a class in that class, if that's the only place it is required.
• Use much more descriptive names for your variables and classes.

That should yield much more readable, maintainable, "tidy" code.