Federation Fan Out Query

Question

I am new in sql azure. Now i am facing a big problem that is because of SQL Azure federation. In one of my project I want to use federation. For that I want to read data from all federation. How can I check total number of federation in the server and read data from the federation? And also how can I insert data into the federation? currently I am using:

string strSQL = @"SELECT f.name, fmc.federation_id, fmc.member_id, fmc.range_low, fmc.range_high " +
                                                            "FROM sys.federations f " +
                                                            "JOIN sys.federation_member_distributions fmc " +
                                                            "ON f.federation_id=fmc.federation_id " +
                                                            "ORDER BY fmc.federation_id, fmc.range_low, fmc.range_high";
        string strTableName = "federation_member_distributions";

        try
        {
            using (SqlConnection connection = new SqlConnection(csb.ToString()))
            {
                connection.Open();
                sbResults.AppendFormat("-- Open {0}\r\n\r\n", csb.ToString());

                data = new DataSet();
                data.Locale = System.Globalization.CultureInfo.InvariantCulture;

                using (SqlCommand command = connection.CreateCommand())
                {
                    // Connect to federation root or federation member
                    command.CommandText = "USE FEDERATION ROOT WITH RESET";
                    command.ExecuteNonQuery();
                    sbResults.AppendFormat("{0}\r\nGO\r\n", command.CommandText);
                }

                //Retrieve federation root metadata, and populate the data to the DataGridView control
                SqlDataAdapter sqlDataAdapter = new SqlDataAdapter(strSQL, connection);
                sqlDataAdapter.Fill(data, strTableName);
                sbResults.AppendFormat("{0}\r\nGO\r\n", strSQL);
            }

        }
        catch (Exception ex)
        {

        }

but it does not work

Answer 1

For your above questions you really need to look these articles which talks about how to use SQL Azure Federation:

Introduction to Fan-out Queries for Federations in SQL Azure (Part 1): Scalable Queries over Multiple Federation Members, MapReduce Style!

Above you can find Fanout sample tool which has the code that show you how to get data out from Federation(s).

The second part of this sample " Fan-out Querying for Federations in SQL Azure (Part 2): Scalable Fan-out Queries with TOP, ORDER BY, DISTINCT and Other Powerful Aggregates, MapReduce Style! " show how you can run fan-out queries with particular examples.

Try above sample and let us know if you met any problem.

Answer 2

I had discussion with Cihan Biyikoglu from SQL Azure Team and following is what he suggested:

The idea is that you do not have to cache the 'map' of where the data is with federations so you should not have to;

Here is the pseudo app code and the actual sample that does the fanout; you can try out the code in this UI as well.

http://federationsutility-seasia.cloudapp.net/About.aspx

Start at the 
@i=MIN_VALUE (of the data_type or federation key like customerID=-1)
While (@i not null)
{
 Try
 {
    -- switch to the next member
    USE FEDERATION fedname(id=@i)...

    -- Run your transaction here for this member 
    SELECT * FROM table join other_table … WHERE fedkeycolumn >= @i group by … order by …

    -- grab the next value
    SELECT @i=range_high FROM sys.federation_member_distributions 
 }
 Catch ()
 {
           If retry_count < total_retries
                          Exit the loop
 }
}

If you still would like to find out the members you can always run the query in the db that contain federations;

Select * from sys.federation_member_distributions fmd join sys.federations f on fmd.federation_id=f.federation_id 

The problem with this approach is you may miss reading data if there is a split between the time you read the info and finish processing the query.

The above method without caching the 'map' is not prone to that problem. It will catch all members regardless of any repartitioning operations.