Issue looping through a DataTable Rows multiple times

Question

::EDIT::

Ok folks, it seems that I'm an idiot after all. The problem had nothing at all to do with my code, and everything to do with Visual Studio having overwritten my SQLite Database with a previous (and empty) version. It does seem that there is a great discussion about thread safety going on, so I will stick around to read more!

::/EDIT::

I am attempting to use multiple background workers to loop through the rows of a database 100 records at a time while avoiding duplication, but I seem to be having some issues. Basically, I start off by creating 10 background workers in a loop, and adding them to a List. Then I loop through the background workers in the List, and for each one I do RunWorkerAsync(), and then sleep the main thread for 5 seconds. Inside the DoWork method of each background worker, I have the worker select 100 rows from the database where a particular field is set to its default value. From here, I want to first loop through every returned row and change that default value to an "In Progress" value, and then loop through the values again and actually do the processing required to find correct values for those fields. The problem I seem to be having is that I seem to be have an empty DataTable after the first iteration through the results, and I suspect that my issues stem from shallow copying. Here is the code:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.IO;
using System.Text;
using DBFill.GeoCodeService;
using System.Diagnostics;
using System.Runtime.Serialization.Formatters.Binary;

namespace DBFill {
    class Program {
        public static int completedGeocodes = 0;
        static void Main(string[] args) {
            SQLiteDatabase db = new SQLiteDatabase("zipCodes.s3db");
            List<BackgroundWorker> workers = new List<BackgroundWorker>();
            for (int i = 0; i < 10; i++) {
                BackgroundWorker b = new BackgroundWorker();
                b.DoWork += new DoWorkEventHandler(worker_DoWork);
                b.RunWorkerCompleted += new RunWorkerCompletedEventHandler(worker_RunWorkerCompleted);
                b.WorkerReportsProgress = true;
                b.ProgressChanged += new ProgressChangedEventHandler(b_ProgressChanged);
                workers.Add(b);
            }
            int counter = 0;
            foreach (BackgroundWorker b in workers) {
                Debug.WriteLine("Worker {0} is starting.", counter);
                b.RunWorkerAsync(b);
                counter++;
                System.Threading.Thread.Sleep(5000);
            }
            Boolean running = true;
            while (running) {
                running = false;
                foreach (BackgroundWorker b in workers) {
                    Debug.WriteLine("Checking background Worker");
                    if (b.IsBusy) {
                        running = true;
                    }
                }
                System.Threading.Thread.Sleep(5000);
            }

        }

        static void b_ProgressChanged(object sender, ProgressChangedEventArgs e) {
            Console.WriteLine(".");
        }

        static void worker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) {

        }

        static void worker_DoWork(object sender, DoWorkEventArgs e) {
            BackgroundWorker b = (BackgroundWorker)e.Argument;
            SQLiteDatabase db = new SQLiteDatabase("zipCodes.s3db");
            DataTable results = get100Records();
            DataTable temp = DeepClone<DataTable>(results);//results;
            Dictionary<String, String> marker = new Dictionary<string, string>();
            marker["LATITUDE"] = "In Progress";
            foreach (DataRow row in temp.Rows) {
                Debug.WriteLine("Working with zip {0}", row["ZIP_CODE"]);
                db.Update("ZIP_CODES", marker, String.Format("ZIP_CODE = '{0}'", row["ZIP_CODE"]));
            }
            foreach (DataRow row in results.Rows) {
                String geoCodeResponse = GeoCodeZip(row["ZIP_CODE"].ToString());
                Debug.WriteLine(String.Format("Attempting Zip: {0}", row["ZIP_CODE"].ToString()));
                if (geoCodeResponse != "There was an error") {
                    marker["LATITUDE"] = geoCodeResponse.Split(',')[0];
                    marker["LONGITUDE"] = geoCodeResponse.Split(',')[1];
                    Console.WriteLine(String.Format("#{0} updated successfully", completedGeocodes));
                }
                else {
                    marker["LATITUDE"] = "Not Set";
                    Console.WriteLine(String.Format("#{0} failed", completedGeocodes));
                }
                db.Update("ZIP_CODES", marker, String.Format("ZIP_CODE = '{0}'", row["ZIP_CODE"]));
                db.ExecuteNonQuery("commit");
                b.ReportProgress(1);
                completedGeocodes++;
            }
        }

        private static DataTable get100Records() {
            SQLiteDatabase db = new SQLiteDatabase("zipCodes.s3db");
            DataTable results = db.GetDataTable("select ZIP_CODE from ZIP_CODES where LATITUDE = 'Not Set' LIMIT 100");
            return results;
        }

        private static String GeoCodeZip(String zip) {
            try {
                GeocodeRequest request = new GeocodeRequest();
                request.Credentials = new GeoCodeService.Credentials();
                request.Credentials.ApplicationId = "API_KEY";
                request.Query = zip;
                ConfidenceFilter[] filters = new ConfidenceFilter[1];
                filters[0] = new ConfidenceFilter();
                filters[0].MinimumConfidence = Confidence.High;
                GeocodeOptions opts = new GeocodeOptions();
                opts.Filters = filters;
                request.Options = opts;
                GeocodeServiceClient service = new GeocodeServiceClient("BasicHttpBinding_IGeocodeService");
                GeocodeResponse response = service.Geocode(request);
                if (response.Results.Length > 0) {
                    return String.Format("{0},{1}", response.Results[0].Locations[0].Latitude, response.Results[0].Locations[0].Longitude);
                }
                else {
                    Debug.WriteLine(String.Format("{0}", response.ResponseSummary.FaultReason));
                    return "There was an error";
                }
            }
            catch (Exception e) {
                Debug.WriteLine(e.Message);
                return "There was an error";
            }
        }

        public static T DeepClone<T>(T obj) {
            using (var ms = new MemoryStream()) {
                var formatter = new BinaryFormatter();
                formatter.Serialize(ms, obj);
                ms.Position = 0;

                return (T)formatter.Deserialize(ms);
            }
        }

    }
}

Any Ideas?

Answer 1

It seems that your delay and why you want to multi-thread it, is not to read records from the database, but rather calling out to the GeocodeServiceClient .

You could try reworking your main method to fetch all the records from the DB sequentially and parse them. You then split that list into even chunks and spin up background workers to run them through the Geocode service.

Another option would be to put the records into a queue and have each background worker pop one off the queue work on it, then go back to the queue while there are still unprocessed records. You will need to be careful of locking In C# would it be better to use Queue.Synchronized or lock() for thread safety? .