Oracle Connection and TCP Client/Server Connection at same time

Question

I am writing a program that will live on a "Super PC" in my lab at work. Its job is to constantly query our customers databases proactively looking for common errors that we encounter.

It accomplishes this by using an adjustable timer and simply running down the list of queries and databases and interpreting the results.(Queries and Database connections are added using a Configuration UI)

This program has a TCP client/server connection with another app that I have written that lives on my team members personal machines. Messages get sent from the Server(Query) program to the client program alerting my team of errors found in the Databases.

The problem I keep encountering is occasionally a message gets sent through the socket at the exact same time a DB connection is made or a query is run and it causes the server program to crash with no explanation.

The methods that run the queries is always called in its own thread as well as the server connection is made in its own thread.

In addition I have placed all of the DB methods and Client Server methods in Try/Catch blocks that are setup to catch all exceptions and display the stack trace, but for some reason when it crashes its not hitting any of the catch.

The runQueries Methods is called off a C# .Net System.Timers.Timer which is supposed to launch a new thread on every tick.

Below are my Client/Server Methods and Query Methods

    private void serverMeth()
    {
        try
        {
            /*conbox.Text = conbox.Text + "The server is running at port 8001..." + Environment.NewLine;
            conbox.Text = conbox.Text + "The local End point is  :" + myList.LocalEndpoint + Environment.NewLine;
            conbox.Text = conbox.Text + "Waiting for a connection....." + Environment.NewLine;*/

            msglist.Add("The server is running at port 8001...");
            msglist.Add("The local end point is: " + myList.LocalEndpoint);
            msglist.Add("Waiting for a connection...");
            writeToLog("Listening for connection at "+myList.LocalEndpoint);

            /* Start Listeneting at the specified port */

            while (true)
            {
                gatherErrors();
                myList.Start();

                //Socket s = myList.AcceptSocket();
                s = myList.AcceptSocket();
                //conbox.Text = conbox.Text + "Connection accepted from " + s.RemoteEndPoint + Environment.NewLine;
                msglist.Add("Connection accepted from " + s.RemoteEndPoint);
                writeToLog("Connection Established @" + myList.LocalEndpoint);

                byte[] b = new byte[20000];
                int k = s.Receive(b);

                    //conbox.Text = conbox.Text + "Recieved..." + Environment.NewLine;
                    msglist.Add("Recieved...");
                writeToLog("Message Recieved from Command Center");
                //for (int i = 0; i < k; i++)
                //{ conbox.Text = conbox.Text + Convert.ToChar(b[i]); }

                string message = ""; //allows it to store the entire message in one line
                string dt = DateTime.Now.ToString("MM-dd-yyyy hh:mm:ss");
                string eid = "TEST ID";

                for (int i = 0; i < k; i++)
                { message = message + Convert.ToChar(b[i]); }

                if (message.Contains(uniquekey))
                {
                    writeToLog("Message contains correct passcode");
                    message = message.Replace(uniquekey, "");
                    String[] splits = message.Split(',');
                    message = message.Replace("," + splits[1], "");
                    writeToLog("Connection from " + splits[1]);
                    msglist.Add(message); //this takes the completed message and adds it.
                    //DO IGNORE STUF HERE
                    if (!message.Equals(""))
                    {
                        //Message contains error key
                        Error erro = null;
                        for (int i = 0; i < errorss.Count; i++)
                        {
                            if (errorss[i].getKey().Equals(message)) {
                                erro = errorss[i];
                            }
                        }
                        Stage st = null;
                        if (erro != null)
                        {

                            for (int i = 0; i < stages.Count; i++)
                            {
                                if (stages[i].errContainCheck(erro))
                                {
                                    st = stages[i];
                                }
                            }
                        }
                        if (st != null)
                        {
                            st.addIgnore(erro);
                            writeToLog("Error: " + erro.getKey() + "ignored");
                        }
                    }

                    //conbox.Text = conbox.Text + Environment.NewLine;
                    msglist.Add(" ");
                    string msg = "";
                    string log = "Error(s): ";
                    for (int i = 0; i < errorss.Count; i++)
                    {
                        msg += errorss[i].getTime() + "|"+errorss[i].getMsg()+"|" + errorss[i].getSite()+"|"+errorss[i].getKey();
                        msg += ",";
                        log += errorss[i].getKey()+",";
                    }
                    log+= "sent to Command Center";
                        ASCIIEncoding asen = new ASCIIEncoding();
                        s.Send(asen.GetBytes(msg));
                        //conbox.Text = conbox.Text + "\nSent Acknowledgement" + Environment.NewLine;
                        msglist.Add("\nSent Acknowledgement");
                    writeToLog(log);

                }
                else
                {
                    message = "Unauthorized access detected. Disregarding message.";
                    //.Add(message); //this takes the completed message and adds it.
                    //conbox.Text = conbox.Text + Environment.NewLine;
                    msglist.Add(" ");
                    writeToLog("Passcode mismatch");

                    ASCIIEncoding asen = new ASCIIEncoding();
                    s.Send(asen.GetBytes("Access Denied.  Unique key mismatch."));
                    //conbox.Text = conbox.Text + "\nSent Acknowledgement" + Environment.NewLine;
                    msglist.Add("\nSent Denial Acknowledgement");
                }


                /* clean up */
                s.Close();
                myList.Stop();
                if (quit == true)
                {
                    break;
                }

            }
        }
        catch (Exception err)
        {
            //conbox.Text = conbox.Text + "Error..... " + err.StackTrace + Environment.NewLine;
            msglist.Add("Error..... " + err.StackTrace);
            writeToLog("ERROR: "+err.StackTrace);
        }
    }

    private void sasDownload()
    {
        int selItemList;
        selItemList = saerrlist.SelectedIndex;

        saerrlist.Items.Clear();  //clears the list before redownload
        saerrgrid.Rows.Clear();

        try
        {
            TcpClient tcpclnt = new TcpClient();
            //clibox.Text = clibox.Text + "Connecting....." + Environment.NewLine;

            tcpclnt.Connect(staralertip.Text, 8001);
            // use the ipaddress as in the server program

            //clibox.Text = clibox.Text + "Connected" + Environment.NewLine;
            //clibox.Text = clibox.Text + "Enter the string to be transmitted : " + Environment.NewLine;


            String str = "982jsdf293jsadd02jkdas20dka2";
            Stream stm = tcpclnt.GetStream();
            if (ackClick)
            {
                str += ackKey;
                ackClick = false;
            }
            String name = "User not found";
            for (int i = 0; i < users.Count(); i++)
            {
                if(users[i].sys_id.Equals(SNLists.loggedinuser)){
                    name = users[i].name;
                    break;
                }
            }
            str += "," + name;
            ASCIIEncoding asen = new ASCIIEncoding();
            byte[] ba = asen.GetBytes(str);
            //clibox.Text = clibox.Text + "Transmitting....." + Environment.NewLine;

            stm.Write(ba, 0, ba.Length);

            byte[] bb = new byte[20000];
            int k = stm.Read(bb, 0, 20000);

            string incmsg = "";

            for (int i = 0; i < k; i++)
                incmsg = incmsg + Convert.ToChar(bb[i]);


            string tempmsg = "";
            foreach (char c in incmsg)
            {
                if (c != ',')
                {
                    tempmsg = tempmsg + c;
                }
                else if (c == ',')
                {
                    saerrlist.Items.Add(tempmsg);
                    saerrgrid.Rows.Add(tempmsg.Split('|')[0], tempmsg.Split('|')[1], tempmsg.Split('|')[2], tempmsg.Split('|')[3]);
                    tempmsg = "";
                }
            }
            saerrlist.Items.Add(tempmsg);
            //saerrgrid.Rows.Add(tempmsg.Split('|')[0], tempmsg.Split('|')[1], tempmsg.Split('|')[2]);


            //MessageBox.Show(incmsg);

            tcpclnt.Close();
        }

        catch (Exception err)
        {
            //MessageBox.Show("Error..... " + err.StackTrace, "STAR Command Center: Connectivity Error");
            staralertTimer.Enabled = false;
            MessageBox.Show("Error downloading recent errors from STAR Alert System.\n\nPlease confirm STAR Alert is running " +
                            "and hit \"Refresh\" in the STAR Alert tab." + "\n" + err, "STAR Command Center: Connectivity Error");
        }

        saerrlist.SelectedIndex = selItemList;
    }



    public void runQueries()
    {
        OracleConnection con = new OracleConnection();
        int siteNum = -1;
        try
        {
            writeToLog("Call to runQueries");
            List<List<Error>> results = new List<List<Error>>();
            for (int i = 0; i < ppreset.getSites().Count(); i++)
            {
                siteNum = i;
                Site s = ppreset.getSites()[i];
                if (s.getStatus().Equals("ACTIVE"))
                {
                    //OracleConnection con = new OracleConnection();
                    String it = "User Id=" + s.getLogin() + ";Password=" + s.getPassword() + ";Data Source=" + s.getDbName();
                    con.ConnectionString = it;
                    //con.OpenAsync();
                    con.Open();
                    writeToLog("Connection opened for site: " + s.getSite());
                    List<Error> subResults = new List<Error>();

                    for (int j = 0; j < s.getQueries().Count(); j++)
                    {
                        Query q = s.getQueries()[j];
                        string sql = q.getCode();
                        List<string> columns = getColumns(sql);
                        List<List<String>> mast = new List<List<String>>();
                        for (int m = 0; m < columns.Count(); m++)
                        {
                            mast.Add(new List<String>());
                        }

                        OracleCommand cmd = new OracleCommand(sql, con);
                        cmd.CommandType = CommandType.Text;
                        OracleDataReader dr = cmd.ExecuteReader();
                        writeToLog("Execute SQL, Begin Reading results...");
                        // dr.Read();
                        //List<String> results = new List<String>();
                        if (dr.HasRows)
                        {
                            //MessageBox.Show("has rows");
                            while (dr.Read())
                            {
                                //string result = "";
                                for (int p = 0; p < columns.Count(); p++)
                                {
                                    if (columns.Count() != 0)
                                    {
                                        mast[p].Add(dr[columns[p]].ToString());
                                    }
                                }
                                //results.Add(result);


                            }
                        }


                        subResults.AddRange(ruleCheck(mast, q.getRules(), s.getQueries()[j], ppreset.getSites()[i].getSite()));
                        cmd.Dispose();
                        writeToLog("Done reading");


                    }

                    results.Add(subResults);
                    // con.Dispose();
                    con.Close();
                    writeToLog("Connection Closed for site: " + s.getSite());
                }

            }
            //we now have all of the error strings gathered from running all queries on all sites in the given preset
            //lets send them to a method that will change the status' of the modules and makem RAVVEEEEE
            //update(results);


            if (errors == null && results.Count != 0)
            {
                //MessageBox.Show("errors = results");
                errors = results;
                writeToLog("First error found...errors = results");
                for (int i = 0; i < results.Count; i++)
                {
                    for (int j = 0; j < results[i].Count; j++)
                    {
                        if (foncig.lallerdisc == true)
                        {
                            sendLyncMsg(results[i][j].getMsg());
                            writeToLog("Lync Msg sent");
                        }
                    }
                }
            }
            else
            {
                for (int i = 0; i < results.Count; i++)
                {
                    for (int j = 0; j < results[i].Count; j++)
                    {

                        errors[i].Add(results[i][j]);
                        writeToLog("Error: " + results[i][j].getKey() + " added");
                        // MessageBox.Show("Error added to errors");
                        //////////////////////////////////////////////////////////////////////////////
                        ////////////////////////////////////////////////////////////////////////////////
                        //LYNC STUFF CAN GO HERE///////////////////////////////////////////////////////
                        ///////////////////////////////////////////////////////////////////////////////
                        /////FOR EACH ERROR ADDED TO MASTER LIST SEND LYNC///////////////////////////
                        //////////////////////////////////////////////////////////////////////////////
                        if (foncig.lallerdisc == true)
                        {
                            sendLyncMsg(results[i][j].getMsg());
                            writeToLog("Lync msg sent");
                        }
                    }
                }
            }
        }
        catch (Exception e)
        {//MessageBox.Show("Err: " + e);
            badConn[siteNum] += "x";
            writeToLog("Connection error strike added to " + ppreset.getSites()[siteNum].getSite());
            con.Close();

            checkForBadConn();

            writeToLog( e.StackTrace);
        }
        //foncig.errors = errors;
        //here we will check all of the errors in results against all of the errors  in errors
        //if the error is not in errors add it
        //errors = results;



    }

Query timer initialization

    queryTimer = new System.Timers.Timer(120000);
        queryTimer.Elapsed += queryTimer_Tick;
        queryTimer.Enabled = true;

Tick method for query timer

    private void queryTimer_Tick(object sender, EventArgs e)
    {
        GC.Collect();
        try
        {
            runQueries();
        }
        catch (OracleException err)
        {
           // MessageBox.Show("Err: " + err);
        }
    }

How the server thread is started

    server = new Thread(serverMeth);
        server.Start();

Does anyone have any idea why this might be happening? Both programs work as described and do what they are supposed to its just when more and more clients begin connecting to the server it becomes more and more likely that a TCP connection and DB connection will occur at the same time.

Update 12:20 9/15/14

So I'm trying to upload a pic of the log from the last crash... but i don't have enough rep points... derp, anyway

This time the last line in the log is from a log purge(deletes log entries older than 24 hours)

However the program will only crash when I have the client program set to auto refresh. IE Client app has a button to request messages from server or user can select auto-refresh which sets request function to timer.

My only other thought is that multiple threads are trying to write to the log file at the same time, however my write to log method uses writeLinesAsync() so that shouldn't be a problem?

Answer 1

This issue turned out to be caused by multiple threads trying to write to the log file at the same time.

I resolved it by following @user469104's advice and creating an internal lock object to lock the writeToLog methods.

Both client and server apps have been running for multiple days now without breaking.

Thanks Everyone!