Simple select query takes more time in very large table in MySQL database in C# application

Question

I am using a MySQL database in my ASP.NET with C# web application. The MySQL Server version is 5.7 and there is 8 GB RAM in the PC. When I am executing the select query in MySQL database table, it takes more time in execution; a simple select query takes around 42 seconds . Across 1 crorerecord (10 million records) in the table. I have also done indexing for the table. How can I fix this?

The following is my table structure.

CREATE TABLE `smstable_read` (
    `MessageID` int(11) NOT NULL AUTO_INCREMENT,
    `ApplicationID` int(11) DEFAULT NULL,
    `Api_userid` int(11) DEFAULT NULL,
    `ReturnMessageID` varchar(255) DEFAULT NULL,
    `Sequence_Id` int(11) DEFAULT NULL,
    `messagetext` longtext,
    `adtextid` int(11) DEFAULT NULL,
    `mobileno` varchar(255) DEFAULT NULL,
    `deliverystatus` int(11) DEFAULT NULL,
    `SMSlength` int(11) DEFAULT NULL,
    `DOC` varchar(255) DEFAULT NULL,
    `DOM` varchar(255) DEFAULT NULL,
    `BatchID` int(11) DEFAULT NULL,
    `StudentID` int(11) DEFAULT NULL,
    `SMSSentTime` varchar(255) DEFAULT NULL,
    `SMSDeliveredTime` varchar(255) DEFAULT NULL,
    `SMSDeliveredTimeTicks` decimal(28,0) DEFAULT '0',
    `SMSSentTimeTicks` decimal(28,0) DEFAULT '0',
    `Sent_SMS_Day` int(11) DEFAULT NULL,
    `Sent_SMS_Month` int(11) DEFAULT NULL,
    `Sent_SMS_Year` int(11) DEFAULT NULL,
    `smssent` int(11) DEFAULT '1',
    `Batch_Name` varchar(255) DEFAULT NULL,
    `User_ID` varchar(255) DEFAULT NULL,
    `Year_ID` int(11) DEFAULT NULL,
    `Date_Time` varchar(255) DEFAULT NULL,
    `IsGroup` double DEFAULT NULL,
    `Date_Time_Ticks` decimal(28,0) DEFAULT NULL,
    `IsNotificationSent` int(11) DEFAULT NULL,
    `Module_Id` double DEFAULT NULL,
    `Doc_Batch` decimal(28,0) DEFAULT NULL,
    `SMS_Category_ID` int(11) DEFAULT NULL,
    `SID` int(11) DEFAULT NULL,
    PRIMARY KEY (`MessageID`),
    KEY `index2` (`ReturnMessageID`),
    KEY `index3` (`mobileno`),
    KEY `BatchID` (`BatchID`),
    KEY `smssent` (`smssent`),
    KEY `deliverystatus` (`deliverystatus`),
    KEY `day` (`Sent_SMS_Day`),
    KEY `month` (`Sent_SMS_Month`),
    KEY `year` (`Sent_SMS_Year`),
    KEY `index4` (`ApplicationID`,`SMSSentTimeTicks`),
    KEY `smslength` (`SMSlength`),
    KEY `studid` (`StudentID`),
    KEY `batchid_studid` (`BatchID`,`StudentID`),
    KEY `User_ID` (`User_ID`),
    KEY `Year_Id` (`Year_ID`),
    KEY `IsNotificationSent` (`IsNotificationSent`),
    KEY `isgroup` (`IsGroup`),
    KEY `SID` (`SID`),
    KEY `SMS_Category_ID` (`SMS_Category_ID`),
    KEY `SMSSentTimeTicks` (`SMSSentTimeTicks`)
) ENGINE=MyISAM AUTO_INCREMENT=16513292 DEFAULT CHARSET=utf8;

The following is my select query:

SELECT messagetext, SMSSentTime, StudentID, batchid,
User_ID,MessageID,Sent_SMS_Day, Sent_SMS_Month,
Sent_SMS_Year,Module_Id,Year_ID,Doc_Batch
FROM smstable_read
WHERE StudentID=977 AND SID = 8582 AND MessageID>16013282

Answer 1

You need to learn about compound indexes and covering indexes. Read about those things.

Your query is slow because it's doing a half-scan of the table. It uses the primary key to find the first row with a qualifying MessageID , then looks at every row of the table to find matching rows.

Your filter criteria are StudentID = constant , SID = constant AND MessageID > constant . That means you need those three columns, in that order, in an index. The first two filter criteria will random-access your index to the correct place. The third criterion will scan the index starting right after the constant value in your query. It's called an Index Range Scan operation, and it's quite efficient.

ALTER TABLE smstable_read
  ADD INDEX StudentSidMessage (StudentId, SID, MessageId);

This compound index should make your query efficient. Notice that in MyISAM, the primary key column of a table should appear in compound indexes. That's cool in this case because it's also part of your query criteria.

If this query is used very frequently, you could make a covering index: you could add the other columns of the query (the ones mentioned in your SELECT clause) to the index.

But, unfortunately you have defined your messageText column with a longtext data type. That allows for each message to contain up to four gigabytes. (Why? Is this really SMS data? There's a limit of 160 bytes per message in SMS. Four gigabytes >> 160 bytes.)

Now the point of a covering index is to allow the query to be satisfied entirely from the index, without referring back to the table. But when you include a longtext or any other LOB column in an index, it only contains a subset of the data. So the point of the covering index is lost.

If I were you I would change my table so messageText was a VARCHAR(255) data type, and then create this covering index:

ALTER TABLE smstable_read
  ADD INDEX StudentSidMessage (StudentId, SID, MessageId,
            SMSSentTime, batchid,
            User_ID, Sent_SMS_Day, Sent_SMS_Month,
            Sent_SMS_Year,Module_Id,Year_ID,Doc_Batch,
            messageText);

(Notice that you should put variable-length items last in the index if you can.)

If you can't change your application to handle VARCHAR(255) then go with the first index I mentioned.

Pro tip: putting lots of single-column indexes on MySQL tables rarely helps SELECT performance and always harms INSERT and UPDATE performance. You need an index on your primary key, and you need indexes to support the queries you run. Extra indexes are harmful.

Answer 2

It looks like your database is not properly indexed and even not properly normalized. Normalizing your database will go a long way to speed up all your queries. Particularly in view of the fact that mysql used only one index per table in a query. Even though you have lot's of indexes, they cannot be used.

Your current query filters on StudentID , SID , and MessageID . The last is an inequality comparision so an index will not be very effective with that but the other two columns are equality comparisons. I suggest an index like this:

KEY `studid` (`StudentID`,`SID`)

Follow that up by dropping your existing index on SID . If you find that you don't want to drop it because it's used in another query, further evidence that your table is in desperate need of normalization.

Too many indexes slow down inserts and adds a little overhead to each SELECT because the query planner needs more effort to figure out which index to use.