无法让我的查询在具有2M条目的MySQL数据库上更快地运行

Question

I have this payments table, with about 2M entries

CREATE TABLE IF NOT EXISTS `payments` (
    `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
    `user_id` int(11) unsigned NOT NULL,
    `date` datetime NOT NULL,
    `valid_until` datetime NOT NULL,
     PRIMARY KEY (`id`),
     KEY `date_id` (`date`,`id`),
     KEY `user_id` (`user_id`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 AUTO_INCREMENT=2113820 ;

and this users table from ion_auth plugin/library for CodeIgniter, with about 320k entries

CREATE TABLE IF NOT EXISTS `users` (
    `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
    `ip_address` varbinary(16) NOT NULL,
    `username` varchar(100) NOT NULL,
    `password` varchar(80) NOT NULL,
    `salt` varchar(40) DEFAULT NULL,
    `email` varchar(100) NOT NULL,
    `activation_code` varchar(40) DEFAULT NULL,
    `forgotten_password_code` varchar(40) DEFAULT NULL,
    `forgotten_password_time` int(11) unsigned DEFAULT NULL,
    `remember_code` varchar(40) DEFAULT NULL,
    `created_on` int(11) unsigned NOT NULL,
    `last_login` int(11) unsigned DEFAULT NULL,
    `active` tinyint(1) unsigned DEFAULT NULL,
    `first_name` varchar(50) DEFAULT NULL,
    `last_name` varchar(50) DEFAULT NULL,
    `company` varchar(100) DEFAULT NULL,
    `phone` varchar(20) DEFAULT NULL,
     PRIMARY KEY (`id`),
     KEY `name` (`first_name`,`last_name`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 AUTO_INCREMENT=322435 ;

I'm trying to get both the user information and his last payment. Ordering(ASC or DESC) by ID, first and last name, the date of the payment, or the payment expiration date. To create a table showing users with expired payments, and valid ones

I've managed to get the data correctly, but most of the time, my queries take 1+ second for a single user, and 40+ seconds for 30 users. To be honest I have no idea if it's possible to get the information under 1 second. Also probably my application is never going to reach this number of entries, probably a maximum of 10k payments and 300 users

My query, works pretty well with few entries and it's easy to change the ordering:

SELECT users.id, users.first_name, users.last_name, users.email, final.id AS payment_id, payment_date, final.valid_until AS payment_valid_until 
FROM users 
LEFT JOIN ( 
    SELECT * FROM ( 
        SELECT payments.id, payments.user_id, payments.date AS payment_date, payments.valid_until 
        FROM payments 
        ORDER BY payments.valid_until DESC 
        ) AS p GROUP BY p.user_id
) AS final ON final.user_id = users.id 
ORDER BY id ASC 
LIMIT 0, 30"

Explain:

id  select_type         table               type              possible_keys   key       key_len   ref    rows      Extra
1   PRIMARY             users               ALL               NULL            NULL      NULL      NULL   322269    Using where; Using temporary; Using filesort
1   PRIMARY             <derived2>          ALL               NULL            NULL      NULL      NULL   50 
4   DEPENDENT SUBQUERY  users_deactivated   unique_subquery   user_id         user_id   4         func   1         Using index
2   DERIVED             <derived3>          ALL               NULL            NULL      NULL      NULL   2072327   Using temporary; Using filesort
3   DERIVED             payments            ALL               NULL            NULL      NULL      NULL   2072566   Using filesort

I'm open to any suggestions and tips, since I'm new to PHP, MySQL and stuff, and don't really know if I'm doing the correct way

Answer 1

I would first suggest removing the ORDER BY clause from your subquery -- I don't see how it's helping as you're reordering by id in your outer query.

You should also be able to move your GROUP BY statement into your subquery:

SELECT users.id, users.first_name, users.last_name, users.email, final.id AS payment_id, payment_date, final.valid_until AS payment_valid_until 
FROM users 
    LEFT JOIN ( 
        SELECT payments.id, payments.user_id, payments.date AS payment_date, payments.valid_until 
        FROM payments 
        GROUP BY payments.user_id
    ) AS final ON final.user_id = users.id 
ORDER BY users.id ASC 
LIMIT 0, 30

---

Given your comments, how about this -- not sure it would be better than your current query, but ORDER BY can be expensive:

SELECT users.id, users.first_name, users.last_name, users.email, p.id AS payment_id, p.payment_date, p.valid_until AS payment_valid_until 
FROM users 
    LEFT JOIN payments p ON p..user_id = users.id 
    LEFT JOIN ( 
        SELECT user_id, MAX(valid_until) Max_Valid_Until
        FROM payments 
        GROUP BY user_id
    ) AS maxp ON p.user_id = maxp.user_id and p.valid_until = maxp.max_valid_until
ORDER BY users.id ASC 
LIMIT 0, 30

Answer 2

The problem with joining to a sub query is that MySql internally generates the result of the sub query before performing the join. This is expensive in resources and is probably taking the time. Best solution is to change the query to avoid sub queries.

SELECT users.id, users.first_name, users.last_name, users.email, max(payments.id) AS payment_id, max(payments.date) as payment_date, max(payments.valid_until) AS payment_valid_until 
FROM users 
LEFT JOIN payments use index (user_id) on payments.user_id=users.id
group by users.id
ORDER BY id ASC 
LIMIT 0, 30

This query is only correct , however, if the largest values for valid_until, payment_date and payment_date are always in the same record.

Answer 3

SELECT payments.users_id, users.first_name, users.last_name,
    users.email, (final.id), MAX(payment.date), MAX(final.valid_until) 
FROM payments final
JOIN users ON final.user_id = users.id
GROUP BY final.user_id
ORDER BY final.user_id ASC
LIMIT 0, 30

The idea is to flatten the payments first. The MAX fields of course are of different payment records.

---

Speed up

Above I did a MySQL specific thing: final.id without MAX. Better not use the field at all.

If you could leave out the payments.id, it would be faster (with the appropiate index).

 KEY `user_date` (`user_id`, `date` DESC ),
 KEY `user_valid` (`user_id`, `valid_until` DESC ),

Answer 4

Maybe something like this...

SELECT u.id
     , u.first_name
     , u.last_name
     , u.email
     , p.id payment_id
     , p.payment_date
     , p.payment_valid_until 
  FROM users u
  JOIN payments p
    ON p.user_id = u.id
  JOIN 
     ( SELECT user_id,MAX(p.valid_until) max_valid_until FROM payments GROUP BY user_id ) x
    ON x.user_id = p.user_id
   AND x.may_valid_until = p.valid_until;

Answer 5

use an index on the payments table for users, that and do the group by on the payments table...

alter table payments add index (user_id);

your query

ORDER BY users.id ASC 
alter table payments drop index user_id;

and why don't you use the payments "id" instead of "valid_until" ? Is there a reason to not trust the ids are sequential? if you don't trust the id add index to the valid_until field:

alter table payments add index (valid_until) desc;

and don't forget to drop it later

alter table payments drop index valid_intil;

---

if the query is still slow you will need to cache the results... this means you need to improve your schema, here is a suggestion:

create table last_payment
(user_id int,
constraint pk_last_payment primary key user_id references users(id),
payment_id int,
constraint fk_last_payment foreign key payment_id references payments(id)
);

alter table payments add index (user_id);

insert into last_payment (user_id, payment_id)
(select user_id, max(id) from payments group by user_id);
#here you probably use your own query if the max (id) does not refer to the last payment...

alter table payments drop index user_id;

and now comes the magic:

delimiter |

CREATE TRIGGER payments_trigger AFTER INSERT ON payments
  FOR EACH ROW BEGIN
    DELETE FROM last_payment WHERE user_id = NEW.user_id;
    INSERT INTO last_payment (user_id, payment_id) values (NEW.user_id, NEW.id);
  END;
|

delimiter ;

and now every-time you want to know the last payment made you need to query the payments_table.

select u.*, p.* 
    from users u inner join last_payment lp on (u.id = lp.user_id)
       inner join payments on (lp.payment_id = p.id) 
    order by user_id asc;