Optimize MySQL JOIN with subquery

Question

My current MySQL query is very slow. I am looking for ways to optimize it.

I have a table with giveaways for Steam games (12.711 records).

CREATE TABLE IF NOT EXISTS `giveaway` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `creatorid` bigint(20) unsigned NOT NULL,
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `appid` int(10) DEFAULT NULL,
  `packageid` int(10) DEFAULT NULL,
  `gamename` varchar(256) NOT NULL,
  `gametype` varchar(64) NOT NULL,
  `copies` int(3) unsigned NOT NULL,
  `multiplier` float unsigned NOT NULL DEFAULT '1',
  `closed` timestamp NULL DEFAULT NULL,
  `winnerid` bigint(20) unsigned DEFAULT NULL,
  `cancelled` tinyint(1) unsigned DEFAULT NULL,
  `noentry` tinyint(1) unsigned DEFAULT NULL,
  `url` varchar(256) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `winnerid` (`winnerid`),
  KEY `creatorid` (`creatorid`),
  KEY `packageid` (`packageid`),
  KEY `appid` (`appid`),
  KEY `gametype` (`gametype`)
) ENGINE=InnoDB AUTO_INCREMENT=12003 DEFAULT CHARSET=utf8;

INSERT INTO `giveaway` (`id`, `creatorid`, `created`, `appid`, `packageid`, `gamename`, `gametype`, `copies`, `multiplier`, `closed`, `winnerid`, `cancelled`, `noentry`, `url`) VALUES
    (148, 76561198043198608, '2014-08-05 15:41:12', 72200, NULL, 'Universe Sandbox', 'bundle', 1, 1, '2014-08-05 15:41:29', 76561198051609534, NULL, NULL, 'http://www.steamgifts.com/giveaway/ZvyXL/universe-sandbox'),
    (149, 76561197993840952, '2014-08-05 15:41:49', 287860, NULL, '8-Bit Commando', 'bundle', 1, 1, '2014-08-12 18:54:15', 76561198001912161, NULL, NULL, 'http://www.steamgifts.com/giveaway/qlFrc/8-bit-commando'),
    (150, 76561198043198608, '2014-08-05 15:42:09', 212010, NULL, 'Galaxy on Fire 2 Full HD', 'bundle', 1, 1, '2014-08-05 15:43:17', 76561198031159289, NULL, NULL, 'http://www.steamgifts.com/giveaway/eyNT1/galaxy-on-fire-2-full-hd');

I also have a table with Steam games that a user already owns (130.117 records).

CREATE TABLE IF NOT EXISTS `steam_ownedgames` (
  `steamid` bigint(20) unsigned NOT NULL,
  `appid` bigint(20) unsigned NOT NULL,
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  KEY `steamid` (`steamid`),
  KEY `appid` (`appid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO `steam_ownedgames` (`steamid`, `appid`, `last_update`) VALUES
    (76561198112359563, 4000, '2015-05-20 18:03:10'),
    (76561198112359563, 2990, '2015-05-20 18:03:10'),
    (76561198112359563, 12200, '2015-05-20 18:03:10'),
    (76561198112359563, 12210, '2015-05-20 18:03:10'),
    (76561198112359563, 29800, '2015-05-20 18:03:10'),
    (76561198112359563, 9870, '2015-05-20 18:03:10'),
    (76561198112359563, 34900, '2015-05-20 18:03:10'),
    (76561198112359563, 11390, '2015-05-20 18:03:10'),
    (76561198112359563, 23490, '2015-05-20 18:03:10'),
    (76561198112359563, 18820, '2015-05-20 18:03:10'),
    (76561198112359563, 24960, '2015-05-20 18:03:10'),
    (76561198112359563, 43110, '2015-05-20 18:03:10'),
    (76561198112359563, 46410, '2015-05-20 18:03:10'),
    (76561198112359563, 50620, '2015-05-20 18:03:10'),
    (76561198112359563, 70300, '2015-05-20 18:03:10'),
    (76561198112359563, 63800, '2015-05-20 18:03:10');

I want to show all giveaways and be able to indicate if the user already owns it.

I can get this info by comparing the appid from giveaway with the appid from steam_ownedgames for the current user.

Output Example:

giveawayid  owned   appid   gamename
12810       12810   264060  Full Bore
12809       \N      263100  9.03m
12808       12808   107200  Space Pirates and Zombies
12807       12807   231910  Leisure Suit Larry in the Land of the Lounge Lizards: Reloaded
12806       \N      278620  TinyKeep
12805       \N      315430  Polarity
12804       12804   341060  The Lady

I get the info I want with the following query.

SELECT
    giveaway.id as giveawayid,
    owned.id as owned,
    appid,
    gamename
FROM giveaway 
LEFT JOIN
    ( /*
        * query - giveaways with an appid that is in the user's owned games list
        */ 
        SELECT giveaway.id as id
        FROM giveaway
        LEFT JOIN steam_ownedgames
            ON giveaway.appid = steam_ownedgames.appid
        WHERE steam_ownedgames.steamid = 76561197962290563
        GROUP BY giveaway.id
    ) as owned
    ON giveaway.id = owned.id 
ORDER BY created DESC, giveaway.id DESC 
LIMIT 0, 500 

However this one is very slow and takes 27 seconds to complete. And the time is increasing fast with the database growing larger.

Who has advice on how to optimize better?

Answer 1

You didn't need the subselect. It is a bit unintuitive at first, but if you want to join only against a subset, this is the standard way to do it. What this is doing is grabbing all the giveaway records and finding any steam_ownedgames records for the same game (appid) that have a particular user (steamid).

SELECT giveaway.id as giveawayid
   , appid, gamename
   , sog.last_update AS ownedSince
FROM giveaway 
LEFT JOIN steam_ownedgames AS sog
        ON giveaway.appid = sog.appid
        AND sog.steamid = 76561197962290563
ORDER BY created DESC, giveaway.id DESC 
LIMIT 0, 500 
;

It is equivalent to this (below), but allows taking advantage of indices. Actually, this below might be faster under certain circumstances, but it would be highly dependent on the specific data in the tables. Ex: If a user only has 3 games, the unindexed pairing might be faster than the indexed pairing of a game owned by 1000 users.

SELECT giveaway.id as giveawayid
   , appid, gamename
   , sog.last_update AS ownedSince
FROM giveaway 
LEFT JOIN (
   SELECT * 
   FROM steam_ownedgames 
   WHERE steamid = 76561197962290563
  ) AS sog
  ON giveaway.appid = sog.appid
ORDER BY created DESC, giveaway.id DESC 
LIMIT 0, 500 
;

In this example, ownedSince would be null if it were not owned; but any field in steam_owned games should suffice (as long as it cannot normally be null).