MySql选择查询。 4百万行的响应时间> 3分钟

Question

Agree there are a few mathematical calculations in the select query but surely not something which should affect the performance in such a way.

Below is the select query.

SELECT `p`.`id` as post_id, `p`.`description` as description, `p`.`rent` as rent, 
`p`.`created_at` as created_at, `p`.`title` as title, 
UNIX_TIMESTAMP(p.created_at) as timestamp,
`p`.`user_id` as post_user_id, `p`.`bathrooms`, `p`.`bedrooms`, `p`.`created_at`, 
`p`.`address`, `p`.`lat`, `p`.`lng`, `p`.`posted_by`, `p`.`amenities`, `p`.`user_id`, 
`p`.`smoking_policy`, `p`.`sqft`, `p`.`dogs`, `p`.`cats`, `p`.`dwelling_type`,
`p`.`deposit`, 
`p`.`furnished`, `p`.`sublease`, `p`.`sublease_duration`, `p`.`lease`,          
`p`.`property_type`,`p`.`source`, `p`.`images_json`, `sub`.`name` as sub_category_name,   
`sub`.`id` as sub_category_id, `sub`.`text` as sub_category_text, `p`.`lat` as lat, 
`p`.`lng` as lng, `p`.`phone` as phone, 
(3959 * acos( cos( radians(42.3584308) ) * cos( radians( p.lat ) ) * cos( radians(  
 p.lng ) - radians(-71.0597732) ) + sin( radians(42.3584308) ) * sin( radians( p.lat ) 
) ) ) AS distance
FROM (`T1` p)
JOIN `sub_categories` as sub ON `sub`.`id` = `p`.`sub_category_id`
AND `p`.`lng` between (-71.0597732 - 20/abs(cos(radians(42.3584308 ))*69)) 
and (-71.0597732 + 20/abs(cos(radians(42.3584308))*69)) 
AND `p`.`lat` between 42.3584308 - (20/69) and 42.3584308 + (20/69)
AND `rent` <= '9200'
AND `rent` >= '7000'
AND `bedrooms` <= '4'
AND `bathrooms` <= '3'
AND `dogs` =  '1'
AND `p`.`sub_category_id` =  '2'
HAVING `distance` <= '100'
ORDER BY `p`.`created_at` desc
LIMIT 0,12;

The search should provide available listings within a periphery of input address (lat, long coordinates).

AND condition parameters (rent, bedrooms etc…) and associated values are dynamically assigned based on front end selection.

Table structure is herewith.

CREATE TABLE `T1` (
`id` varchar(40) NOT NULL DEFAULT '',`user_id` varchar(100) NOT NULL DEFAULT '',
`sub_category_id` bigint(20) NOT NULL,
`description` text,`title` text,
`rent` int(11) DEFAULT NULL,
`utilities` int(11) DEFAULT NULL,
`bathrooms` float DEFAULT NULL,
`bedrooms` int(11) DEFAULT NULL,
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`address` varchar(100) DEFAULT NULL,
`lat` double DEFAULT NULL,`lng` doubleDEFAULT NULL,
`dwelling_type` varchar(40) DEFAULT NULL,
`furnished` varchar(20) DEFAULT NULL,
`lease_transfer_fees` int(10) DEFAULT NULL,
`dogs` int(11) DEFAULT NULL,
`cats` int(11) DEFAULT NULL,
`parking_spots` int(10) DEFAULT NULL,
`smoking_policy` varchar(5) DEFAULT NULL,
`deposit` varchar(20) DEFAULT NULL,
`sqft` bigint(20) DEFAULT NULL,
`posted_by` varchar(20) DEFAULT NULL,
`amenities` varchar(500) DEFAULT NULL,
`sublease` varchar(20) DEFAULT NULL,
`sublease_duration` int(11) DEFAULT NULL,
`lease` varchar(20) DEFAULT NULL,
`external_id` varchar(40) DEFAULT NULL,
`source` varchar(10) DEFAULT 'np',
`anchor` varchar(40) DEFAULT NULL,
`property_type` varchar(40) DEFAULT NULL,
`deleted` tinyint(1) DEFAULT '0',
`images_json` text,
`phone` varchar(20) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `user_id_index` (`user_id`),
KEY `filter_combined_index` (`created_at`,`lat`,`lng`,`sub_category_id`,`rent`,     
`bedrooms`,`bathrooms`,`sqft`,`dogs`,`cats`),
KEY `sub_category_id` (`sub_category_id`),
FULLTEXT KEY `text_search_index`    
(`title`,`description`,`smoking_policy`,`posted_by`,`dwelling_type`)
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1;

The explain statement result is below.

id  select_type table   type    possible_keys   key             key_len    ref     rows      Extra
1   SIMPLE      sub     const   PRIMARY,id      PRIMARY         8          const    1      Using filesort
1   SIMPLE      p       ref     sub_category_id sub_category_id 8          const    188122  Using where

Is the table structure not efficient or the select query or mixture of both?

Surely 4m rows should not be a limiting factor. Thanks in advance for advice of resident experts.

TA!

Answer 1

You basically calculate the distance for every row that exists on the table:

SELECT [...]
(3959 * acos( cos( radians(42.3584308) ) * cos( radians( p.lat ) ) *
cos( radians(    p.lng ) - radians(-71.0597732) ) + sin(
radians(42.3584308) ) * sin( radians( p.lat )  ) ) ) AS distance
[...]
HAVING `distance` <= '100'

That forces MySQL to read the complete table on every query.

Additionally, the only index that includes coordinates is not usable for searches because it starts with created_at :

KEY `filter_combined_index` (`created_at`,`lat`,`lng`,`sub_category_id`,`rent`,
`bedrooms`,`bathrooms`,`sqft`,`dogs`,`cats`),

You can try a simple per-coordinate search, with an appropriate index:

WHERE lat BETWEEN :lat_from AND :lat_to
and lng BETWEEN :lng_from AND :lng_to

... where those from and two values belong to the bounding square. Once you've identified only the potential matches, you can fine-tune the results with an actual circle.

Answer 2

You are asking the database to do a lot of work here, but I'd also suggest restructuring your query slightly?

Firstly, you're using a join, but you're not explicitly using a WHERE clause, so you're actually specifying a very big JOIN condition. Internally there's a good chance MySql will automatically figure out that it's actually just a WHERE clause, but given that none of the rows in that join seem to have any bearing on the join itself, they should probably be in their own WHERE . This might make a big difference as it will theoretically reduce the number of rows before doing the join.

Secondly, you're using a HAVING clause, yet you don't have any aggregation in the query. General rule of thumb is that the HAVING clause is used for aggregation (eg COUNT or AVG), and the WHERE clause is used everywhere else.

As @Joachim and @LHristov have both touched on, doing those calculations at query time might not be such a great idea. You're already asking for a lot of data, but you're now asking it do run the calculation for every row it finds, and then run a separate calculation for the join. Unfortunately you've said you can't persist this so that can't be solved, but @Álvaro's suggesting might improve things if the following changes don't

Restructure the query to use WHERE instead of the JOIN and remove the having. I'd expect a resulting query to look like the following

SELECT 
  `p`.`id` as post_id, 
, `p`.`description` as description, 
, `p`.`rent` as rent
, `p`.`created_at` as created_at
, `p`.`title` as title
, UNIX_TIMESTAMP(p.created_at) as timestamp
, `p`.`user_id` as post_user_id
, `p`.`bathrooms`
, `p`.`bedrooms`
, `p`.`created_at`
, `p`.`address`
, `p`.`lat`
, `p`.`lng`
, `p`.`posted_by`
, `p`.`amenities`
, `p`.`user_id`,
, `p`.`smoking_policy`
, `p`.`sqft`
, `p`.`dogs`
, `p`.`cats`
, `p`.`dwelling_type`
, `p`.`deposit`
, `p`.`furnished`
, `p`.`sublease`
, `p`.`sublease_duration`
, `p`.`lease`
, `p`.`property_type`
, `p`.`source`
, `p`.`images_json`
, `sub`.`name` as sub_category_name
, `sub`.`id` as sub_category_id
, `sub`.`text` as sub_category_text
, `p`.`lat` as lat
, `p`.`lng` as lng
, `p`.`phone` as phone,
, (3959 * acos( cos( radians(42.3584308) ) * cos( radians( p.lat ) ) * cos( radians(  

p.lng ) - radians(-71.0597732) ) + sin( radians(42.3584308) ) * sin( radians( p.lat ) ) ) ) AS distance FROM ( T1 p) JOIN sub_categories as sub ON sub . id = p . sub_category_id AND p . sub_category_id = '2' WHERE ( p . lng BETWEEN (-71.0597732 - 20/abs(cos(radians(42.3584308 ))*69)) AND (-71.0597732 + 20/abs(cos(radians(42.3584308))*69)) ) AND ( p . lat BETWEEN (42.3584308 - (20/69)) and (42.3584308 + (20/69)) ) AND rent <= 9200 AND rent >= 7000 AND bedrooms <= 4 AND bathrooms <= 3 AND dogs = 1 AND distance <= 100 ORDER BY p . created_at desc LIMIT 0,12;

Hopefully as mentioned before, this will cause the number of rows that the calculation is performed on to be reduced considerably before doing any calculation and JOIN , whereas just having the JOIN may be causing it to return all rows with calculation before checking if they join. MUCH slower as you can imagine

I also just noticed that you're selecting a few of the columns multiple times such as created_at and user_id ? Not sure if intentional or not but can make a minor difference.

Also, the where clause conditions for fields such as bedrooms , rent , dogs etc that are integers, are all being compared as if they're strings? I've changed that in the query above.

Answer 3

This is a comment, taking advantage of the answer window formatting options.

FWIW, I find this easier to read... and I'd bind distance as a function...

SELECT p.id post_id
     , p.description 
     , p.rent 
     , p.title
     , p.user_id post_user_id
     , p.bathrooms
     , p.bedrooms
     , p.created_at
     , p.address
     , p.posted_by
     , p.amenities
     , p.smoking_policy
     , p.sqft
     , p.dogs
     , p.cats
     , p.dwelling_type
     , p.deposit
     , p.furnished
     , p.sublease
     , p.sublease_duration
     , p.lease
     , p.property_type
     , p.source
     , p.images_json
     , sub.name sub_category_name
     , sub.id sub_category_id
     , sub.text sub_category_text
     , p.lat 
     , p.lng 
     , p.phone 
     , my_distance_function(p.lat,p.lng,71.0597732,42.3584308) distance
  FROM T1 p
  JOIN sub_categories sub
    ON sub.id = p.sub_category_id     
 WHERE my_distance_function(p.lat,p.lng,71.0597732,42.3584308) <= 100
   AND p.lng BETWEEN -71.452028 AND -70.6675175
   AND p.lat BETWEEN 42.0685757 AND 42.6482859
   AND rent <= 9200     
   AND rent >= 7000
   AND bedrooms <= 4
   AND bathrooms <= 3
   AND dogs =  1
   AND p.sub_category_id = 2

 ORDER 
    BY p.created_at DESC
 LIMIT 0,12;