对 MySQL 子查询或范围内的连接使用索引

Question

I have a list of requests and their respective IP addresses (~2 million rows). I'm trying to do a simple JOIN on a list of non-overlapping and complete list of IP ranges (~12 million rows). I have indexed the IP ranges with ip_from b_tree ascending and ip_to b_tree ascending.

I have tried several techniques for managing to combine the data from these two tables, all have shown to be very inefficient so far.

I have tried regular JOIN , JOIN with maximum difference of IP range and using sub-queries. Using EXPLAIN they have all shown to have possible_keys , without using them. I have tried using FORCE INDEX without any luck.

Regular select separately shows that the IP lookup should take about 2ms with SELECT * FROM ip_ranges WHERE INET_ATON(<some ip>) <= ip_to LIMIT 1; and the request table takes about 16ms for every 200 lookups.

Here is my current query. This takes about 30 seconds to return any results simply because the indexes are not fully utilized:

SELECT 
rs.fingerprint,
rs.ip,
ipr.country_code,
ipr.country_name,
ipr.region,
ipr.city,
ipr.isp_name,
ipr.domain_name,
ipr.usage_type
FROM requests AS rs
JOIN ip_ranges AS ipr ON INET_ATON(rs.ip) BETWEEN ipr.ip_from AND ipr.ip_to
LIMIT 10;

So, is there some way to optimize this for MySQL? Or should I rather just call the database individually for each request using Python? (join them manually outside of SQL).

Update:

I have now tried converting each IP address into their respective numerical format stored in a DECIMAL(39) column called ip_numeric as suggested in the answers below. 39 is used to also support IPv6 addresses. Database still wont use index keys for range lookup.

Answer 1

Because a join can not be optimized on a FUNCTION RESULT (your INET_ATON of the IP address), it will not take advantage of the index.

To correct this, I would do the following... Apply the INET_ATON() of the address before inserting into the requests file. This way, the IP address is already in its properly formatted standard in the file. Do the same for the IP_Ranges (from and to) so they are also in proper pre-confirmed proper format consistency.

Then a join on the ip does not have to get assessed/converted every time before the "between" is applied to the test.

FEEDBACK

Indexes on columns, not functions... No specific documents, just from experience. The index is based on the value of a COLUMN. If you are joining on a function result, it has to run that based on the original column each record. So, by storing the pre-computed final value of the IP, you now HAVE that properly formatted address and the index can run directly on that with no more conversion. Likewise, when populating the JOIN TO table with from/to addresses, you are now pre-forcing your data IN the final format for comparison.

Much like date indexes. Just index on a date field, not a month / year. Then when you run a query and you would want something like for the last month, you would not do a month( someDateColumn ) = 10 and year( someDateColumn ) = 2019. You would just do someDateColumn >= '2019-10-01' and someDateColumn < '2019-11-01'. An index on the date will work faster than the function comparison.

Answer 2

You can add a virtual column to the table and index that:

ALTER TABLE requests ADD ip_numeric bigint GENERATED ALWAYS AS (INET_ATON(ip)) virtual;

CREATE INDEX ip_numeric_ind ON requests (ip_numeric)

Then use that in your query:

SELECT 
rs.fingerprint,
rs.ip,
ipr.country_code,
ipr.country_name,
ipr.region,
ipr.city,
ipr.isp_name,
ipr.domain_name,
ipr.usage_type
FROM requests AS rs
JOIN ip_ranges AS ipr ON ip_numeric BETWEEN ipr.ip_from AND ipr.ip_to
LIMIT 10;

Answer 3

If you can guarantee that the from..to pairs do not overlap, there is a way to significantly speed up such tables. It involves building a table with only ip_from , knowing that the ip_to is one less than the ip_from of the next row.

Discussion, including reference code for IPv4 and IPv6: http://mysql.rjweb.org/doc.php/ipranges

This may be a rare case where CURSORs work faster than trying to do it in a single query.

That is, doing 10 separate lookups with the above technique will be very fast. If you need to do 2M lookups, we need to start over. A thought: Sort the 2M and the 12M; match up as you go. (A la the "merge" part of traditional "sort-merge" algorithms.) (No, I have not thought through the details.)

Answer 4

It can be quite hard to get indexes to use ranges. I recommend the following approach:

• Find the first range where the range ends on or after the given ip.
• Join back in the range table to get the start point
• compare!

As SQL:

select r.*, ir.*
from (select r.*,
             (select ir.ip_to
              from ip_ranges ir
              where ir.ip_to >= inet_aton(r.ip)
              order by ir.ip_to
              limit 1
             ) as range_to
      from requests r
     ) r join
     ip_ranges ir
     on ir.ip_to = r.range_to
where r.ip >= ir.ip_to;

This wants an index on ip_ranges(ip_to) , both for the correlated subquery and the final join .