如何遍历3500万行的表-MySQL

Question

I have a really large mysql table which stores domains and subdomains. And its create syntax like below

CREATE TABLE `domain` (
  `domain_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `domain_name` varchar(255) COLLATE utf8_turkish_ci NOT NULL,
  PRIMARY KEY (`domain_id`),
  UNIQUE KEY `dn` (`domain_name`),
  KEY `ind_domain_name` (`domain_name`)
) ENGINE=InnoDB AUTO_INCREMENT=78364364 DEFAULT CHARSET=utf8 COLLATE=utf8_turkish_ci;

and it stores values like

domain_id | domain_name

1         | a.example.com

2         | b.example.com 

3         | example.com

4         | facebook.com

5         | a.facebook.com

6         | google.com

I want to find subdomains of any top domain and then I'll match subdomains with its' parent domain.

for example a.example.com and b.example.com are subdomains of example.com so on my new column which named parent_domain_id I will set example.com's domain_id. (if domain is a top domain, it's parent_domain_id will be 0 )

I work with PHP and mysql and my machine has 8GB of RAM so I have some device limitations. Is there a trick about checking a huge data set row by row with PHP?

Answer 1

EDIT: for most domain names this should work.

You could get a sorted list which contains all the domains (up to a fixed length) ordered by domain and increasing length.

I would start with Comment of @RyanVincent.

select domain_id, domain_name, reverse(domain_name) as reversed
from domain
order by 
rpad(reverse(domain_name),130,' '),
length(domain_name),
reverse(domain_name), 
domain_id

Goal is to be able to get row by row ordered by length and then by alphabet.

Your example would give

        google.com -> moc.elgoog, 
      a.google.com -> moc.elgoog.a
   xy.a.google.com -> moc.elgoog.a.yx
      b.google.com -> moc.elgoog.a

In php:

  $currentDomains = array(); 
  /*
     array domainpart => id
     moc     => 0
     elgoog  => id of google.com as long as we are in subdomains of google.com
     a       => id of a.google.com as long as we are in subdomains of a.google.com

     this gets never longer then the number of domainparts, so usually a very
     short array!
   */

  $sql = "select domain_id, domain_name, reverse(domain_name) as reversed\n"
      . " from domain \n"
      . " order by  \n"
      . " rpad(reverse(domain_name),130,' '), \n"
      . " length(domain_name), \n"
      . " reverse(domain_name),  \n"
      . " domain_id"
      ;

  doSelect($sql);

  while($row = getRow()){
    $parts = preg_split('/\./', $row["reversed"]);
    # print("parts = \n");print_r($parts);
    $rid = $row["domain_id"];

    $matchedDomains = array();
    $parentId = $rid; // if no parent, show to yourself

    $i = 0;
    // 1. match identical parts
    //     php is funny, the array is a key=>value but with 
    //     foreach it restores the key-values in the inserted order.
    foreach($currentDomains as $name => $cid){
      # print("> check $i '$name' '{$parts[$i]}'\n");
      if($parts[$i] == $name){
        $matchedDomains[$name] = $cid;
        if($cid > 0){
          $parentId = $cid;
        }
        $i++;
      }
      else{
        break;
      }
    }
    // 2.
    // new parts
    while ($i < count($parts)-1){
      # print("> store '{$parts[$i]}' \n");
      $matchedDomains[$parts[$i]] = 0; // no row matches those
      $i++;
    }
    $matchedDomains[$parts[count($parts)-1]] = $rid;
    $currentDomains = $matchedDomains;
    print(" update domain set parent_id = $parentId where id = $rid\n"); // use PDO
  }

So google.com is its own parent domain, a.google.com has google.com as parent and so on.