用链接替换单词

Question

On my site I have the Catholic Encyclopedia. It has over 11,000 articles.

I'm interested in replacing words and phrases on the articles on my site with links to the relevant entries in the Catholic Encyclopedia. So, if someone says:

St. Peter was the first pope.

It should replace St. Peter with a link to the article on St. Peter, and pope with a link to the article on the Pope.

I have it working, but it is very slow. There are over 30,000 possible replacements, so it is important to optimize. I'm just not sure where to go from here.

Here's my existing code. Note that it's using Drupal. Also, it replaces the words with a [cathenlink] tag, and that tag is replaced by a real HTML link later in the code.

function ce_execute_filter($text)
{

    // If text is empty, return as-is
    if (!$text) {
        return $text;
    }

    // Split by paragraph
    $lines = preg_split('/\n+/', $text, -1, PREG_SPLIT_DELIM_CAPTURE);

    // Contains the parsed and linked text
    $linked_text = '';

    foreach ($lines as $line)
    {

        // If this fragment is only one or more newline characters,
        // Add it to $linked_text and continue without parsing
        if (preg_match('/^\n+$/', $line)) {
            $linked_text .= $line;
            continue;
        }

        // Select any terms that might be in this line
        // Ordered by descending length of term,
        // so that the longest terms get replaced first
        $result = db_query('SELECT title, term FROM {catholic_encyclopedia_terms} ' .
                "WHERE :text LIKE CONCAT('%', CONCAT(term, '%')) " .
                'GROUP BY term ' .
                'ORDER BY char_length(term) DESC',
                array(
                    ':text' => $line
                    ))
            ->fetchAll();

        // Array with lowercase term as key, title of entry as value
        $terms = array();

        // Array of the terms only in descending order of length
        $ordered_terms = array();

        foreach ($result as $r)
        {
            $terms[strtolower($r->term)] = $r->title;
            $ordered_terms[] = preg_quote($r->term);
        }

        // If no terms were returned, add the line and continue without parsing.
        if (empty($ordered_terms)) {
            $linked_text .= $line;
            continue;
        }

        // Do the replace
        // Get the regexp by joining $ordered_terms with |
        $line = preg_replace_callback('/\b('.
                    implode('|', $ordered_terms) .
                    ')\b/i', function ($matches) use($terms)
                {
                if ($matches[1]) {
                return "[cathenlink=" .
                $terms[strtolower($matches[1])] . "]" .
                $matches[1] . "[/cathenlink]";
                }
                },
                $line);

        $linked_text .= $line;
    }

    return $linked_text;
}

I'm doing the preg_replace like this so that it doesn't replace a word twice. I would use strtr, but then there's no way to ensure it is a full word and not just part of a word.

Is there any way to make this faster? Right now it is pretty slow.

Answer 1

I think the LIKE keyword is slowing you down. Is it indexed ?

You can find some clues here

Answer 2

You could use an indexing system such as Lucene to index the Catholic Encyclopedia. I don't suspect it changes very often, so indexing can be updated on a daily bassis. Lucene is written in Java, but I know that Zend has a PHP module that can read the index.

Answer 3

OK, I think the way I am doing it is probably the most efficient. What I came up with is to cache the results for one week, so that posts don't have to be parsed more than once per week. Implementing this solution, I've seen a marked improvement in speed on my site, so it seems to be working.