Doctrine 大规模一对多性能

Question

I've built a news feed with post having a one-to-many relationship with post_reaction . The concept is simple, a post can be liked, and each like is stored in the post_reaction table alongside with who liked it and the type of reaction (like, love, etc.)

Everything works just fine™, however as things scale, performance decreases, namely as the post_reaction table grows.

For testing purposes, I generated 200 posts and gave each post 1,000 reactions. This results in 200,000 total reactions stored in the post_reaction table.

My Twig template is provided a list of posts, limited to 20. As the template iterates through displaying each post, it then calls post.reactions|length to count the number of reactions. This executes the following db query:

SELECT
  t0.reaction AS reaction_1,
  t0.id AS id_2,
  t0.created AS created_3,
  t0.post_id AS post_id_4,
  t0.user AS user_5
FROM
  post_reaction t0
WHERE
  t0.post_id = ?

This query takes on average 4-7ms to run each time it runs for the 20 posts I'm rendering. This totals to ~100ms worth of DB queries just to count the posts.

That doesn't seem too bad, however we observe some overhead processing this much data in the application.

Taking a look at the profiler for the entire request, we see the following: Our overall processing time in this request was 585ms

components/news_post.html.twig is the component that calls post.reactions|length which triggers the db query. If we initiate the same request without querying reactions, we observe the following. Our overall processing time in this request was 179ms

406ms/69.4% faster . I believe this is mostly attributed by overhead in doctrine while it processes the 20,000 rows into objects, only for us to count them later.

In an attempt to mitigate this, I wanted to see if joining the reactions onto my post query would help.

SELECT
  p0_.replies_allowed AS replies_allowed_0,
  p0_.highlight_date AS highlight_date_1,
  p0_.title AS title_2,
  p0_.content AS content_3,
  p0_.id AS id_4,
  p0_.created AS created_5,
  p0_.updated AS updated_6,
  p0_.news_feed_id AS news_feed_id_7,
  p0_.created_by_id AS created_by_id_8,
  p0_.updated_by_id AS updated_by_id_9
FROM
  post p0_
  INNER JOIN post_reaction p1_ ON (p1_.post_id = p0_.id)
WHERE
  p0_.news_feed_id = ?
ORDER BY
  CASE WHEN p0_.highlight_date > ? THEN 0 ELSE 1 END ASC,
  p0_.created DESC
LIMIT
  20

However it causes issues with the LIMIT 20 clause in the query, as joining reactions only allows one post to return due to the number of reactions in this dataset.

I'm not sure if I should continue to develop a way to make joining possible, or explore an alternative, whatever that may be. Ideally I'd like to reduce the 406ms of extra execution time since it's almost 70% of the total page processing time, just to count likes..

---

Edit: As requested, the output for show create table post_reaction

CREATE TABLE `post_reaction` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `post_id` int(11) DEFAULT NULL,
  `user` int(11) DEFAULT NULL,
  `reaction` int(11) NOT NULL,
  `reaction_timestamp` datetime NOT NULL,
  PRIMARY KEY (`id`),
  KEY `IDX_1B3A8E564B89032C` (`post_id`),
  KEY `IDX_1B3A8E568D93D649` (`user`),
  CONSTRAINT `FK_1B3A8E564B89032C` FOREIGN KEY (`post_id`) REFERENCES `post` (`id`),
  CONSTRAINT `FK_1B3A8E568D93D649` FOREIGN KEY (`user`) REFERENCES `user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=200786 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Answer 1

• (2nd query) Don't JOIN to post_reaction since you are not using any columns from it.

• The complexity of the ORDER BY makes it impossible to do anything faster look at all 1000 reactions. Hence the LIMIT has very little effect on performance.

• Please provide SHOW CREATE TABLE post_reaction , there may be some improvements we can make there. But you certainly need some index starting with post_id . We may get some improvement by rearranging the PRIMARY KEY to start with that column.

• (I don't know anything about the Controller or twigs. They seem to be the costly part?)

More

"count the number of reactions [for every post]" -- That is a single SQL query that won't take long:

SELECT post_id,
       COUNT(*) AS reaction_count
    FROM post_reaction
    GROUP BY post_id;

No iterating through posts; no 20 at a time; everything done in one simple pass over an index in that one table.

I tried an equivalent query on a table of 500K cities in 92 countries. It took 0.13 seconds.

The lesson here is that SQL shines when it is asked to do lots of the same thing on lots of rows.