简体   繁体   中英

Most efficient algorithm to retrieve data from database

So I am removing the Joins from my queries as I prepare to move to Cassandra which does not support this, but rather supports many select statements instead. I made a benchmark test on 50 rows of data in my mysql table (what I am currently using) which resulted in 101 queries (all select) and it took ~0.035 seconds to complete all of these queries. I then changed this around to some array manipulation (currently in PHP) and reduced this down to 3 queries with a bunch of O(n) for loops.

I assume whether my system is on PHP, Python, MySQL, or Cassandra (NoSQL) that it is way faster to process the data using a few O(n) for loops rather than a lot more queries, I had cut down the time from 0.035s to 0.004s using this new method as I show below.

Any alternate methods to shortening this down more? Or am I on the right track? Any cases where it's faster to run all of the queries (besides when it becomes O(n^2))? Thanks:

// Now go through and get all of the user information (This is slower in mysql, but maybe faster in cassandra)
        /*foreach ($results as $key => $row)
        {
            // Create query
            $query = DB::select('id', 'username', 'profile_picture')->from('users')->where('id', '=', $row['uid']);

            // Execute it
            $results2 = $query->execute(null, false);

            // Join it
            $data[$key] = array_merge($row, $results2[0]);
        }*/

        // Get all the user information (faster in mysql since less queries)
        $uids = array();
        $ids = array();
        foreach ($results as $key => $row)
        {
            if (!in_array($row['uid'], $uids))
                $uids[] = $row['uid'];
            if (!in_array($type, array('userProfile')))
                $ids[] = $row['comment_id'];
        }

        // Create query
        $query = DB::select('id', 'username', 'profile_picture')->from('users')->where('id', '=', $uids);

        // Execute it
        $results2 = $query->execute(null, false);

        $user_data = array();

        foreach ($results2 as $key => $row)
        {
            $user_data[$row['id']] = array('uid' => $row['id'], 'username' => $row['username'], 'profile_picture' => $row['profile_picture']);
        }

        foreach ($results as $key => $row)
        {
            $data[$key] = array_merge($row, $user_data[$row['uid']]);
        }
        // End faster user info section

With Cassandra you can ask for all your keys in one query using a multi get, which is much faster than a bunch of single queries. I sometimes ask for thousands of keys in a query, and the response time is effectively instant.

There are more and more tools like playOrm(also has a raw ad-hoc tool coming) that support joins BUT only on partitions of tables(not entire tables) and do indexing with nosql patterns behind the scenes. Check out the wide-row pattern and see if that is useful to you. IT can help speed things up sometimes.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM