I took the same table as I used for MySQL Groupby Performance Tests to see how much MySQL can sort 1.000.000rows, or rather return top 10 rows from sorted result set which is the mosttypical way sorting is used in practice.
I tested full table scan of the table completes in 0.22seconds giving us about 4.5 Million of rows/sec. Obviously we can’t get sortedresult set faster than that.
I placed temporary sort files on tmpfs (/dev/shm) to avoiddisk IO as a variable as my data set fits in memory anyway and decided toexperiment with sort_buffer_size variable.
The minimum value for sort_buffer_size is32K which gives us the following speed:
mysql> select * from gt order by i desc limit 10;
+--------+------------------------------------------+
| i | c |
+--------+------------------------------------------+
| 100000 | 635e8e8f8e3b9dc547bbd3deaadb1f297f691729 |
| 100000 | 0a7750a1393e77a2871ecfb39d5032d0b0f7c37c |
| 100000 | 0db0601036fb9d1d5e17631d4d1bed9149675bb3 |
| 100000 | eb6d2b5ed1897bdd0ff6e22ee1b44814ffb8f912 |
| 100000 | 1bff67cc134e316dad5370de38020bef818ec45c |
| 99999 |635da2e73d88dbe5f7297253680398e58d32ff65 |
| 99999 |a1feec5f8ee6c6a96723a2a0b57c418bb3ced929 |
| 99999 |72b934f76863791f740b96858d5acb6a60459644 |
| 99999 |855b47aaa25054e77dcc27de5def8de1e265f371 |
| 99999 |81980bcd9dbaa565f22a93ce1faf9e9d53407f0a |
+--------+------------------------------------------+
10 rows in set (0.56 sec)
Not bad ! Even though MySQL does not optimize “get top Nsorted rows” very well it takes just 2.5 times longer than full table scan toget the data. And this is with minimum sort_buffer allowed when a lot of sortmerge passes are required for sort completion:
mysql> show status like "sort%";
+-------------------+-------+
| Variable_name |Value |
+-------------------+-------+
| Sort_merge_passes | 321 |
| Sort_range |0 |
| Sort_rows |10 |
| Sort_scan |1 |
+-------------------+-------+
4 rows in set (0.00 sec)
As you can see from this show status output MySQL only countscompletely sorted rows in Sort_rows variable. In this case1.000.000 of rows were partially sorted but only 10 rows fetched from the datafile and sent and only they are counted. In practice this means Sort_rows maywell understate sort activity happening on the system.
Lets now increase sort_buffer_size and seehow performance is affected:
set sort_buffer_size=100000;
mysql> select * from gt order by i desc limit 10;
10 rows in set (0.44 sec)
mysql> show status like "sort%";
+-------------------+-------+
| Variable_name |Value |
+-------------------+-------+
| Sort_merge_passes | 104 |
| Sort_range |0 |
| Sort_rows |10 |
| Sort_scan |1 |
+-------------------+-------+
4 rows in set (0.00 sec)
OK raising sort_buffer_size to 100K gives quite expectedperformance benefit, now we’re just 2 times slower than table scan of the queryand considering table size was about 60MB we have 120MB/sec sort speed, while2.000.000 rows/sec is of course more relevant in this case.
Still a lot of sort merge passes lets go with even higherbuffer sizes.
set sort_buffer_size=1000000;
mysql> select * from gt order by i desc limit 10;
10 rows in set (0.70 sec)
mysql> show status like "sort%";
+-------------------+-------+
| Variable_name |Value |
+-------------------+-------+
| Sort_merge_passes | 10 |
| Sort_range |0 |
| Sort_rows |10 |
| Sort_scan |1 |
+-------------------+-------+
4 rows in set (0.00 sec)
set sort_buffer_size=10000000;
mysql> select * from gt order by i desc limit 10;
10 rows in set (1.34 sec)
mysql> show status like "sort%";
+-------------------+-------+
| Variable_name |Value |
+-------------------+-------+
| Sort_merge_passes | 1 |
| Sort_range |0 |
| Sort_rows |10 |
| Sort_scan |1 |
+-------------------+-------+
4 rows in set (0.00 sec)
Wait it is not right. We’re increasing sort_buffer_size andnumber of sort_merge_passes decreases appropriately but itdoes not help sort speed instead it drops 3 times from 0.44sec to do 1.34sec !
Lets try it even higher to finally get rid of sort merge passes– may be it is sort merge which is inefficient with large sort_buffer_size ?
mysql> set sort_buffer_size=100000000;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from gt order by i desc limit 10;
+--------+------------------------------------------+
| i | c |
+--------+------------------------------------------+
| 100000 | eb6d2b5ed1897bdd0ff6e22ee1b44814ffb8f912 |
| 100000 | 635e8e8f8e3b9dc547bbd3deaadb1f297f691729 |
| 100000 | 1bff67cc134e316dad5370de38020bef818ec45c |
| 100000 | 0db0601036fb9d1d5e17631d4d1bed9149675bb3 |
| 100000 | 0a7750a1393e77a2871ecfb39d5032d0b0f7c37c |
| 99999 |41f091f4074717bf80d2b1a788e6a4a122057d11 |
| 99999 |049d9591ef0f584deaaf0433c0f3eda8631bdb85 |
| 99999 |72b934f76863791f740b96858d5acb6a60459644 |
| 99999 |f0a42a16a41b4249da7c31f2d9556f05622a87b4 |
| 99999 |35de8ae483779e6024c51998eb5b5e69e02eb74c |
+--------+------------------------------------------+
10 rows in set (1.55 sec)
mysql> show status like "sort%";
+-------------------+-------+
| Variable_name |Value |
+-------------------+-------+
| Sort_merge_passes | 0 |
| Sort_range |0 |
| Sort_rows |10 |
| Sort_scan |1 |
+-------------------+-------+
4 rows in set (0.00 sec)
Nope. We finally got rid of sort_merge_passes but our sortperformance got even worse !
I decided to experiment a bit further to see what sort_buffer_size isoptimal for given platform and given query (I did not test if it is the samefor all platforms or data sets) – The optimal sort_buffer_size inthis case was 70K-250K which is quite smaller than even default value.
The CPU in question was Pentium 4 having 1024K of cache.
A while ago I already wrote what largebuffers are not always better but I never expected optimalbuffer to be so small at least in some conditions.
What do we learn from these results:
· Benchmarkyour application Unfortunatelygeneral tuning guidelines can be wrong for your particular case, or generallywrong because they tend to reprint the manual which is often written based ontheoretical expectations rather than supported by large amount of testing.
· sort_merge_passesare not that bad. Settingyour sort_buffer_size large enough so there is zerosort_merge_passes may not be optimal.
· Worldis full of surprises Iobviously did not expect to get such results, and this is not any exception.Even spending a lot of time optimizing and otherwise working with MySQL Icontinue to run in results which surprise me. Some are later expected otherscome from underlying bugs and later fixed.