Compressor Improvements and LZSSE2 vs LZSSE8
One of the things that I hadn’t put a lot of time into with LZSSE, when I shoved it out into the open last week, was the compressor side of things. The compressors were mainly there as a way to show that the codec/decompression functioned and although it was pretty young and experimental, I thought it was worth sharing and getting some feedback on.
One of the things that became fairly apparent (and anyone who eyeballed the original benchmarks would’ve seen) is that the compression speed for LZSSE2’s optimal parser was terrible. Thanks to some pointers from the community, particularly from Charles Bloom, the match finding has been replaced with something better (an implementation of the same algorithm used in LzFind) and compression speed improved substantially. There is a very minor cost in compression ratio though on some files, although some was gained back due to fixing a bug in the cost calculations for matches. I’ve also created an LZSSE8 optimal parsing variant, LZSSE4’s is on the way.
One of the changes, to prevent really bad performance with long matches (especially of the same character), is to skip completely over match finding after long matches. This can have some minor impact on compression ratio, so there is the option to disable this by pushing the compression level up to “17” on the optimal parser, which is otherwise the same as level 16. Levels 0 through to 15 just check through progressively less matches as the level gets lower (amortization).
The compression changes to LZSSE8 proved rather important, because there are some files LZSSE2 really struggled with, particularly machine code binaries and less compressible data. With the compressors for LZSSE2 and LZSSE8 previously not being on equal footing, it wasn’t possible to show how much better LZSSE8 worked on those files. The quality of the compression really has quite a large impact on the performance of the decompression on those files too.
Part of the end goal is to make a codec that either adapts per iteration (32 steps), or allows changing between the different codecs per block to get the ideal compression ratio and performance on a wider range of files without changing codec. This is not there yet, but the results below make a strong case for it.
I was lucky enough to get some code contributed by Brian Marshall that got things working with Gcc and Clang (although, it was ported elsewhere as part of turbobench). Performance of the decompression on Gcc lags Visual Studio by over 10% (sometimes even higher) and investigating why that is will be an interesting experiment in and of itself. The code is the same for both, there is nothing compiler specific in the decompression.
I’m still intending to release a stand-alone single file utility (both as a binary and on GitHub), but there are still some changes I want to make there before it’s public.
Another note, there has been a few problems with the blog comments, if you make a comment and it doesn’t show up, I haven’t deleted it. I’m still investigating why this is.
Updated Results
Note, the decompression implementation hasn’t changed here. I’m using the same method for benchmarks as before (same stock clocked i7 4790, Windows 10 machine, same build settings, same lzbench code), but with the results reduced to LZSSE variants, because I mainly want to compare LZSSE2 against LZSSE8 here (the results for the other compressors haven’t changed, if you want the comparisons).
The tl;dr version is that LZSSE8 lags a little on text, but is a better general purpose option, when used with the new optimal parser, than LZSSE2. LZSSE8 isn’t that far behind on text, but where LZSSE2 does badly (less compressible data, machine code and long literal runs), LZSSE8 does a lot better (this makes a convincing case for an adaptive codec). Compression speed, especially on the previous terrible cases, is a lot better.
First up, enwik8; our new LZSSE2 compression implementation has given up a few bytes here and the new optimal parser for LZSSE8 is a little bit behind. Compression speed is significantly improved. No real surprises here.
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11511 MB/s | 11562 MB/s | 100000000 | 100.00 |
| lzsse2 0.1 level 1 | 18 MB/s | 3138 MB/s | 45854485 | 45.85 |
| lzsse2 0.1 level 2 | 13 MB/s | 3403 MB/s | 41941798 | 41.94 |
| lzsse2 0.1 level 4 | 9.37 MB/s | 3739 MB/s | 38121396 | 38.12 |
| lzsse2 0.1 level 8 | 9.30 MB/s | 3720 MB/s | 38068526 | 38.07 |
| lzsse2 0.1 level 12 | 9.28 MB/s | 3718 MB/s | 38068509 | 38.07 |
| lzsse2 0.1 level 16 | 9.29 MB/s | 3748 MB/s | 38068509 | 38.07 |
| lzsse2 0.1 level 17 | 9.05 MB/s | 3717 MB/s | 38063182 | 38.06 |
| lzsse4 0.1 | 213 MB/s | 3122 MB/s | 47108403 | 47.11 |
| lzsse8f 0.1 | 212 MB/s | 3108 MB/s | 47249359 | 47.25 |
| lzsse8o 0.1 level 1 | 15 MB/s | 3434 MB/s | 41889439 | 41.89 |
| lzsse8o 0.1 level 2 | 12 MB/s | 3559 MB/s | 40129972 | 40.13 |
| lzsse8o 0.1 level 4 | 10 MB/s | 3668 MB/s | 38745570 | 38.75 |
| lzsse8o 0.1 level 8 | 10 MB/s | 3671 MB/s | 38721328 | 38.72 |
| lzsse8o 0.1 level 12 | 10 MB/s | 3670 MB/s | 38721328 | 38.72 |
| lzsse8o 0.1 level 16 | 10 MB/s | 3698 MB/s | 38721328 | 38.72 |
| lzsse8o 0.1 level 17 | 10 MB/s | 3673 MB/s | 38716643 | 38.72 |
LZSSE8 shows that with an optimal parser it’s a better generalist on the tar’d silesia corpus than LZSSE2 (which has poor handling of hard to compress data), with both stronger compression and significantly faster decompression. Compression speed again is significantly improved:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11406 MB/s | 11347 MB/s | 211948032 | 100.00 |
| lzsse2 0.1 level 1 | 19 MB/s | 3182 MB/s | 87976190 | 41.51 |
| lzsse2 0.1 level 2 | 14 MB/s | 3392 MB/s | 82172004 | 38.77 |
| lzsse2 0.1 level 4 | 9.57 MB/s | 3636 MB/s | 76093504 | 35.90 |
| lzsse2 0.1 level 8 | 8.62 MB/s | 3637 MB/s | 75830436 | 35.78 |
| lzsse2 0.1 level 12 | 8.56 MB/s | 3662 MB/s | 75830178 | 35.78 |
| lzsse2 0.1 level 16 | 8.77 MB/s | 3663 MB/s | 75830178 | 35.78 |
| lzsse2 0.1 level 17 | 3.42 MB/s | 3669 MB/s | 75685829 | 35.71 |
| lzsse4 0.1 | 275 MB/s | 3243 MB/s | 95918518 | 45.26 |
| lzsse8f 0.1 | 276 MB/s | 3405 MB/s | 94938891 | 44.79 |
| lzsse8o 0.1 level 1 | 17 MB/s | 3932 MB/s | 81866286 | 38.63 |
| lzsse8o 0.1 level 2 | 14 MB/s | 4085 MB/s | 78727052 | 37.14 |
| lzsse8o 0.1 level 4 | 10 MB/s | 4262 MB/s | 78723935 | 37.14 |
| lzsse8o 0.1 level 8 | 9.77 MB/s | 4284 MB/s | 75464683 | 35.61 |
| lzsse8o 0.1 level 12 | 9.70 MB/s | 4283 MB/s | 75464456 | 35.61 |
| lzsse8o 0.1 level 16 | 9.68 MB/s | 4283 MB/s | 75464456 | 35.61 |
| lzsse8o 0.1 level 17 | 3.50 MB/s | 4289 MB/s | 75328279 | 35.54 |
Dickens favours LZSSE2, but LZSSE8 is not that far behind:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 13773 MB/s | 13754 MB/s | 10192446 | 100.00 |
| lzsse2 0.1 level 1 | 19 MB/s | 2938 MB/s | 5259008 | 51.60 |
| lzsse2 0.1 level 2 | 13 MB/s | 3326 MB/s | 4626067 | 45.39 |
| lzsse2 0.1 level 4 | 7.96 MB/s | 3935 MB/s | 3878822 | 38.06 |
| lzsse2 0.1 level 8 | 7.92 MB/s | 3932 MB/s | 3872650 | 38.00 |
| lzsse2 0.1 level 12 | 7.90 MB/s | 3876 MB/s | 3872650 | 38.00 |
| lzsse2 0.1 level 16 | 7.90 MB/s | 3944 MB/s | 3872650 | 38.00 |
| lzsse2 0.1 level 17 | 7.78 MB/s | 3942 MB/s | 3872373 | 37.99 |
| lzsse4 0.1 | 209 MB/s | 2948 MB/s | 5161515 | 50.64 |
| lzsse8f 0.1 | 209 MB/s | 2953 MB/s | 5174322 | 50.77 |
| lzsse8o 0.1 level 1 | 15 MB/s | 3256 MB/s | 4515350 | 44.30 |
| lzsse8o 0.1 level 2 | 12 MB/s | 3448 MB/s | 4207915 | 41.28 |
| lzsse8o 0.1 level 4 | 9.12 MB/s | 3734 MB/s | 3922639 | 38.49 |
| lzsse8o 0.1 level 8 | 9.15 MB/s | 3752 MB/s | 3920885 | 38.47 |
| lzsse8o 0.1 level 12 | 9.15 MB/s | 3741 MB/s | 3920885 | 38.47 |
| lzsse8o 0.1 level 16 | 9.26 MB/s | 3640 MB/s | 3920885 | 38.47 |
| lzsse8o 0.1 level 17 | 8.98 MB/s | 3751 MB/s | 3920616 | 38.47 |
Mozilla, LZSSE8 provides a significant decompression performance advantage and a slightly better compression ratio:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11448 MB/s | 11433 MB/s | 51220480 | 100.00 |
| lzsse2 0.1 level 1 | 21 MB/s | 2744 MB/s | 24591826 | 48.01 |
| lzsse2 0.1 level 2 | 16 MB/s | 2868 MB/s | 23637008 | 46.15 |
| lzsse2 0.1 level 4 | 11 MB/s | 3037 MB/s | 22584848 | 44.09 |
| lzsse2 0.1 level 8 | 9.97 MB/s | 3063 MB/s | 22474739 | 43.88 |
| lzsse2 0.1 level 12 | 9.93 MB/s | 3064 MB/s | 22474508 | 43.88 |
| lzsse2 0.1 level 16 | 9.82 MB/s | 3063 MB/s | 22474508 | 43.88 |
| lzsse2 0.1 level 17 | 2.01 MB/s | 3038 MB/s | 22460261 | 43.85 |
| lzsse4 0.1 | 257 MB/s | 2635 MB/s | 27406939 | 53.51 |
| lzsse8f 0.1 | 261 MB/s | 2817 MB/s | 26993974 | 52.70 |
| lzsse8o 0.1 level 1 | 18 MB/s | 3369 MB/s | 23512213 | 45.90 |
| lzsse8o 0.1 level 2 | 15 MB/s | 3477 MB/s | 22941395 | 44.79 |
| lzsse8o 0.1 level 4 | 12 MB/s | 3646 MB/s | 22243004 | 43.43 |
| lzsse8o 0.1 level 8 | 10 MB/s | 3647 MB/s | 22148536 | 43.24 |
| lzsse8o 0.1 level 12 | 10 MB/s | 3688 MB/s | 22148366 | 43.24 |
| lzsse8o 0.1 level 16 | 10 MB/s | 3688 MB/s | 22148366 | 43.24 |
| lzsse8o 0.1 level 17 | 1.96 MB/s | 3689 MB/s | 22135467 | 43.22 |
LZSSE8 again shows that it can significantly best LZSSE2 on some files on mr, the magnetic resonance image:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 13848 MB/s | 13925 MB/s | 9970564 | 100.00 |
| lzsse2 0.1 level 1 | 30 MB/s | 2801 MB/s | 4847277 | 48.62 |
| lzsse2 0.1 level 2 | 21 MB/s | 3004 MB/s | 4506789 | 45.20 |
| lzsse2 0.1 level 4 | 12 MB/s | 3250 MB/s | 4018986 | 40.31 |
| lzsse2 0.1 level 8 | 11 MB/s | 3263 MB/s | 4013037 | 40.25 |
| lzsse2 0.1 level 12 | 11 MB/s | 3263 MB/s | 4013037 | 40.25 |
| lzsse2 0.1 level 16 | 11 MB/s | 3263 MB/s | 4013037 | 40.25 |
| lzsse2 0.1 level 17 | 1.89 MB/s | 3333 MB/s | 4007016 | 40.19 |
| lzsse4 0.1 | 249 MB/s | 2560 MB/s | 7034206 | 70.55 |
| lzsse8f 0.1 | 279 MB/s | 2990 MB/s | 6856430 | 68.77 |
| lzsse8o 0.1 level 1 | 27 MB/s | 3421 MB/s | 4697617 | 47.11 |
| lzsse8o 0.1 level 2 | 20 MB/s | 3677 MB/s | 4384872 | 43.98 |
| lzsse8o 0.1 level 4 | 14 MB/s | 4058 MB/s | 4037906 | 40.50 |
| lzsse8o 0.1 level 8 | 14 MB/s | 4010 MB/s | 4033597 | 40.46 |
| lzsse8o 0.1 level 12 | 14 MB/s | 4138 MB/s | 4033597 | 40.46 |
| lzsse8o 0.1 level 16 | 14 MB/s | 4009 MB/s | 4033597 | 40.46 |
| lzsse8o 0.1 level 17 | 1.87 MB/s | 4026 MB/s | 4029062 | 40.41 |
The nci database of chemical structures shows that LZSSE2 still definitely has some things it’s better at:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11550 MB/s | 11642 MB/s | 33553445 | 100.00 |
| lzsse2 0.1 level 1 | 20 MB/s | 5616 MB/s | 5295570 | 15.78 |
| lzsse2 0.1 level 2 | 14 MB/s | 6368 MB/s | 4423662 | 13.18 |
| lzsse2 0.1 level 4 | 7.86 MB/s | 6981 MB/s | 3810991 | 11.36 |
| lzsse2 0.1 level 8 | 6.17 MB/s | 7110 MB/s | 3749513 | 11.17 |
| lzsse2 0.1 level 12 | 6.16 MB/s | 7178 MB/s | 3749513 | 11.17 |
| lzsse2 0.1 level 16 | 6.17 MB/s | 7107 MB/s | 3749513 | 11.17 |
| lzsse2 0.1 level 17 | 3.00 MB/s | 7171 MB/s | 3703443 | 11.04 |
| lzsse4 0.1 | 497 MB/s | 5524 MB/s | 5711408 | 17.02 |
| lzsse8f 0.1 | 501 MB/s | 5325 MB/s | 5796797 | 17.28 |
| lzsse8o 0.1 level 1 | 18 MB/s | 5975 MB/s | 4558097 | 13.58 |
| lzsse8o 0.1 level 2 | 14 MB/s | 6225 MB/s | 4202797 | 12.53 |
| lzsse8o 0.1 level 4 | 7.94 MB/s | 6538 MB/s | 3903767 | 11.63 |
| lzsse8o 0.1 level 8 | 6.48 MB/s | 6601 MB/s | 3872165 | 11.54 |
| lzsse8o 0.1 level 12 | 6.48 MB/s | 6601 MB/s | 3872165 | 11.54 |
| lzsse8o 0.1 level 16 | 6.54 MB/s | 6601 MB/s | 3872165 | 11.54 |
| lzsse8o 0.1 level 17 | 3.09 MB/s | 6638 MB/s | 3828113 | 11.41 |
LZSSE8 again shows it’s strength on binaries with the ooffice dll:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 17379 MB/s | 17379 MB/s | 6152192 | 100.00 |
| lzsse2 0.1 level 1 | 21 MB/s | 2205 MB/s | 3821230 | 62.11 |
| lzsse2 0.1 level 2 | 17 MB/s | 2297 MB/s | 3681963 | 59.85 |
| lzsse2 0.1 level 4 | 12 MB/s | 2444 MB/s | 3505982 | 56.99 |
| lzsse2 0.1 level 8 | 12 MB/s | 2424 MB/s | 3496155 | 56.83 |
| lzsse2 0.1 level 12 | 12 MB/s | 2457 MB/s | 3496154 | 56.83 |
| lzsse2 0.1 level 16 | 12 MB/s | 2455 MB/s | 3496154 | 56.83 |
| lzsse2 0.1 level 17 | 7.03 MB/s | 2457 MB/s | 3495182 | 56.81 |
| lzsse4 0.1 | 166 MB/s | 2438 MB/s | 3972687 | 64.57 |
| lzsse8f 0.1 | 162 MB/s | 2712 MB/s | 3922013 | 63.75 |
| lzsse8o 0.1 level 1 | 19 MB/s | 2936 MB/s | 3635572 | 59.09 |
| lzsse8o 0.1 level 2 | 17 MB/s | 3041 MB/s | 3564756 | 57.94 |
| lzsse8o 0.1 level 4 | 14 MB/s | 3063 MB/s | 3499888 | 56.89 |
| lzsse8o 0.1 level 8 | 14 MB/s | 3097 MB/s | 3494693 | 56.80 |
| lzsse8o 0.1 level 12 | 14 MB/s | 3099 MB/s | 3494693 | 56.80 |
| lzsse8o 0.1 level 16 | 14 MB/s | 3062 MB/s | 3494693 | 56.80 |
| lzsse8o 0.1 level 17 | 7.25 MB/s | 3100 MB/s | 3493814 | 56.79 |
The MySQL database, osdb, strongly favours LZSSE8 as well:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 14086 MB/s | 14185 MB/s | 10085684 | 100.00 |
| lzsse2 0.1 level 1 | 18 MB/s | 3200 MB/s | 4333712 | 42.97 |
| lzsse2 0.1 level 2 | 14 MB/s | 3308 MB/s | 4172322 | 41.37 |
| lzsse2 0.1 level 4 | 11 MB/s | 3546 MB/s | 3960769 | 39.27 |
| lzsse2 0.1 level 8 | 11 MB/s | 3555 MB/s | 3957654 | 39.24 |
| lzsse2 0.1 level 12 | 11 MB/s | 3556 MB/s | 3957654 | 39.24 |
| lzsse2 0.1 level 16 | 11 MB/s | 3557 MB/s | 3957654 | 39.24 |
| lzsse2 0.1 level 17 | 11 MB/s | 3558 MB/s | 3957654 | 39.24 |
| lzsse4 0.1 | 265 MB/s | 4098 MB/s | 4262592 | 42.26 |
| lzsse8f 0.1 | 259 MB/s | 4484 MB/s | 4175474 | 41.40 |
| lzsse8o 0.1 level 1 | 16 MB/s | 4628 MB/s | 3962308 | 39.29 |
| lzsse8o 0.1 level 2 | 13 MB/s | 4673 MB/s | 3882195 | 38.49 |
| lzsse8o 0.1 level 4 | 11 MB/s | 4723 MB/s | 3839571 | 38.07 |
| lzsse8o 0.1 level 8 | 11 MB/s | 4723 MB/s | 3839569 | 38.07 |
| lzsse8o 0.1 level 12 | 11 MB/s | 4717 MB/s | 3839569 | 38.07 |
| lzsse8o 0.1 level 16 | 11 MB/s | 4719 MB/s | 3839569 | 38.07 |
| lzsse8o 0.1 level 17 | 11 MB/s | 4721 MB/s | 3839569 | 38.07 |
The Polish literature PDF Reymont definitely favours LZSSE2 though, but LZSSE8 isn’t that far out:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 16693 MB/s | 16906 MB/s | 6627202 | 100.00 |
| lzsse2 0.1 level 1 | 19 MB/s | 3416 MB/s | 2657481 | 40.10 |
| lzsse2 0.1 level 2 | 13 MB/s | 4004 MB/s | 2309022 | 34.84 |
| lzsse2 0.1 level 4 | 7.55 MB/s | 4883 MB/s | 1867895 | 28.19 |
| lzsse2 0.1 level 8 | 7.32 MB/s | 4967 MB/s | 1852687 | 27.96 |
| lzsse2 0.1 level 12 | 7.30 MB/s | 4994 MB/s | 1852684 | 27.96 |
| lzsse2 0.1 level 16 | 7.30 MB/s | 4990 MB/s | 1852684 | 27.96 |
| lzsse2 0.1 level 17 | 7.26 MB/s | 4960 MB/s | 1852682 | 27.96 |
| lzsse8f 0.1 | 252 MB/s | 2989 MB/s | 2952727 | 44.55 |
| lzsse4 0.1 | 251 MB/s | 3020 MB/s | 2925190 | 44.14 |
| lzsse8o 0.1 level 1 | 16 MB/s | 3659 MB/s | 2402352 | 36.25 |
| lzsse8o 0.1 level 2 | 12 MB/s | 4001 MB/s | 2179138 | 32.88 |
| lzsse8o 0.1 level 4 | 8.10 MB/s | 4409 MB/s | 1905807 | 28.76 |
| lzsse8o 0.1 level 8 | 8.00 MB/s | 4394 MB/s | 1901961 | 28.70 |
| lzsse8o 0.1 level 12 | 8.01 MB/s | 4391 MB/s | 1901958 | 28.70 |
| lzsse8o 0.1 level 16 | 8.01 MB/s | 4415 MB/s | 1901958 | 28.70 |
| lzsse8o 0.1 level 17 | 7.97 MB/s | 4415 MB/s | 1901955 | 28.70 |
The samba source code offers a bit of a surprise, also doing better on LZSSE8. I haven’t studied the compression characteristics of source code too much, but expected it to favour LZSSE2 due to it’s favouring matches over literals.
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11781 MB/s | 11781 MB/s | 21606400 | 100.00 |
| lzsse2 0.1 level 1 | 20 MB/s | 4017 MB/s | 6883306 | 31.86 |
| lzsse2 0.1 level 2 | 16 MB/s | 4201 MB/s | 6511387 | 30.14 |
| lzsse2 0.1 level 4 | 12 MB/s | 4330 MB/s | 6177369 | 28.59 |
| lzsse2 0.1 level 8 | 9.45 MB/s | 4336 MB/s | 6161485 | 28.52 |
| lzsse2 0.1 level 12 | 8.56 MB/s | 4333 MB/s | 6161465 | 28.52 |
| lzsse2 0.1 level 16 | 8.54 MB/s | 4292 MB/s | 6161465 | 28.52 |
| lzsse2 0.1 level 17 | 2.57 MB/s | 4379 MB/s | 6093318 | 28.20 |
| lzsse4 0.1 | 300 MB/s | 3997 MB/s | 7601765 | 35.18 |
| lzsse8f 0.1 | 298 MB/s | 4136 MB/s | 7582300 | 35.09 |
| lzsse8o 0.1 level 1 | 18 MB/s | 4656 MB/s | 6433602 | 29.78 |
| lzsse8o 0.1 level 2 | 15 MB/s | 4761 MB/s | 6220494 | 28.79 |
| lzsse8o 0.1 level 4 | 12 MB/s | 4860 MB/s | 6043129 | 27.97 |
| lzsse8o 0.1 level 8 | 10 MB/s | 4880 MB/s | 6030715 | 27.91 |
| lzsse8o 0.1 level 12 | 9.57 MB/s | 4931 MB/s | 6030693 | 27.91 |
| lzsse8o 0.1 level 16 | 9.58 MB/s | 4930 MB/s | 6030693 | 27.91 |
| lzsse8o 0.1 level 17 | 2.58 MB/s | 4956 MB/s | 5965309 | 27.61 |
The SAO star catalog is less compressible and LZSSE2 does very poorly compared to LZSSE8 here:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 15938 MB/s | 16044 MB/s | 7251944 | 100.00 |
| lzsse2 0.1 level 1 | 24 MB/s | 1785 MB/s | 6487976 | 89.47 |
| lzsse2 0.1 level 2 | 20 MB/s | 1817 MB/s | 6304480 | 86.94 |
| lzsse2 0.1 level 4 | 15 MB/s | 1888 MB/s | 6068279 | 83.68 |
| lzsse2 0.1 level 8 | 15 MB/s | 1885 MB/s | 6066923 | 83.66 |
| lzsse2 0.1 level 12 | 15 MB/s | 1903 MB/s | 6066923 | 83.66 |
| lzsse2 0.1 level 16 | 15 MB/s | 1870 MB/s | 6066923 | 83.66 |
| lzsse2 0.1 level 17 | 15 MB/s | 1887 MB/s | 6066923 | 83.66 |
| lzsse4 0.1 | 167 MB/s | 2433 MB/s | 6305407 | 86.95 |
| lzsse8f 0.1 | 167 MB/s | 3268 MB/s | 6045723 | 83.37 |
| lzsse8o 0.1 level 1 | 23 MB/s | 3486 MB/s | 5826841 | 80.35 |
| lzsse8o 0.1 level 2 | 20 MB/s | 3684 MB/s | 5713888 | 78.79 |
| lzsse8o 0.1 level 4 | 17 MB/s | 3875 MB/s | 5575656 | 76.88 |
| lzsse8o 0.1 level 8 | 16 MB/s | 3857 MB/s | 5575361 | 76.88 |
| lzsse8o 0.1 level 12 | 17 MB/s | 3832 MB/s | 5575361 | 76.88 |
| lzsse8o 0.1 level 16 | 17 MB/s | 3804 MB/s | 5575361 | 76.88 |
| lzsse8o 0.1 level 17 | 17 MB/s | 3861 MB/s | 5575361 | 76.88 |
The webster dictionary has LZSSE2 edging out LZSSE8, but not by much:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11596 MB/s | 11545 MB/s | 41458703 | 100.00 |
| lzsse2 0.1 level 1 | 17 MB/s | 3703 MB/s | 15546126 | 37.50 |
| lzsse2 0.1 level 2 | 12 MB/s | 4087 MB/s | 13932531 | 33.61 |
| lzsse2 0.1 level 4 | 8.30 MB/s | 4479 MB/s | 12405168 | 29.92 |
| lzsse2 0.1 level 8 | 8.12 MB/s | 4477 MB/s | 12372822 | 29.84 |
| lzsse2 0.1 level 12 | 8.12 MB/s | 4483 MB/s | 12372822 | 29.84 |
| lzsse2 0.1 level 16 | 8.12 MB/s | 4441 MB/s | 12372822 | 29.84 |
| lzsse2 0.1 level 17 | 8.12 MB/s | 4444 MB/s | 12372794 | 29.84 |
| lzsse4 0.1 | 242 MB/s | 3542 MB/s | 16613479 | 40.07 |
| lzsse8f 0.1 | 238 MB/s | 3376 MB/s | 16775799 | 40.46 |
| lzsse8o 0.1 level 1 | 14 MB/s | 3910 MB/s | 14218534 | 34.30 |
| lzsse8o 0.1 level 2 | 11 MB/s | 4069 MB/s | 13404951 | 32.33 |
| lzsse8o 0.1 level 4 | 9.12 MB/s | 4191 MB/s | 12700224 | 30.63 |
| lzsse8o 0.1 level 8 | 8.98 MB/s | 4191 MB/s | 12685372 | 30.60 |
| lzsse8o 0.1 level 12 | 8.98 MB/s | 4192 MB/s | 12685372 | 30.60 |
| lzsse8o 0.1 level 16 | 8.98 MB/s | 4229 MB/s | 12685372 | 30.60 |
| lzsse8o 0.1 level 17 | 9.01 MB/s | 4196 MB/s | 12685361 | 30.60 |
LZSSE2 edges out LZSSE8 on xml, but it is pretty close:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 18624 MB/s | 18624 MB/s | 5345280 | 100.00 |
| lzsse2 0.1 level 1 | 21 MB/s | 5873 MB/s | 1003195 | 18.77 |
| lzsse2 0.1 level 2 | 15 MB/s | 6303 MB/s | 885118 | 16.56 |
| lzsse2 0.1 level 4 | 9.64 MB/s | 6706 MB/s | 778294 | 14.56 |
| lzsse2 0.1 level 8 | 9.49 MB/s | 6706 MB/s | 776774 | 14.53 |
| lzsse2 0.1 level 12 | 9.40 MB/s | 6706 MB/s | 776774 | 14.53 |
| lzsse2 0.1 level 16 | 9.42 MB/s | 6706 MB/s | 776774 | 14.53 |
| lzsse2 0.1 level 17 | 3.23 MB/s | 6766 MB/s | 768186 | 14.37 |
| lzsse4 0.1 | 380 MB/s | 4413 MB/s | 1388503 | 25.98 |
| lzsse8f 0.1 | 377 MB/s | 4189 MB/s | 1406119 | 26.31 |
| lzsse8o 0.1 level 1 | 19 MB/s | 5932 MB/s | 919525 | 17.20 |
| lzsse8o 0.1 level 2 | 15 MB/s | 6215 MB/s | 849715 | 15.90 |
| lzsse8o 0.1 level 4 | 10 MB/s | 6424 MB/s | 793567 | 14.85 |
| lzsse8o 0.1 level 8 | 10 MB/s | 6455 MB/s | 792369 | 14.82 |
| lzsse8o 0.1 level 12 | 10 MB/s | 6455 MB/s | 792369 | 14.82 |
| lzsse8o 0.1 level 16 | 10 MB/s | 6447 MB/s | 792369 | 14.82 |
| lzsse8o 0.1 level 17 | 3.27 MB/s | 6494 MB/s | 784226 | 14.67 |
The x-ray file is interesting, it favours LZSSE2 on compression ratio and LZSSE8 is significantly faster on decompression speed (which you would expect for this less compressible data):
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 15186 MB/s | 15268 MB/s | 8474240 | 100.00 |
| lzsse2 0.1 level 1 | 25 MB/s | 1617 MB/s | 7249233 | 85.54 |
| lzsse2 0.1 level 2 | 19 MB/s | 1623 MB/s | 7181547 | 84.75 |
| lzsse2 0.1 level 4 | 15 MB/s | 1637 MB/s | 7036083 | 83.03 |
| lzsse2 0.1 level 8 | 15 MB/s | 1637 MB/s | 7036044 | 83.03 |
| lzsse2 0.1 level 12 | 15 MB/s | 1625 MB/s | 7036044 | 83.03 |
| lzsse2 0.1 level 16 | 15 MB/s | 1624 MB/s | 7036044 | 83.03 |
| lzsse2 0.1 level 17 | 15 MB/s | 1624 MB/s | 7036044 | 83.03 |
| lzsse4 0.1 | 175 MB/s | 2361 MB/s | 7525821 | 88.81 |
| lzsse8f 0.1 | 172 MB/s | 3078 MB/s | 7248659 | 85.54 |
| lzsse8o 0.1 level 1 | 27 MB/s | 3020 MB/s | 7183700 | 84.77 |
| lzsse8o 0.1 level 2 | 26 MB/s | 3053 MB/s | 7171372 | 84.63 |
| lzsse8o 0.1 level 4 | 25 MB/s | 3058 MB/s | 7169088 | 84.60 |
| lzsse8o 0.1 level 8 | 26 MB/s | 3052 MB/s | 7169084 | 84.60 |
| lzsse8o 0.1 level 12 | 25 MB/s | 3061 MB/s | 7169084 | 84.60 |
| lzsse8o 0.1 level 16 | 25 MB/s | 3060 MB/s | 7169084 | 84.60 |
| lzsse8o 0.1 level 17 | 25 MB/s | 3051 MB/s | 7169084 | 84.60 |
The enwik9 text heavy benchmark again favours LZSSE2, but not by much:
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 3644 MB/s | 3699 MB/s | 1000000000 | 100.00 |
| lzsse2 0.1 level 1 | 18 MB/s | 3424 MB/s | 405899410 | 40.59 |
| lzsse2 0.1 level 2 | 13 MB/s | 3685 MB/s | 372353993 | 37.24 |
| lzsse2 0.1 level 4 | 9.49 MB/s | 3935 MB/s | 340990265 | 34.10 |
| lzsse2 0.1 level 8 | 9.24 MB/s | 3929 MB/s | 340559550 | 34.06 |
| lzsse2 0.1 level 12 | 9.34 MB/s | 3980 MB/s | 340558897 | 34.06 |
| lzsse2 0.1 level 16 | 9.34 MB/s | 3908 MB/s | 340558894 | 34.06 |
| lzsse2 0.1 level 17 | 7.89 MB/s | 3954 MB/s | 340293998 | 34.03 |
| lzsse4 0.1 | 221 MB/s | 3344 MB/s | 425848263 | 42.58 |
| lzsse8f 0.1 | 216 MB/s | 3281 MB/s | 427471454 | 42.75 |
| lzsse8o 0.1 level 1 | 15 MB/s | 3689 MB/s | 374183459 | 37.42 |
| lzsse8o 0.1 level 2 | 12 MB/s | 3643 MB/s | 358878280 | 35.89 |
| lzsse8o 0.1 level 4 | 10 MB/s | 3926 MB/s | 347111696 | 34.71 |
| lzsse8o 0.1 level 8 | 10 MB/s | 3905 MB/s | 346907396 | 34.69 |
| lzsse8o 0.1 level 12 | 10 MB/s | 3923 MB/s | 346907021 | 34.69 |
| lzsse8o 0.1 level 16 | 10 MB/s | 3937 MB/s | 346907020 | 34.69 |
| lzsse8o 0.1 level 17 | 9.25 MB/s | 3926 MB/s | 346660075 | 34.67 |
Finally, this is a new benchmark, the app3 file from compression ratings. It was brought to my attention on encode.ru. LZSSE2 does very poorly on this file, while LZSSE8 does significantly better in terms of compression ratio and decompression speed (note, I’ve included more results here because this wasn’t in the previous post):
| Compressor name | Compression | Decompress. | Compr. size | Ratio |
|---|---|---|---|---|
| memcpy | 11442 MB/s | 11390 MB/s | 100098560 | 100.00 |
| blosclz 2015-11-10 level 1 | 1369 MB/s | 10755 MB/s | 99783925 | 99.69 |
| blosclz 2015-11-10 level 3 | 733 MB/s | 10115 MB/s | 98845453 | 98.75 |
| blosclz 2015-11-10 level 6 | 311 MB/s | 2129 MB/s | 72486401 | 72.42 |
| blosclz 2015-11-10 level 9 | 267 MB/s | 1166 MB/s | 62088582 | 62.03 |
| brieflz 1.1.0 | 124 MB/s | 260 MB/s | 56188157 | 56.13 |
| crush 1.0 level 0 | 27 MB/s | 244 MB/s | 52205980 | 52.15 |
| crush 1.0 level 1 | 9.85 MB/s | 256 MB/s | 50786667 | 50.74 |
| crush 1.0 level 2 | 2.35 MB/s | 260 MB/s | 49887073 | 49.84 |
| fastlz 0.1 level 1 | 239 MB/s | 766 MB/s | 63274586 | 63.21 |
| fastlz 0.1 level 2 | 291 MB/s | 752 MB/s | 61682776 | 61.62 |
| lz4 r131 | 731 MB/s | 3361 MB/s | 61938601 | 61.88 |
| lz4fast r131 level 3 | 891 MB/s | 3567 MB/s | 65030520 | 64.97 |
| lz4fast r131 level 17 | 1494 MB/s | 4653 MB/s | 76451494 | 76.38 |
| lz4hc r131 level 1 | 115 MB/s | 3143 MB/s | 57430191 | 57.37 |
| lz4hc r131 level 4 | 69 MB/s | 3330 MB/s | 54955555 | 54.90 |
| lz4hc r131 level 9 | 44 MB/s | 3367 MB/s | 54423515 | 54.37 |
| lz4hc r131 level 12 | 30 MB/s | 3381 MB/s | 54396882 | 54.34 |
| lz4hc r131 level 16 | 17 MB/s | 3346 MB/s | 54387597 | 54.33 |
| lzf 3.6 level 0 | 294 MB/s | 833 MB/s | 63635905 | 63.57 |
| lzf 3.6 level 1 | 291 MB/s | 840 MB/s | 62477667 | 62.42 |
| lzg 1.0.8 level 1 | 34 MB/s | 619 MB/s | 63535191 | 63.47 |
| lzg 1.0.8 level 4 | 30 MB/s | 633 MB/s | 59258839 | 59.20 |
| lzg 1.0.8 level 6 | 23 MB/s | 641 MB/s | 57202956 | 57.15 |
| lzg 1.0.8 level 8 | 13 MB/s | 659 MB/s | 55031252 | 54.98 |
| lzham 1.0 -d26 level 0 | 9.64 MB/s | 203 MB/s | 47035821 | 46.99 |
| lzham 1.0 -d26 level 1 | 2.42 MB/s | 314 MB/s | 34502612 | 34.47 |
| lzjb 2010 | 313 MB/s | 574 MB/s | 71511986 | 71.44 |
| lzma 9.38 level 0 | 18 MB/s | 47 MB/s | 48231641 | 48.18 |
| lzma 9.38 level 2 | 15 MB/s | 51 MB/s | 45721294 | 45.68 |
| lzma 9.38 level 4 | 8.74 MB/s | 54 MB/s | 44384338 | 44.34 |
| lzma 9.38 level 5 | 3.79 MB/s | 56 MB/s | 41396555 | 41.36 |
| lzrw 15-Jul-1991 level 1 | 231 MB/s | 580 MB/s | 68040073 | 67.97 |
| lzrw 15-Jul-1991 level 2 | 191 MB/s | 700 MB/s | 67669680 | 67.60 |
| lzrw 15-Jul-1991 level 3 | 329 MB/s | 665 MB/s | 65336925 | 65.27 |
| lzrw 15-Jul-1991 level 4 | 329 MB/s | 488 MB/s | 63955770 | 63.89 |
| lzrw 15-Jul-1991 level 5 | 125 MB/s | 487 MB/s | 61433299 | 61.37 |
| lzsse2 0.1 level 1 | 21 MB/s | 2245 MB/s | 62862916 | 62.80 |
| lzsse2 0.1 level 2 | 18 MB/s | 2320 MB/s | 61190732 | 61.13 |
| lzsse2 0.1 level 4 | 13 MB/s | 2384 MB/s | 59622902 | 59.56 |
| lzsse2 0.1 level 8 | 12 MB/s | 2393 MB/s | 59490753 | 59.43 |
| lzsse2 0.1 level 12 | 12 MB/s | 2394 MB/s | 59490213 | 59.43 |
| lzsse2 0.1 level 16 | 12 MB/s | 2393 MB/s | 59490213 | 59.43 |
| lzsse2 0.1 level 17 | 2.77 MB/s | 2396 MB/s | 59403900 | 59.35 |
| lzsse4 0.1 | 241 MB/s | 2751 MB/s | 64507471 | 64.44 |
| lzsse8f 0.1 | 244 MB/s | 3480 MB/s | 62616595 | 62.55 |
| lzsse8o 0.1 level 1 | 20 MB/s | 3846 MB/s | 57314135 | 57.26 |
| lzsse8o 0.1 level 2 | 17 MB/s | 3936 MB/s | 56417687 | 56.36 |
| lzsse8o 0.1 level 4 | 14 MB/s | 4015 MB/s | 55656294 | 55.60 |
| lzsse8o 0.1 level 8 | 13 MB/s | 4033 MB/s | 55563258 | 55.51 |
| lzsse8o 0.1 level 12 | 13 MB/s | 4035 MB/s | 55562882 | 55.51 |
| lzsse8o 0.1 level 16 | 13 MB/s | 4032 MB/s | 55562882 | 55.51 |
| lzsse8o 0.1 level 17 | 2.76 MB/s | 4039 MB/s | 55482271 | 55.43 |
| quicklz 1.5.0 level 1 | 437 MB/s | 487 MB/s | 62035785 | 61.97 |
| quicklz 1.5.0 level 2 | 190 MB/s | 394 MB/s | 59010932 | 58.95 |
| quicklz 1.5.0 level 3 | 50 MB/s | 918 MB/s | 56738552 | 56.68 |
| yalz77 2015-09-19 level 1 | 57 MB/s | 583 MB/s | 58727991 | 58.67 |
| yalz77 2015-09-19 level 4 | 35 MB/s | 577 MB/s | 56893671 | 56.84 |
| yalz77 2015-09-19 level 8 | 24 MB/s | 580 MB/s | 56199900 | 56.14 |
| yalz77 2015-09-19 level 12 | 17 MB/s | 575 MB/s | 55916030 | 55.86 |
| yappy 2014-03-22 level 1 | 120 MB/s | 2722 MB/s | 64009309 | 63.95 |
| yappy 2014-03-22 level 10 | 102 MB/s | 3031 MB/s | 62084911 | 62.02 |
| yappy 2014-03-22 level 100 | 84 MB/s | 3116 MB/s | 61608971 | 61.55 |
| zlib 1.2.8 level 1 | 55 MB/s | 309 MB/s | 52928473 | 52.88 |
| zlib 1.2.8 level 6 | 28 MB/s | 328 MB/s | 49962674 | 49.91 |
| zlib 1.2.8 level 9 | 12 MB/s | 329 MB/s | 49860696 | 49.81 |
| zstd v0.4.1 level 1 | 377 MB/s | 710 MB/s | 53079608 | 53.03 |
| zstd v0.4.1 level 2 | 298 MB/s | 648 MB/s | 51425209 | 51.37 |
| zstd v0.4.1 level 5 | 110 MB/s | 604 MB/s | 48827923 | 48.78 |
| zstd v0.4.1 level 9 | 36 MB/s | 644 MB/s | 46859280 | 46.81 |
| zstd v0.4.1 level 13 | 19 MB/s | 644 MB/s | 46356567 | 46.31 |
| zstd v0.4.1 level 17 | 7.83 MB/s | 643 MB/s | 45814107 | 45.77 |
| zstd v0.4.1 level 20 | 1.48 MB/s | 849 MB/s | 35601905 | 35.57 |
| shrinker 0.1 | 389 MB/s | 1342 MB/s | 58260187 | 58.20 |
| wflz 2015-09-16 | 194 MB/s | 1246 MB/s | 65416104 | 65.35 |
| lzmat 1.01 | 38 MB/s | 525 MB/s | 52250313 | 52.20 |
blog comments powered by Disqus