Zstd is Impressive: A Real-World Example

Gzip is +30 years old, and most of the time it's still the default choice to compress text-based log files. It's decently fast and efficient for that type of file. But Zstd is much better!

Comparing Zstd with Gzip

I have a 111 MB log file from one of my Cloudfront distributions. Let's compress it to the maximum level, with the following commands: 

pigz -p1 -9 < cfront.log > cfront.log.gz
1.12s user 0.02s system 99% cpu 1.145 total

zstd -T1 -9 < cfront.log > cfront.log.zst
0.57s user 0.08s system 111% cpu 0.580 total

Speed

Efficiency

Zstd is better without compromise. Faster and producing a smaller output

Comparing Zstd with Bzip2, Brotli, Xz

Other compression algorithms are known for their output efficiency. Let's see how Zstd competes against them. 

bzip2 -9 < cfront.log > cfront.log.bz2
7.17s user 0.03s system 99% cpu 7.225 total

brotli -Z < cfront.log > cfront.log.br
59.49s user 0.10s system 99% cpu 59.732 total

xz -9 < cfront.log > cfront.log.xz
13.60s user 0.24s system 99% cpu 13.879 total

Efficiency (from smallest output to biggest)

Speed (from fastest to slowest)

Brotli is very efficient... and very slow. Xz almost matches the same efficiency, but it's almost 5 times faster. Brotli only gains 3 points of efficiency compared to Zstd, but it's 100 times slower! That makes Brotli and Xz less practical to use in this scenario.

Zstd is faster and more efficient than Gzip in a medium text-based log file scenario. Zstd is well-balanced in comparison with high-efficiency algorithms such as Brotli and Xz, making it suitable as a general-purpose compression algorithm, just like Gzip.

Versions

In use: Zstd v1.5.5, Gzip v1.12, Brotli v1.1.0, Xz v5.4.4, Bzip2 v1.0.8, packages from the Fedora 39 repositories.