| commit | 3f72c47f2ba1c52adb002d527138b3672fc7510c | [log] [tgz] |
|---|---|---|
| author | Mostyn Bramley-Moore <mostyn@antipode.se> | Sun Aug 08 10:57:06 2021 |
| committer | Mostyn Bramley-Moore <mostyn@antipode.se> | Sun Aug 08 11:00:01 2021 |
| tree | 9311855a9e0d5a40eceff6dc15e9e44a1ef0604d | |
| parent | 94959083aaa77874afd953e7343c99274d468050 [diff] |
upgrade github.com/klauspost/compress to v1.13.3 This includes the following zstd improvements since v1.13.0: * Add configurable Decoder window size https://github.com/klauspost/compress/pull/394 * Add stream content size https://github.com/klauspost/compress/pull/401 * Simplify hashing functions https://github.com/klauspost/compress/pull/402 * use SpeedBestCompression for level >= 10 https://github.com/klauspost/compress/pull/410 * Fix WriteTo error forwarding https://github.com/klauspost/compress/pull/411 * Improve Best compression https://github.com/klauspost/compress/pull/404
github.com/klauspost/compress/zstd is a native Go implementation of Zstandard, with an API that leaks memory and goroutines if you don't call Close() on every Encoder and Decoder that you create, and Decoders cannot be reused after they are closed.
zstdpool-syncpool is a Go wrapper for github.com/klauspost/compress/zstd and sync.Pool, which automatically frees resources if you forget to call Close() and/or when items are dropped from the sync.Pool.
Background: https://github.com/klauspost/compress/issues/264
import ( "github.com/klauspost/compress/zstd" syncpool "github.com/mostynb/zstdpool-syncpool" ) // Create a sync.Pool which returns wrapped *zstd.Decoder's. decoderPool := syncpool.NewDecoderPool(zstd.WithDecoderConcurrency(1)) // Get a DecoderWrapper and use it. decoder := decoderPool.Get().(*syncpool.DecoderWrapper) decoder.Reset(compressedDataReader) _, err = io.Copy(uncompressedDataWriter, decoder) // Return the decoder to the pool. If we forget this, then the zstd.Decoder // won't leak resources. decoderPool.Put(decoder) // Create a sync.Pool which returns wrapped *zstd.Endoder's. encoderPool := syncpool.NewEncoderPool(zstd.WithEncoderConcurrency(1)) // Get an EncoderWrapper and use it. encoder := encoderPool.Get().(*syncpool.EncoderWrapper) encoder.Reset(compressedDataWriter) _, err = io.Copy(encoder, uncompressedDataReader) // Return the encoder to the pool. If we forget this, then the zstd.Encoder // won't leak resources. encoderPool.Put(encoder)
Bug reports, feature requests, PRs welcome.
Licensed under the Apache License, Version 2.0. See the LICENSE file.