Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog.
[3.2011.0] - 2021-01-15
This release is not backward compatible with Badger v2.2007.x.
Breaking:
- opt(compactions): Improve compaction performance (#1574)
- Change how Badger handles WAL (#1555)
- feat(index): Use flatbuffers instead of protobuf (#1546)
Fixed:
- Fix(GC): Set bits correctly for moved keys (#1619)
- Fix(tableBuilding): reduce scope of valuePointer (#1617)
- Fix(compaction): fix table size estimation on compaction (#1613)
- Fix(OOM): Reuse pb.KVs in Stream (#1609)
- Fix race condition in L0StallMs variable (#1605)
- Fix(stream): Stop produceKVs on error (#1604)
- Fix(skiplist): Remove z.Buffer from skiplist (#1600)
- Fix(readonly): fix the file opening mode (#1592)
- Fix: Disable CompactL0OnClose by default (#1586)
- Fix(compaction): Don't drop data when split overlaps with top tables (#1587)
- Fix(subcompaction): Close builder before throttle.Done (#1582)
- Fix(table): Add onDisk size (#1569)
- Fix(Stream): Only send done markers if told to do so
- Fix(value log GC): Fix a bug which caused value log files to not be GCed.
- Fix segmentation fault when cache sizes are small. (#1552)
- Fix(builder): Too many small tables when compression is enabled (#1549)
- Fix integer overflow error when building for 386 (#1541)
- Fix(writeBatch): Avoid deadlock in commit callback (#1529)
- Fix(db): Handle nil logger (#1534)
- Fix(maxVersion): Use choosekey instead of KeyToList (#1532)
- Fix(Backup/Restore): Keep all versions (#1462)
- Fix(build): Fix nocgo builds. (#1493)
- Fix(cleanup): Avoid truncating in value.Open on error (#1465)
- Fix(compaction): Don't use cache for table compaction (#1467)
- Fix(compaction): Use separate compactors for L0, L1 (#1466)
- Fix(options): Do not implicitly enable cache (#1458)
- Fix(cleanup): Do not close cache before compaction (#1464)
- Fix(replay): Update head for LSM entires also (#1456)
- fix(levels): Cleanup builder resources on building an empty table (#1414)
Performance
- perf(GC): Remove move keys (#1539)
- Keep the cheaper parts of the index within table struct. (#1608)
- Opt(stream): Use z.Buffer to stream data (#1606)
- opt(builder): Use z.Allocator for building tables (#1576)
- opt(memory): Use z.Calloc for allocating KVList (#1563)
- opt: Small memory usage optimizations (#1562)
- KeySplits checks tables and memtables when number of splits is small. (#1544)
- perf: Reduce memory usage by better struct packing (#1528)
- perf(tableIterator): Don't do next on NewIterator (#1512)
- Improvements: Manual Memory allocation via Calloc (#1459)
- Various bug fixes: Break up list and run DropAll func (#1439)
- Add a limit to the size of the batches sent over a stream. (#1412)
- Commit does not panic after Finish, instead returns an error (#1396)
- levels: Compaction incorrectly drops some delete markers (#1422)
- Remove vlog file if bootstrap, syncDir or mmap fails (#1434)
Features:
- Use opencensus for tracing (#1566)
- Export functions from Key Registry (#1561)
- Allow sizes of block and index caches to be updated. (#1551)
- Add metric for number of tables being compacted (#1554)
- feat(info): Show index and bloom filter size (#1543)
- feat(db): Add db.MaxVersion API (#1526)
- Expose DB options in Badger. (#1521)
- Feature: Add a Calloc based Buffer (#1471)
- Add command to stream contents of DB into another DB. (#1463)
- Expose NumAlloc metrics via expvar (#1470)
- Support fully disabling the bloom filter (#1319)
- Add --enc-key flag in badger info tool (#1441)
New APIs
- Badger.DB
- CacheMaxCost (#1551)
- Levels (#1574)
- LevelsToString (#1574)
- Opts (#1521)
- Badger.Options
- WithBaseLevelSize (#1574)
- WithBaseTableSize (#1574)
- WithMemTableSize (#1574)
- Badger.KeyRegistry
- DataKey (#1561)
- LatestDataKey (#1561)
Removed APIs
- Badger.Options
- WithKeepL0InMemory (#1555)
- WithLevelOneSize (#1574)
- WithLoadBloomsOnOpen (#1555)
- WithLogRotatesToFlush (#1574)
- WithMaxTableSize (#1574)
- WithTableLoadingMode (#1555)
- WithTruncate (#1555)
- WithValueLogLoadingMode (#1555)
Fixed
- Compaction: Use separate compactors for L0, L1 (#1466)
- Rework Block and Index cache (#1473)
- Add IsClosed method (#1478)
- Cleanup: Avoid truncating in vlog.Open on error (#1465)
- Cleanup: Do not close cache before compactions (#1464)
New APIs
- Badger.DB
- BlockCacheMetrics (#1473)
- IndexCacheMetrics (#1473)
- Badger.Option
- WithBlockCacheSize (#1473)
- WithIndexCacheSize (#1473)
Removed APIs [Breaking Changes]
- Badger.DB
- DataCacheMetrics (#1473)
- BfCacheMetrics (#1473)
- Badger.Option
- WithMaxCacheSize (#1473)
- WithMaxBfCacheSize (#1473)
- WithKeepBlockIndicesInCache (#1473)
- WithKeepBlocksInCache (#1473)
Fixed
- Remove vlog file if bootstrap, syncDir or mmap fails (#1434)
- levels: Compaction incorrectly drops some delete markers (#1422)
- Replay: Update head for LSM entires also (#1456)
Fixed
- Add a limit to the size of the batches sent over a stream. (#1412)
- Fix Sequence generates duplicate values (#1281)
- Fix race condition in DoesNotHave (#1287)
- Fail fast if cgo is disabled and compression is ZSTD (#1284)
- Proto: make badger/v2 compatible with v1 (#1293)
- Proto: Rename dgraph.badger.v2.pb to badgerpb2 (#1314)
- Handle duplicates in ManagedWriteBatch (#1315)
- Ensure
bitValuePointer
flag is cleared for LSM entry values written to LSM (#1313) - DropPrefix: Return error on blocked writes (#1329)
- Confirm
badgerMove
entry required before rewrite (#1302) - Drop move keys when its key prefix is dropped (#1331)
- Iterator: Always add key to txn.reads (#1328)
- Restore: Account for value size as well (#1358)
- Compaction: Expired keys and delete markers are never purged (#1354)
- GC: Consider size of value while rewriting (#1357)
- Force KeepL0InMemory to be true when InMemory is true (#1375)
- Rework DB.DropPrefix (#1381)
- Update head while replaying value log (#1372)
- Avoid panic on multiple closer.Signal calls (#1401)
- Return error if the vlog writes exceeds more than 4GB (#1400)
Performance
- Clean up transaction oracle as we go (#1275)
- Use cache for storing block offsets (#1336)
Features
- Support disabling conflict detection (#1344)
- Add leveled logging (#1249)
- Support entry version in Write batch (#1310)
- Add Write method to batch write (#1321)
- Support multiple iterators in read-write transactions (#1286)
New APIs
- Badger.DB
- NewManagedWriteBatch (#1310)
- DropPrefix (#1381)
- Badger.Option
- WithDetectConflicts (#1344)
- WithKeepBlockIndicesInCache (#1336)
- WithKeepBlocksInCache (#1336)
- Badger.WriteBatch
- DeleteAt (#1310)
- SetEntryAt (#1310)
- Write (#1321)
Changes to Default Options
- DefaultOptions: Set KeepL0InMemory to false (#1345)
- Increase default valueThreshold from 32B to 1KB (#1346)
Deprecated
Reverts
This sections lists the changes which were reverted because of non-reproducible crashes.
- Compress/Encrypt Blocks in the background (#1227)
2.0.3 - 2020-03-24
Fixed
- Add support for watching nil prefix in subscribe API (#1246)
Performance
- Compress/Encrypt Blocks in the background (#1227)
- Disable cache by default (#1257)
Features
- Add BypassDirLock option (#1243)
- Add separate cache for bloomfilters (#1260)
New APIs
- badger.DB
- BfCacheMetrics (#1260)
- DataCacheMetrics (#1260)
- badger.Options
- WithBypassLockGuard (#1243)
- WithLoadBloomsOnOpen (#1260)
- WithMaxBfCacheSize (#1260)
2.0.3 - 2020-03-24
Fixed
- Add support for watching nil prefix in subscribe API (#1246)
Performance
- Compress/Encrypt Blocks in the background (#1227)
- Disable cache by default (#1257)
Features
- Add BypassDirLock option (#1243)
- Add separate cache for bloomfilters (#1260)
New APIs
- badger.DB
- BfCacheMetrics (#1260)
- DataCacheMetrics (#1260)
- badger.Options
- WithBypassLockGuard (#1243)
- WithLoadBloomsOnOpen (#1260)
- WithMaxBfCacheSize (#1260)
2.0.2 - 2020-03-02
Fixed
- Cast sz to uint32 to fix compilation on 32 bit. (#1175)
- Fix checkOverlap in compaction. (#1166)
- Avoid sync in inmemory mode. (#1190)
- Support disabling the cache completely. (#1185)
- Add support for caching bloomfilters. (#1204)
- Fix int overflow for 32bit. (#1216)
- Remove the ‘this entry should’ve caught' log from value.go. (#1170)
- Rework concurrency semantics of valueLog.maxFid. (#1187)
Performance
- Use fastRand instead of locked-rand in skiplist. (#1173)
- Improve write stalling on level 0 and 1. (#1186)
- Disable compression and set ZSTD Compression Level to 1. (#1191)
2.0.1 - 2020-01-02
New APIs
badger.Options
- WithInMemory (f5b6321)
- WithZSTDCompressionLevel (3eb4e72)
Badger.TableInfo
Features
- Introduce in-memory mode in badger. (#1113)
Fixed
- Limit manifest's change set size. (#1119)
- Cast idx to uint32 to fix compilation on i386. (#1118)
- Fix request increment ref bug. (#1121)
- Fix windows dataloss issue. (#1134)
- Fix VerifyValueChecksum checks. (#1138)
- Fix encryption in stream writer. (#1146)
- Fix segmentation fault in vlog.Read. (header.Decode) (#1150)
- Fix merge iterator duplicates issue. (#1157)
Performance
- Set level 15 as default compression level in Zstd. (#1111)
- Optimize createTable in stream_writer.go. (#1132)
2.0.0 - 2019-11-12
New APIs
badger.DB
- NewWriteBatchAt (7f43769)
- CacheMetrics (b9056f1)
badger.Options
- WithMaxCacheSize (b9056f1)
- WithEventLogging (75c6a44)
- WithBlockSize (1439463)
- WithBloomFalsePositive (1439463)
- WithKeepL0InMemory (ee70ff2)
- WithVerifyValueChecksum (ee70ff2)
- WithCompression (5f3b061)
- WithEncryptionKey (a425b0e)
- WithEncryptionKeyRotationDuration (a425b0e)
- WithChecksumVerificationMode (7b4083d)
Features
- Data cache to speed up lookups and iterations. (#1066)
- Data compression. (#1013)
- Data encryption-at-rest. (#1042)
Fixed
- Fix deadlock when flushing discard stats. (#976)
- Set move key's expiresAt for keys with TTL. (#1006)
- Fix unsafe usage in Decode. (#1097)
- Fix race condition on db.orc.nextTxnTs. (#1101)
- Fix level 0 GC dataloss bug. (#1090)
- Fix deadlock in discard stats. (#1070)
- Support checksum verification for values read from vlog. (#1052)
- Store entire L0 in memory. (#963)
- Fix table.Smallest/Biggest and iterator Prefix bug. (#997)
- Use standard proto functions for Marshal/Unmarshal and Size. (#994)
- Fix boundaries on GC batch size. (#987)
- VlogSize to store correct directory name to expvar.Map. (#956)
- Fix transaction too big issue in restore. (#957)
- Fix race condition in updateDiscardStats. (#973)
- Cast results of len to uint32 to fix compilation in i386 arch. (#961)
- Making the stream writer APIs goroutine-safe. (#959)
- Fix prefix bug in key iterator and allow all versions. (#950)
- Drop discard stats if we can't unmarshal it. (#936)
- Fix race condition in flushDiscardStats function. (#921)
- Ensure rewrite in vlog is within transactional limits. (#911)
- Fix discard stats moved by GC bug. (#929)
- Fix busy-wait loop in Watermark. (#920)
Performance
- Introduce fast merge iterator. (#1080)
- Binary search based table picker. (#983)
- Flush vlog buffer if it grows beyond threshold. (#1067)
- Introduce StreamDone in Stream Writer. (#1061)
- Performance Improvements to block iterator. (#977)
- Prevent unnecessary safecopy in iterator parseKV. (#971)
- Use pointers instead of binary encoding. (#965)
- Reuse block iterator inside table iterator. (#972)
- [breaking/format] Remove vlen from entry header. (#945)
- Replace FarmHash with AESHash for Oracle conflicts. (#952)
- [breaking/format] Optimize Bloom filters. (#940)
- [breaking/format] Use varint for header encoding (without header length). (#935)
- Change file picking strategy in compaction. (#894)
- [breaking/format] Block level changes. (#880)
- [breaking/format] Add key-offset index to the end of SST table. (#881)
1.6.0 - 2019-07-01
This is a release including almost 200 commits, so expect many changes - some of them not backward compatible.
Regarding backward compatibility in Badger versions, you might be interested on reading VERSIONING.md.
Note: The hashes in parentheses correspond to the commits that impacted the given feature.
New APIs
badger.DB
- DropPrefix (291295e)
- Flatten (7e41bba)
- KeySplits (4751ef1)
- MaxBatchCount (b65e2a3)
- MaxBatchSize (b65e2a3)
- PrintKeyValueHistogram (fd59907)
- Subscribe (26128a7)
- Sync (851e462)
badger.DefaultOptions() and badger.LSMOnlyOptions() (91ce687)
- badger.Options.WithX methods
badger.Entry (e9447c9)
- NewEntry
- WithMeta
- WithDiscard
- WithTTL
badger.Item
- KeySize (fd59907)
- ValueSize (5242a99)
badger.IteratorOptions
- PickTable (7d46029, 49a49e3)
- Prefix (7d46029)
badger.Logger (fbb2778)
badger.Options
- CompactL0OnClose (7e41bba)
- Logger (3f66663)
- LogRotatesToFlush (2237832)
badger.Stream (14cbd89, 3258067)
badger.StreamWriter (7116e16)
badger.TableInfo.KeyCount (fd59907)
badger.TableManifest (2017987)
badger.Tx.NewKeyIterator (49a49e3)
badger.WriteBatch (6daccf9, 7e78e80)
Modified APIs
Breaking changes:
- badger.DefaultOptions and badger.LSMOnlyOptions are now functions rather than variables (91ce687)
- badger.Item.Value now receives a function that returns an error (439fd46)
- badger.Txn.Commit doesn't receive any params now (6daccf9)
- badger.DB.Tables now receives a boolean (76b5341)
Not breaking changes:
- badger.LSMOptions changed values (799c33f)
- badger.DB.NewIterator now allows multiple iterators per RO txn (41d9656)
- badger.Options.TableLoadingMode's new default is options.MemoryMap (6b97bac)
Removed APIs
- badger.ManagedDB (d22c0e8)
- badger.Options.DoNotCompact (7e41bba)
- badger.Txn.SetWithX (e9447c9)
Tools:
- badger bank disect (13db058)
- badger bank test (13db058) --mmap (03870e3)
- badger fill (7e41bba)
- badger flatten (7e41bba)
- badger info --histogram (fd59907) --history --lookup --show-keys --show-meta --with-prefix (09e9b63) --show-internal (fb2eed9)
- badger benchmark read (239041e)
- badger benchmark write (6d3b67d)
1.5.5 - 2019-06-20
- Introduce support for Go Modules
1.5.3 - 2018-07-11
Bug Fixes:
- Fix a panic caused due to item.vptr not copying over vs.Value, when looking for a move key.
1.5.2 - 2018-06-19
Bug Fixes:
Fix the way move key gets generated.
If a transaction has unclosed, or multiple iterators running simultaneously, throw a panic. Every iterator must be properly closed. At any point in time, only one iterator per transaction can be running. This is to avoid bugs in a transaction data structure which is thread unsafe.
Warning: This change might cause panics in user code. Fix is to properly close your iterators, and only have one running at a time per transaction.
1.5.1 - 2018-06-04
Bug Fixes:
Cleanup:
- Clarify comments and documentation.
- Move badger tool one directory level up.
1.5.0 - 2018-05-08
- Introduce
NumVersionsToKeep
option. This option is used to discard many versions of the same key, which saves space. - Add a new
SetWithDiscard
method, which would indicate that all the older versions of the key are now invalid. Those versions would be discarded during compactions. - Value log GC moves are now bound to another keyspace to ensure latest versions of data are always at the top in LSM tree.
- Introduce
ValueLogMaxEntries
to restrict the number of key-value pairs per value log file. This helps bound the time it takes to garbage collect one file.
1.4.0 - 2018-05-04
- Make mmap-ing of value log optional.
- Run GC multiple times, based on recorded discard statistics.
- Add MergeOperator.
- Force compact L0 on clsoe (#439).
- Add truncate option to warn about data loss (#452).
- Discard key versions during compaction (#464).
- Introduce new
LSMOnlyOptions
, to make Badger act like a typical LSM based DB.
Bug fix:
- (Temporary) Check max version across all tables in Get (removed in next release).
- Update commit and read ts while loading from backup.
- Ensure all transaction entries are part of the same value log file.
- On commit, run unlock callbacks before doing writes (#413).
- Wait for goroutines to finish before closing iterators (#421).
1.3.0 - 2017-12-12
- Add
DB.NextSequence()
method to generate monotonically increasing integer sequences. - Add
DB.Size()
method to return the size of LSM and value log files. - Tweaked mmap code to make Windows 32-bit builds work.
- Tweaked build tags on some files to make iOS builds work.
- Fix
DB.PurgeOlderVersions()
to not violate some constraints.
1.2.0 - 2017-11-30
- Expose a
Txn.SetEntry()
method to allow setting the key-value pair and all the metadata at the same time.
1.1.1 - 2017-11-28
- Fix bug where txn.Get was returing key deleted in same transaction.
- Fix race condition while decrementing reference in oracle.
- Update doneCommit in the callback for CommitAsync.
- Iterator see writes of current txn.
1.1.0 - 2017-11-13
- Create Badger directory if it does not exist when
badger.Open
is called. - Added
Item.ValueCopy()
to avoid deadlocks in long-running iterations - Fixed 64-bit alignment issues to make Badger run on Arm v7
1.0.1 - 2017-11-06
- Fix an uint16 overflow when resizing key slice