RocksDB ledger backend testing

Information about our RocksDB implementation can be found here:
https://docs.nano.org/running-a-node/rocksdb-ledger-backend/

Initial testing has shown that bootstrapping on lower end devices is faster, while slower on higher end devices. Details below:

Ubuntu, 12 threads 3.4GHZ CPU, 16GB RAM, SSD:

--- LMDB RocksDB
Full sync (live) 3h 15m 4h 15m

Ubuntu, 1vCPU, 2GB RAM, SSD:

--- LMDB RocksDB
* Parital sync 7h 15m 5h 30m

*3.5 million checked blocks

Figure 1 below shows a partial bootstrap of the RockDB backend on the left and LMDB on the right. The peaks in the Disk I/O are due to flushing in bulk and compaction process. While more extreme the overall area underneath the graphs is lower for RocksDB so there is less overall disk I/O.


Figure 1 - Various computer statics for the 1vCPU, 2GB RAM Ubuntu 18.04 Server

However there are a lot of settings which can be changed (see previous link). The default config options used in the statistics above have since been lowered, so it would be beneficial to get comparisons of bootstrapping on as many devices as possible using both the default and extreme values where lots of RAM is available. If changing memtable_size to be higher, also modify total_memtable_size as that will otherwise limit the effectiveness of it.

Where possible, testing during spam tests on the Beta network can provide valuable information as well, as it is expected this is where the RocksDB backend will outperform the LMDB backend.

I did some testing on Beta doing a bootstrap with RocksDB. I noticed the cementing speed was relatively low. I was able to increase the speed by increasing block_cache. I’m curious if there are any machine spec to optimal settings charts. For example if there is 4GB of memory free what should the RocksDB settings look like compared to 16GB free? Would anything change if using an SSD vs HDD or different CPU’s? Something like this would be helpful for tuning on different machine types instead of needing to trial and error changes.

I did run into the following error after leaving it running for a while. Should memtables be increased any in my config?

Build Info: 3d834009 "Microsoft Visual C++ version " "14.1" "BOOST 106700" BUILT "Nov 1 2019"
Database backend: RocksDB
Assertion (status.ok ()) failed C:\projects\myproject\nano\node\rocksdb\rocksdb_txn.cpp:61

0# argon2i_ctx in nano_node
1# argon2i_ctx in nano_node
2# argon2i_ctx in nano_node
3# argon2i_ctx in nano_node
4# argon2i_ctx in nano_node
5# argon2i_ctx in nano_node
6# argon2i_ctx in nano_node
7# argon2i_ctx in nano_node
8# argon2i_ctx in nano_node
9# beginthreadex in ucrtbase
10# BaseThreadInitThunk in KERNEL32
11# RtlUserThreadStart in ntdll

As discussed in DM, there was a bug when reading the total memtable size in the config. Should anyone else be testing and encountering an issue similar to this, please either set total_memtable_size=0 in the config-node.toml [rocksdb] section or use the current master build.

Been running with total_memtable_size=0 for 24 hours without a crash, including through a 200k block test. :ok_hand:

3 Likes