my.cnf tuning

Full MyRocks configuration example for uses other than linkbench

[mysqld]
rocksdb
default-storage-engine=rocksdb
skip-innodb
default-tmp-storage-engine=MyISAM

rocksdb_max_open_files=-1
rocksdb_max_background_compactions=16
rocksdb_max_total_wal_size=4G
rocksdb_max_background_flushes=4
rocksdb_block_size=16384
rocksdb_block_cache_size=32G
rocksdb_table_cache_numshardbits=6

# rate limiter
rocksdb_bytes_per_sync=4194304
rocksdb_wal_bytes_per_sync=4194304
rocksdb_rate_limiter_bytes_per_sec=104857600 #100MB/s

# triggering compaction if there are many sequential deletes
rocksdb_compaction_sequential_deletes=1800
rocksdb_compaction_sequential_deletes_window=2000

rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;
level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;max_write_buffer_number=4;
compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression:kZlibCompression;
compression_opts=-14:1:0;
block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true

Full MyRocks configuration example for linkbench

[mysqld]
rocksdb
default-storage-engine=rocksdb
skip-innodb
default-tmp-storage-engine=MyISAM

rocksdb_max_open_files=-1
rocksdb_max_background_compactions=16
rocksdb_max_total_wal_size=4G
rocksdb_max_background_flushes=4
rocksdb_block_size=16384
rocksdb_block_cache_size=32G
rocksdb_table_cache_numshardbits=6

# rate limiter
rocksdb_bytes_per_sync=4194304
rocksdb_wal_bytes_per_sync=4194304
rocksdb_rate_limiter_bytes_per_sec=104857600 #100MB/s

# triggering compaction if there are many sequential deletes
rocksdb_compaction_sequential_deletes=1800
rocksdb_compaction_sequential_deletes_window=2000

rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;
level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;max_write_buffer_number=4;
compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression:kZlibCompression;
compression_opts=-14:1:0;
block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=0};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;memtable_prefix_bloom_bits=41943040;memtable_prefix_bloom_probes=6;prefix_extractor=capped:12

Tuning Tips

• Compression
• Set kNoCompression (or kSnappyCompression) on L0-1 or L0-2
• If using zlib compression, set kZlibCompression on other levels
• If using zlib compression, set compression level accordingly. The above example (compression_opts=-14:1:0) uses zlib compression level 1.
• Data blocks, files and compactions
• Set level_compaction_dynamic_level_bytes=true
• Set proper rocksdb_block_size (default 4096). Larger block size will reduce space but increase CPU overhead because MyRocks has to uncompress many more bytes. There is a trade-off between space and CPU usage.
• Set rocksdb_max_open_files=-1. If setting greater than 0, RocksDB still use table_cache, which will lock a mutex every time you access the file. I think you'll see much greater benefit with -1 because then you will not need to go through LRUCache to get the table you need.
• Set reasonable rocksdb_max_background_compactions
• Set reasonable rocksdb_max_background_flushes
• Don't set too small target_file_size_base. Too many sst files makes operations more difficult.
• Set Rate Limiter. Without rate limiter, compaction very often writes 300~500MB/s on pure flash, which may cause short stalls. On 4x MyRocks testing, 40MB/s rate limiter per instance gave pretty stable results (less than 200MB/s peak from iostat).
• Enable row checksum. Row checksum adds CRC32 for key and value, which will be useful to detect corrupted rows. But do not enable for 100% rows. 100% rows increases used space a lot (~25% in LinkBench). 5% will give good enough coverage.
• Bloom Filter
• Configure bloom filter and Prefix Extractor. Full Filter is recommended (Block based filter does not work for Get() + prefix bloom). Prefix extractor can be configured per column family. If using one BIGINT column as a primary key, recommended bloom filter size is 12 (first 4 bytes are for internal index id + 8 byte BIGINT).
• Configure Memtable bloom filter. Memtable bloom filter is useful to reduce CPU usage, if you see high CPU usage at rocksdb::MemTable::KeyComparator. Size depends on Memtable size. Set memtable_prefix_bloom_bits=41943040 for 128MB Memtable (30/128M=4M keys * 10 bits per key)
• Cache
• Do not set block_cache at rocksdb_default_cf_options (block_based_table_factory). If you do provide a block cache size on a default column family, the same cache is NOT reused for all such column families.
• Consider setting shared write buffer size (db_write_buffer_size)

Verifying parameters

To verify if configurations are set correctly, view LOG file and search parameter name. LOG file is located at $datadir/.rocksdb/LOG.