Address indexing backport (#201)

* First backport of address indexing from Zend

Todo:
 - txindex.py not working (neither on Zend repo)

* Fixed conflict for DB index tags

* Added some furhter code from zen addressIndexing branch; Added handling for spentIndex in disconnect block stage

* Updated structures for LevelDB indexes

Updated the value structures for the following indexes:
 - TxIndex
 - AddressIndex
 - AddressUnspentIndex

* Changed interface to mempool address indexing functions for accepting CTransactionBase

* Improved CTxIndexValue constructor

* WIP: first versionof code implementation for mempool handling of address indexing data: non tested yet

* Changes on getaddressbalance to don't include by default immature BTs, and added new field immature in the response

* Changes on getaddressutxo to don't include by default immature BTs and added new field maurityHeight for each utxo

* Moved some code for better organization

* WIP: Address Index: Added logic for modifying output entries for a cerificate superseeded by a new top quality accepted in mempool

* Added python test for getaddress* RPC commands

* Minor modifications for certificate handling flow; removed unused code

* Improved code related to address indexing

* Added getaddressmempool check inside sc_cert_addressindex.py test

* Added update of low quality certs in address index data when the top quality cert is evicted from mempool

* Changed value of outStatus for input records in address index mempool map (from -1 to 0xff); used GetType() func instead of if/else blocks

* Used GetType()/GetAddressHash instead of if/else blocks also in addSpentIndex

* Updated the addressindexing Python test with additional scenarios

* Added bwt related dat to json result of getaddressutxos rpc cmd

* Moved instantiation of height and best block hash at the implementation start using lock in getaddressutxos cmd

* Added rpc command getcertmaturityinfo; it relies on txindex option to be on

* Implemented explorer indexes update for certificates in the ConnectBlock

* Moved to_satoshis() func into util.py

* Fixed the serialization of "maturityHeight" for the SpentIndex

* Updated sc_cert_addressindex to test the DisconnectTip

* Implemented indexes update for certificates in the DiconnectBlock

* Implemented indexes update when sidechain ceased state changes

* Added sc_cert_addrmempool.py test expecially aimed at getaddressmempool rpc command

* Added vcsw_ccin field inside TxToJSONExpanded

* Minor fixes for indexes management

- Removed some unused input parameters
- Added a flag check inside the DisconnectBlock function

* Added new index update test cases for ceased sidechains

* Fix for py test using createrawcertificate

* Improved AddressIndex update procedure

Improved the code so that AddressIndex entries are removed and updated in one single batch operation (instead of two different ones).

* Improved code reuse for serialization of VARINT with sign

* Fixed issues after rebasing onto testnet_staging

* Added a conditional compilation flag for AddressIndex related features

* Fixed a flag for optionally run address indexing Python tests

* Added extended information to certificate JSON representation

* Added some code under address indexing conditional compilation

* Fixed the regression tests

* Improved management of the -addressindex parameter

* Removed some debug prints from the build configuration file

* Fix Python test failing after extending getrawtransaction JSON fields

* Fixed a Python test failing after requiring txIndex for addressIndex

* Renamed JSON field 'valueSat' as 'valueZat'

* Put additional code under conditional compilation for Address Indexing

* Removed unused references to wallet-utility

* Improved comment about address-indexing flag

* Added comments to #endif for ENABLE_ADDRESS_INDEXING macro

* Removed nonexistent header from Makefile

* Added some LevelDB collection IDs into the Address Indexing section

* Improved management of flags "dbmaxopenfiles" and "dbcompression"

- Renamed "dbmaxopenfiles" as "blocktreedbmaxopenfiles"
- Renamed "dbcompression" as "blocktreedbcompression"
- Put both flags under #ifdef
- Fixed the default options for the Block Tree LevelDB

* Moved blockOnchainActive() function under #ifdef

* Moved some Address Indexing related code under #ifdef

* Fixed the implementation of blockOnchainActive() function

* Added to ConnectBlock a flag to skip writing Address Indexing DB

* Restored some code to the previous location

* Added management for TxIndex update during Connect/Disconnect of block

* Substituted an assert with an if, since we must update the txindex entry only if it is there

* Added the state INVALID to the output of the rpc cmd getcertmaturityinfo

* Fixed txindex.py test and added it to regression script

* Added checks to update TxIndex DB only if the related flag is enabled

* Added some test for getcertmaturityinfo; to be continued

* Included search in the mempool for the RPC command getcertmaturityinfo

* Fixed sc_cert_addrmempool.py test

* Added a test case for a certificate removed from the blockchain

- Added the new test case to sc_cert_addrmempool.py
- Added a new RPC command to clear the mempool of a node (REGTEST only)

* Added the cleanup of other data structures in the mempool clean() method

* Enabled clearmempool rpc command also for testnet

* Changed the status string of a mempool cert in the output of rpc cmd getcertmaturityinfo

* Added a flag to enable/disable write of TxIndex when checking blocks

* Moved a variable initialization into the correct scope

* Added verifychain rpc call to py tests

Co-authored-by: Alberto Sala <alberto.sala@nttdata.com>
Co-authored-by: Marco <marco.olivero94@libero.it>

Makefile.am

@@ -66,6 +66,7 @@ $(BITCOIN_WIN_INSTALLER): all-recursive
 	$(MKDIR_P) $(top_builddir)/release
 	STRIPPROG="$(STRIP)" $(INSTALL_STRIP_PROGRAM) $(BITCOIND_BIN) $(top_builddir)/release
 	STRIPPROG="$(STRIP)" $(INSTALL_STRIP_PROGRAM) $(BITCOIN_CLI_BIN) $(top_builddir)/release
+#	STRIPPROG="$(STRIP)" $(INSTALL_STRIP_PROGRAM) $(WALLET_UTILITY_BIN) $(top_builddir)/release
 	@test -f $(MAKENSIS) && $(MAKENSIS) -V2 $(top_builddir)/share/setup.nsi || \
 	  echo error: could not build $@
 	@echo built $@
@@ -146,6 +147,8 @@ $(BITCOIND_BIN): FORCE
 $(BITCOIN_CLI_BIN): FORCE
 	$(MAKE) -C src $(@F)
 
+#$(WALLET_UTILITY_BIN): FORCE
+#	$(MAKE) -C src $(@F)
 
 if USE_LCOV
 

configure.ac

@@ -186,6 +186,13 @@ AC_ARG_ENABLE([werror],
     [enable_werror=$enableval],
     [enable_werror=no])
 
+# Enable AddressIndexing (typically for usage with Explorer)
+# Note that this flag enables 3 indexes: AddressIndex/AddressUnspentIndex, SpentIndex and TimestampBlockIndex
+AC_ARG_ENABLE([address-indexing],
+    [AS_HELP_STRING([--enable-address-indexing])],
+    [enable_address_indexing=$enableval],
+    [enable_address_indexing=no])
+
 AC_LANG_PUSH([C++])
 AX_CHECK_COMPILE_FLAG([-Werror],[CXXFLAG_WERROR="-Werror"],[CXXFLAG_WERROR=""])
 
@@ -891,6 +898,16 @@ else
   AC_MSG_RESULT([no])
 fi
 
+dnl enable address indexing
+AC_MSG_CHECKING([if address indexing should be enabled])
+if test x$enable_address_indexing != xno; then
+  AC_MSG_RESULT(yes)
+  AC_DEFINE([ENABLE_ADDRESS_INDEXING],[1],[Define to 1 to enable address indexing functions])
+
+else
+  AC_MSG_RESULT(no)
+fi
+
 if test x$build_bitcoin_utils$build_bitcoin_libs$build_bitcoind$use_tests = xnononono; then
   AC_MSG_ERROR([No targets! Please specify at least one of: --with-utils --with-libs --with-daemon or --enable-tests])
 fi
@@ -906,6 +923,7 @@ AM_CONDITIONAL([GLIBC_BACK_COMPAT],[test x$use_glibc_compat = xyes])
 AM_CONDITIONAL([HARDEN],[test x$use_hardening = xyes])
 AM_CONDITIONAL([ASAN],[test x$use_asan = xyes])
 AM_CONDITIONAL([TSAN],[test x$use_tsan = xyes])
+AM_CONDITIONAL([ENABLE_ADDRESS_INDEXING],[test x$enable_address_indexing = xyes])
 
 AC_DEFINE(CLIENT_VERSION_MAJOR, _CLIENT_VERSION_MAJOR, [Major version])
 AC_DEFINE(CLIENT_VERSION_MINOR, _CLIENT_VERSION_MINOR, [Minor version])
@@ -1001,6 +1019,7 @@ echo "  with zmq      = $use_zmq"
 echo "  with test     = $use_tests"
 echo "  debug enabled = $enable_debug"
 echo "  werror        = $enable_werror"
+echo "  addr index    = $enable_address_indexing"
 echo 
 echo "  target os     = $TARGET_OS"
 echo "  build os      = $BUILD_OS"

doc/man/zend.1

@@ -110,6 +110,21 @@ Create new files with system default permissions, instead of umask 077
 .IP
 Maintain a full transaction index, used by the getrawtransaction rpc
 call (default: 0)
+.HP
+\fB\-addressindex\fR
+.IP
+Maintain a full address index, used to query for the balance, txids and
+unspent outputs for addresses (default: 0)
+.HP
+\fB\-timestampindex\fR
+.IP
+Maintain a timestamp index for block hashes, used to query blocks hashes
+by a range of timestamps (default: 0)
+.HP
+\fB\-spentindex\fR
+.IP
+Maintain a full spent index, used to query the spending txid and input
+index for an outpoint (default: 0)
 .PP
 Connection options:
 .HP

qa/pull-tester/rpc-tests.sh

@@ -125,6 +125,7 @@ testScripts=(
   'sc_stale_ft_and_mbtr.py'
   'sc_cert_getblocktemplate.py'
   'sc_cert_bt_immature_balances.py'
+  'txindex.py'
 );
 testScriptsExt=(
   'getblocktemplate_longpoll.py'
@@ -167,6 +168,14 @@ if [ "x$ENABLE_PROTON" = "x1" ]; then
   testScripts+=('proton_test.py')
 fi
 
+if [ "x$ENABLE_ADDRESS_INDEX" = "x1" ]; then
+  testScripts+=('addressindex.py'
+                'spentindex.py'
+                'timestampindex.py'
+                'sc_cert_addressindex.py'
+                'sc_cert_addrmempool.py')
+fi
+
 # include extended tests
 if [ ! -z "$EXTENDED" ] && [ "${EXTENDED}" = "true" ]; then
   testScripts+=( "${testScriptsExt[@]}" )

qa/pull-tester/tests-config.sh.in

@@ -12,6 +12,7 @@ EXEEXT="@EXEEXT@"
 @BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=1
 @ENABLE_ZMQ_TRUE@ENABLE_ZMQ=1
 @ENABLE_PROTON_TRUE@ENABLE_PROTON=1
+@ENABLE_ADDRESS_INDEXING_TRUE@ENABLE_ADDRESS_INDEX=1
 
 REAL_BITCOIND="$BUILDDIR/src/zend${EXEEXT}"
 REAL_BITCOINCLI="$BUILDDIR/src/zen-cli${EXEEXT}"