Merge pull request #84 from Chia-Network/alpha-1.3

Alpha 1.3 FullNode performance improvements - Syncing up to the blockchain by importing all blocks is faster due to improvements in VDF verification speed and multithreading block verification. Also fixed a bug where the node may not sync if it restarts close to a tip. VDF improvements - VDF verification and generation speed has increased and dependence on flint2 has been removed. We wish to thank Dr. William Hart (@wbhart) for dual licensing parts of his contributions in [FLINT](http://www.flintlib.org/) and [Antic](https://github.com/wbhart/antic) for inclusion in the Chia blockchain. Implemented an RPC interface with JSON serialization for streamables - currently on port 8555 - and moved the ssh UI to use RPC. Mongodb deprecated. Moved to [SQLite](https://www.sqlite.org/) for the blockchain database. This will require nodes to re-sync with the network. Luckily this is now faster. Changed the displayed process names for harvester, farmer, fullnode, timelords, and VDFs to to chia_full node, chia_harvester, etc. Fixed a bug that could cause inadvertent shutdown of other processes like an ongoing plotting session when new chia services were started. Added details on how to contribute in CONTRIBUTING.md. Thanks @RichardLitt. Also moved documentation from inside the repository to the [wiki](https://github.com/Chia-Network/chia-blockchain/wiki). Added color logging and now chia_harvester will periodically announce which plots it is currently farming and their k sizes. Fixed a typo in the UI. Hat tip to @lvcivs for the pr. Clarified the minimum version of boost required to build timelord/VDFs. Hat tip @AdrianScott Lots of smaller bug and documentation fixes.
Chia-Network · Jan 22, 2020 · 4eebd09 · 4eebd09
2 parents 281be67 + 9db4838
commit 4eebd09
Show file tree

Hide file tree

Showing 632 changed files with 46,810 additions and 52,418 deletions.
diff --git a/.github/workflows/pythonpackage.yml b/.github/workflows/pythonpackage.yml
@@ -9,15 +9,14 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        python-version: [3.7]
         os: [ubuntu-latest, macOS-latest]
 
     steps:
     - uses: actions/checkout@v1
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v1
+    - name: Setup Python environment
+      uses: actions/setup-python@v1.1.1
       with:
-        python-version: ${{ matrix.python-version }}
+        python-version: 3.7 # optional, default is 3.x
     - name: Install dependencies
       env:
         CHIA_MACHINE_SSH_KEY: ${{ secrets.CHIA_MACHINE_SSH_KEY }}
@@ -26,17 +25,11 @@ jobs:
         eval "$(ssh-agent -s)"
         ssh-add - <<< "${CHIA_MACHINE_SSH_KEY}"
         git submodule update --init --recursive
-        sudo apt-get install --no-install-recommends mongodb|| echo ""
-        brew tap mongodb/brew || echo ""
-        brew update && brew install gmp mongodb-community@4.2 || echo ""
+        brew update && brew install gmp || echo ""
         python3 -m venv .venv
         . .venv/bin/activate
-        pip install wheel # For building blspy
         pip install -e .
         pip install -r requirements.txt
-    - name: Start mongodb
-      run: |
-        mongod --dbpath ./db/ &
     - name: Lint with flake8
       run: |
         ./.venv/bin/flake8 src

diff --git a/.gitignore b/.gitignore
@@ -11,6 +11,13 @@ __pycache__/
 # Database
 nohup.out
 mongod.log*
+fndb_test*
+blockchain_test*
+*.db
+
+# Logs
+*.log
+*.out
 
 # Keys and plot files
 config/keys.yaml
@@ -52,4 +59,4 @@ pip-delete-this-directory.txt
 .vscode
 
 # Packaging
-chia-blockchain.tar.gz
+chia-blockchain.tar.gz
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,46 @@
+## Introduction
+Welcome to the chia-blockchain project!
+We are happy that you are taking a look at the code for Chia, a proof of space and time cryptocurrency.
+
+A lot of fascinating new cryptography and blockchain concepts are used and implemented here.
+This repo includes the code for the Chia full node, farmer, and timelord (in src), which are all written in python.
+It also includes a verifiable delay function implementation under lib/chiavdf (in c/c++), and a proof of space implementation under lib/chiapos.
+
+If you want to learn more about this project, read the [wiki](https://github.com/Chia-Network/chia-blockchain/wiki), or check out the [green paper](https://www.chia.net/assets/ChiaGreenPaper.pdf).
+
+### Contributions
+We would be pleased to accept code contributions to this project.
+As we are in the alpha stage, the main priority is getting a robust blockchain up and running, with as many of the mainnet features as possible.
+You can visit our [Trello project board](https://trello.com/b/ZuNx7sET) to get a sense of what is in the backlog.
+Generally things to the left are in progess or done. Some things go through "Coming up soon" but some will come directly out of other columns. 
+Usually the things closer to the top of each column are the ones that will be worked on soonest.
+If you are interested in cryptography, math, or just like hacking in python, there are many interesting problems to work on.
+Contact any of the team members on keybase: https://keybase.io/team/chia_network.public, which we use as the main communication method and you can comment on any Trello card.
+
+### Run tests and linting
+The first time the tests are run, BlockTools will create and persist many plots. These are used for creating
+proofs of space during testing. The next time tests are run, this won't be necessary.
+
+```bash
+black src tests && flake8 src && mypy src tests
+py.test tests -s -v
+```
+Black is used as an automatic style formatter to make things easier, and flake8 helps ensure consistent style.
+Mypy is very useful for ensuring objects are of the correct type, so try to always add the type of the return value, and the type of local variables.
+
+### Configure VS code
+1. Install Python extension
+2. Set the environment to ./.venv/bin/python
+3. Install mypy plugin
+4. Preferences > Settings > Python > Linting > flake8 enabled
+5. Preferences > Settings > Python > Linting > mypy enabled
+7. Preferences > Settings > Formatting > Python > Provider > black
+6. Preferences > Settings > mypy > Targets: set to ./src and ./tests
+
+### Submit changes
+To submit changes, please make a pull request to the appropriate development branch.
+For example, after the 1.2 release, the 1.3 branch is used for development, etc.
+The master branch is updated with the latest releases only.
+
+### Copyright
+By contributing to this repository, you agree to license your work under the Apache License Version 2.0, or the MIT License, or release your work to the public domain. Any work contributed where you are not the original author must contain its license header with the original author(s) and be in the public domain, or licensed under the Apache License Version 2.0 or the MIT License.
diff --git a/README.md b/README.md
@@ -1,37 +1,34 @@
 # chia-blockchain
-Python 3.7 is used for this project. Make sure your default python version is >=3.7 by typing python3.
+Please check out the [wiki](https://github.com/Chia-Network/chia-blockchain/wiki) and [FAQ](https://github.com/Chia-Network/chia-blockchain/wiki/FAQ) for information on this project.
 
-You will need to enable [UPnP](https://www.homenethowto.com/ports-and-nat/upnp-automatic-port-forward/) on your router or add a NAT (for IPv4 but not IPv6) and firewall rule to allow TCP port 8444 access to your peer. These methods tend to be router make/model specific.
+Python 3.7 is required. Make sure your default python version is >=3.7 by typing `python3`.
+
+You will need to enable [UPnP](https://www.homenethowto.com/ports-and-nat/upnp-automatic-port-forward/) on your router or add a NAT (for IPv4 but not IPv6) and firewall rules to allow TCP port 8444 access to your peer. These methods tend to be router make/model specific.
 
 For alpha testnet most should only install harvesters, farmers, plotter and full nodes. Building timelords and VDFs is for sophisticated users in most environments. Chia Network and additional volunteers are running sufficient time lords for testnet consensus.
 
-## Install harvester, farmer, plotter, and full node
+## Step 1: Install harvester, farmer, plotter, and full node
 
 ### Debian/Ubuntu
 
 ```bash
 sudo apt-get update
-sudo apt-get install build-essential cmake python3-dev python3-venv libssl-dev libffi-dev --no-install-recommends
+sudo apt-get install build-essential git cmake libgmp3-dev --no-install-recommends
+sudo apt-get install python3-dev python3-venv --no-install-recommends
 
 git clone https://github.com/Chia-Network/chia-blockchain.git
 cd chia-blockchain
 
 sh install.sh
 
-# Install MongoDB Community Edition
-# Instructions - https://docs.mongodb.com/manual/administration/install-on-linux/
-
-# Run mongo database if not running system-wide
-mongod --fork --dbpath ./db/ --logpath mongod.log
-
 . .venv/bin/activate
 ```
 ### Amazon Linux 2
 
 ```bash
 sudo yum update
-sudo yum install gcc-c++ cmake3 wget git openssl openssl-devel
-sudo yum install python3 python3-devel libffi-devel
+sudo yum install gcc-c++ cmake3 wget git openssl openssl-devel 
+sudo yum install python3 python3-devel libffi-devel gmp-devel
 
 # CMake - add a symlink for cmake3 - required by blspy
 sudo ln -s /usr/bin/cmake3 /usr/local/bin/cmake
@@ -41,12 +38,6 @@ cd chia-blockchain
 
 sh install.sh
 
-# Install MongoDB Community Edition
-# Instructions - https://docs.mongodb.com/manual/administration/install-on-linux/
-
-# Run mongo database if not running system-wide
-mongod --fork --dbpath ./db/ --logpath mongod.log
-
 . .venv/bin/activate
 ```
 ### CentOS 7
@@ -55,6 +46,7 @@ mongod --fork --dbpath ./db/ --logpath mongod.log
 sudo yum update
 sudo yum install centos-release-scl-rh epel-release
 sudo yum install devtoolset-8-toolchain cmake3 libffi-devel
+sudo yum install gmp-devel libsqlite3x-devel
 sudo yum install wget git openssl openssl-devel
 
 # CMake - add a symlink for cmake3 - required by blspy
@@ -71,13 +63,6 @@ git clone https://github.com/Chia-Network/chia-blockchain.git
 cd chia-blockchain
 
 sh install.sh
-
-# Install MongoDB Community Edition
-# Instructions - https://docs.mongodb.com/manual/administration/install-on-linux/
-
-# Run mongo database if not running system-wide
-mongod --fork --dbpath ./db/ --logpath mongod.log
-
 . .venv/bin/activate
 ```
 
@@ -98,16 +83,12 @@ sudo apt-get -y update
 sudo apt-get -y upgrade
 sudo do-release-upgrade
 
-sudo apt-get install -y build-essential cmake python3-dev python3-venv mongodb software-properties-common --no-install-recommends
+sudo apt-get install -y build-essential cmake python3-dev python3-venv software-properties-common libgmp3-dev --no-install-recommends
 
 git clone https://github.com/Chia-Network/chia-blockchain.git
 cd chia-blockchain
 
 sudo sh install.sh
-
-# Run mongo database if not running system-wide
-mongod --fork --dbpath ./db/ --logpath mongod.log
-
 . .venv/bin/activate
 ```
 
@@ -118,7 +99,7 @@ Each line that starts with `pip ...` becomes `python -m pip ...`
 
 ```bash
 sudo apt-get -y update
-sudo apt-get install -y build-essential cmake python3-dev python3-venv mongodb software-properties-common --no-install-recommends
+sudo apt-get install -y build-essential cmake python3-dev python3-venv software-properties-common libgmp3-dev --no-install-recommends
 
 # Install python3.7 with ppa
 sudo add-apt-repository -y ppa:deadsnakes/ppa
@@ -129,35 +110,26 @@ git clone https://github.com/Chia-Network/chia-blockchain.git
 cd chia-blockchain
 
 sudo sh install.sh
-
-# Run mongo database if not running system-wide
-mongod --fork --dbpath ./db/ --logpath mongod.log
-
 . .venv/bin/activate
 ```
 
 ### MacOS
 Make sure [brew](https://brew.sh/) is available before starting the setup.
 ```bash
-brew tap mongodb/brew
 brew upgrade python
-brew install cmake mongodb-community@4.2
+brew install cmake gmp
 
 git clone https://github.com/Chia-Network/chia-blockchain.git
 cd chia-blockchain
 
 sh install.sh
-
-# Run mongo database if not running system-wide
-mongod --fork --dbpath ./db/ --logpath mongod.log
-
 . .venv/bin/activate
 ```
 
 
-## Install timelord
+## Step 2: Install timelord (optional)
 Note: this step is needed only if you intend to run a timelord or a local simulation.
-These assume you've already successfully installed harvester, farmer, plotting, and full node above.
+These assume you've already successfully installed harvester, farmer, plotting, and full node above. boost 1.67 or newer is required on all platforms.
 ### Ubuntu/Debian
 ```bash
 cd chia-blockchain
@@ -169,7 +141,7 @@ sh install_timelord.sh
 #Only for Amazon Linux 2
 sudo amazon-linux-extras install epel
 
-sudo yum install gmp-devel mpfr-devel
+sudo yum install mpfr-devel
 
 # Install Boost 1.72.0
 wget https://dl.bintray.com/boostorg/release/1.72.0/source/boost_1_72_0.tar.gz
@@ -178,11 +150,6 @@ cd boost_1_72_0
 ./bootstrap.sh --prefix=/usr/local
 sudo ./b2 install --prefix=/usr/local --with=all; cd ..
 
-# Install Flint2
-git clone https://github.com/wbhart/flint2
-cd flint2; ./configure; sudo make install; cd ..
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
-
 cd chia-blockchain
 
 sh install_timelord.sh
@@ -205,41 +172,43 @@ sudo apt-get install libboost1.70 libboost1.70-dev
 
 ### MacOS
 ```bash
-brew install boost gmp mpir mpfr
+brew install boost
 
 cd chia-blockchain
 
-git clone https://github.com/wbhart/flint2
-
 sh install_timelord.sh
 ```
 
-## Generate keys
+## Step 3: Generate keys
 First, create some keys by running the following script:
 ```bash
 python -m scripts.regenerate_keys
 ```
 
-## Run a full node
-To run a full node on port 8002, and connect to the testnet, run the following command.
-This wil also start an ssh server in port 8222 for the UI, which you can connect to
+
+## Step 4a: Run a full node
+To run a full node on port 8444, and connect to the testnet, run the following command.
+This will also start an ssh server in port 8222 for the UI, which you can connect to
 to see the state of the node.
 ```bash
-python -m src.server.start_full_node "127.0.0.1" 8444 -id 1 -u 8222 &
+python -m src.server.start_full_node "127.0.0.1" 8444 -id 1 -r 8555 &
 ssh -p 8222 localhost
 ```
 
-## Run a farmer + full node
+## Step 4b: Run a farmer + full node
+Instead of running only a full node (as in 4a), you can also run a farmer.
 Farmers are entities in the network who use their hard drive space to try to create
 blocks (like Bitcoin's miners), and earn block rewards. First, you must generate some hard drive plots, which
-can take a long time depending on the size of the plots (the k variable). Then, run the farmer + full node with
-the following script. A full node is also started, which you can ssh into to view the node UI (previous ssh command).
+can take a long time depending on the [size of the plots](https://github.com/Chia-Network/chia-blockchain/wiki/k-sizes)
+(the k variable). Then, run the farmer + full node with the following script. A full node is also started, 
+which you can ssh into to view the node UI (previous ssh command).
 ```bash
 python -m scripts.create_plots -k 20 -n 10
 sh ./scripts/run_farming.sh
 ```
 
-## Run a timelord + full node
+
+## Step 4c: Run a timelord + full node
 Timelords execute sequential verifiable delay functions (proofs of time), that get added to
 blocks to make them valid. This requires fast CPUs and a lot of memory as well as completing
 both install steps above.
@@ -252,12 +221,12 @@ When running the servers on Mac OS, allow the application to accept incoming con
 
 Ubuntu 19.xx, Amazon Linux 2, and CentOS 7.7 or newer are the easiest linux install environments currently.
 
-UPnP is enabled by default, to open the port for incoming connections. If this causes issues,
+UPnP is enabled by default, to open port 8444 for incoming connections. If this causes issues,
 you can disable it in the configuration. Some routers may require port forwarding, or enabling
 UPnP in the router configuration.
 
-Due to the nature of proof of space lookups by the harvester you should limit the number of plots
-on a physical drive to 50 or less. This limit should significantly increase before beta.
+Due to the nature of proof of space lookups by the harvester in the current alpha you should limit 
+the number of plots on a physical drive to 50 or less. This limit should significantly increase before beta.
 
 You can also run the simulation, which runs all servers and multiple full nodes, locally, at once.
 If you want to run the simulation, change the introducer ip in ./config/config.yaml so that the
@@ -271,23 +240,3 @@ ips to external peers.
 ```bash
 sh ./scripts/run_all_simulation.sh
 ```
-
-### Run tests and linting
-The first time the tests are run, BlockTools will create and persist many plots. These are used for creating
-proofs of space during testing. The next time tests are run, this won't be necessary.
-Make sure to run mongo before running the tests.
-```bash
-mongod --dbpath ./db/ &
-black src tests && flake8 src && mypy src tests
-py.test tests -s -v
-```
-
-
-### Configure VS code
-1. Install Python extension
-2. Set the environment to ./.venv/bin/python
-3. Install mypy plugin
-4. Preferences > Settings > Python > Linting > flake8 enabled
-5. Preferences > Settings > Python > Linting > mypy enabled
-7. Preferences > Settings > Formatting > Python > Provider > black
-6. Preferences > Settings > mypy > Targets: set to ./src and ./tests
diff --git a/db/.gitignore b/db/.gitignore