Ci update, ISAC+CTG merged, added converpoints (#495)

* Updated CI and added riscof-plugin directory

* Added RISCV-CTG and RISCV-ISAC tools

* Added covergroups

* Updated Readme

---------

Co-authored-by: James Shi <shiqinghao.sqh@alibaba-inc.com>

.github/workflows/test.yml

@@ -26,27 +26,81 @@ jobs:
 
   ACT-sail-spike:
     runs-on: ubuntu-latest
-    container: ghcr.io/riscv-software-src/riscof/act:latest
     strategy:
       fail-fast: false
       matrix:
         isa_group: 
-          - RV32IMCZicsr_Zifencei
-          - RV64IMCZicsr_Zifencei
-
+          - RVIMAFDCZicsr_Zifencei
 
     steps:
 
       - name: Checkout source
         uses: actions/checkout@v4
 
-      - name: Config riscof
+      - name: Install Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y python3 python3-pip python3-venv
+          sudo apt-get install -y gcc git autoconf automake libtool curl make unzip
+          sudo apt-get install autoconf automake autotools-dev curl python3 python3-pip libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev ninja-build git cmake libglib2.0-dev libslirp-dev
+          pip3 install git+https://github.com/riscv/riscof.git
+      
+      - name: Build RISCV-GNU Toolchain (32 bit)
+        run: |
+          wget -c https://github.com/riscv-collab/riscv-gnu-toolchain/releases/download/2024.09.03/riscv32-elf-ubuntu-20.04-gcc-nightly-2024.09.03-nightly.tar.gz
+          tar -xzf riscv32-elf-ubuntu-20.04-gcc-nightly-2024.09.03-nightly.tar.gz
+          mv riscv riscv32  
+          echo $GITHUB_WORKSPACE/riscv32/bin >> $GITHUB_PATH
+
+      - name: Build RISCV-GNU Toolchain (64 bit)
+        run: |
+          wget -c https://github.com/riscv-collab/riscv-gnu-toolchain/releases/download/2024.09.03/riscv64-elf-ubuntu-20.04-gcc-nightly-2024.09.03-nightly.tar.gz
+          tar -xzf riscv64-elf-ubuntu-20.04-gcc-nightly-2024.09.03-nightly.tar.gz
+          mv riscv riscv64  
+          echo $GITHUB_WORKSPACE/riscv64/bin >> $GITHUB_PATH
+      
+      - name: Install Spike
         run: |
-          git config --global --add safe.directory '*'
-          riscof setup --dutname=spike
-          cp .github/isa_templates/${{ matrix.isa_group }}.yaml spike/spike_isa.yaml
+          git clone https://github.com/riscv/riscv-isa-sim.git
+          sudo apt-get install device-tree-compiler libboost-regex-dev libboost-system-dev
+          cd riscv-isa-sim
+          mkdir build
+          cd build
+          ../configure --prefix=$GITHUB_WORKSPACE/riscv64
+          make -j$(nproc)
+          sudo make install
+          echo $GITHUB_WORKSPACE/riscv64/bin >> $GITHUB_PATH      
 
+      - name: Install Sail
+        run: |
+          sudo apt-get install opam build-essential libgmp-dev z3 pkg-config zlib1g-dev
+          opam init -y --disable-sandboxing
+          opam switch create ocaml-base-compiler
+          opam install sail -y
+          eval $(opam config env)
+          git clone https://github.com/riscv/sail-riscv.git
+          cd sail-riscv
+          ARCH=RV32 make
+          ARCH=RV64 make
+          echo $PWD/c_emulator >> $GITHUB_PATH      
+
+      - name: Install riscv-isac
+        run: |
+          cd riscv-isac
+          pip3 install --editable .
+
+      - name: Install riscv-ctg
+        run: |
+          cd riscv-ctg
+          pip3 install --editable .
+      
+
+      - name: Config and run riscof for RV32
+        run: |
+          cd riscof-plugins/rv32
+          riscof run --config config.ini --suite ../../riscv-test-suite/rv32i_m/ --env ../../riscv-test-suite/env
 
-      - name: Run riscof
+      - name: Config and run riscof for RV64
         run: |
-          riscof --verbose debug run --suite . --env ./riscv-test-suite/env
+          cd riscof-plugins/rv64
+          riscof run --config config.ini --suite ../../riscv-test-suite/rv64i_m/ --env ../../riscv-test-suite/env

README.md

@@ -79,3 +79,178 @@ The files [`COPYING.BSD`](./COPYING.BSD), [`COPYING.APACHE`](./COPYING.APACHE) a
 - [riscvOVPsim](https://github.com/riscv-ovpsim/imperas-riscv-tests): Imperas freeware RISC-V reference simulator for compliance testing
 - [riscvOVPsimPlus](https://www.ovpworld.org/riscvOVPsimPlus/): Imperas enhanced freeware RISC-V reference simulator for test development and verification
 
+# Getting Started
+
+This section serves as a quick guide to set up RISCOF and perform a sample validation check between Spike (DUT in this case) and Sail-riscv (Reference Golden Model). This guide will help you set up all the required tooling for running RISCOF on your system.
+
+### Install Python
+
+### Ubuntu
+
+For Ubuntu, you can directly install Python using the Universe repository:
+
+```bash
+$ sudo apt-get install python3.6
+$ pip3 install --upgrade pip
+```
+You should now have two binaries: `python3` and `pip3` available in your `$PATH`.
+
+### Install RISCOF
+To install RISCOF, run this command in your terminal:
+
+```
+$ pip3 install git+https://github.com/riscv/riscof.git
+```
+This is the preferred method to install RISCOF, as it will always install the most recent stable release.
+
+### Install riscv-ctg
+To install riscv-ctg, run this command in your terminal:
+
+```
+$ cd riscv-ctg
+$ pip3 install --editable .
+```
+
+### Install riscv-isac
+To install riscv-isac, run this command in your terminal:
+
+```
+$ cd riscv-isac
+$ pip3 install --editable .
+```
+This is the preferred method to install riscv-isac and riscv-ctg, as updated riscv-ctg will always be maintained here.
+
+
+### Test RISCOF
+Once you have installed `RISCOF`, you can execute `riscof --help` to print the help routine:
+
+```bash
+$ riscof --help
+Usage: riscof [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --version                       Show the version and exit.
+  -v, --verbose [info|error|debug]
+                                  Set verbose level
+  --help                          Show this message and exit.
+
+Commands:
+  arch-test     Setup and maintenance for Architectural TestSuite.
+  coverage      Run the tests on DUT and reference and compare signatures
+  gendb         Generate Database for the Suite.
+  run           Run the tests on DUT and reference and compare signatures
+  setup         Initiate Setup for riscof.
+  testlist      Generate the test list for the given DUT and suite.
+  validateyaml  Validate the Input YAMLs using riscv-config.
+```
+
+
+## Install RISCV-GNU Toolchain
+
+This guide will use the 32-bit riscv-gnu toolchain to compile the architectural suite. If you already have the 32-bit gnu-toolchain available, you can skip to the next section.
+
+> **Note**: The git clone and installation will take significant time. Please be patient. If you face issues with any of the following steps, please refer to [riscv-gnu-toolchain](https://github.com/riscv-collab/riscv-gnu-toolchain) for further help in installation.
+
+### Ubuntu
+
+```bash
+$ sudo apt-get install autoconf automake autotools-dev curl python3 libmpc-dev \
+      libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool \
+      patchutils bc zlib1g-dev libexpat-dev
+$ git clone --recursive https://github.com/riscv/riscv-gnu-toolchain
+$ git clone --recursive https://github.com/riscv/riscv-opcodes.git
+$ cd riscv-gnu-toolchain
+$ ./configure --prefix=/path/to/install --with-arch=rv32gc --with-abi=ilp32d # for 32-bit toolchain
+$ [sudo] make # sudo is required depending on the path chosen in the previous setup
+```
+
+Make sure to add the path `/path/to/install` to your `$PATH` in the `.bashrc/cshrc`.
+
+With this, you should now have all the following available as command line arguments:
+
+```bash
+riscv32-unknown-elf-addr2line      riscv32-unknown-elf-elfedit
+riscv32-unknown-elf-ar             riscv32-unknown-elf-g++
+riscv32-unknown-elf-as             riscv32-unknown-elf-gcc
+riscv32-unknown-elf-c++            riscv32-unknown-elf-gcc-8.3.0
+riscv32-unknown-elf-c++filt        riscv32-unknown-elf-gcc-ar
+riscv32-unknown-elf-cpp            riscv32-unknown-elf-gcc-nm
+riscv32-unknown-elf-gcc-ranlib     riscv32-unknown-elf-gprof
+riscv32-unknown-elf-gcov           riscv32-unknown-elf-ld
+riscv32-unknown-elf-gcov-dump      riscv32-unknown-elf-ld.bfd
+riscv32-unknown-elf-gcov-tool      riscv32-unknown-elf-nm
+riscv32-unknown-elf-gdb            riscv32-unknown-elf-objcopy
+riscv32-unknown-elf-gdb-add-index  riscv32-unknown-elf-objdump
+riscv32-unknown-elf-ranlib         riscv32-unknown-elf-readelf
+riscv32-unknown-elf-run            riscv32-unknown-elf-size
+riscv32-unknown-elf-strings        riscv32-unknown-elf-strip
+```
+# Installing RISC-V Reference Models: Spike and SAIL
+This section will guide you through the installation of two important RISC-V reference models: Spike and SAIL. These models are often used as reference models in the RISCOF framework.
+
+
+### 1. Spike (riscv-isa-sim)
+
+Spike is the official RISC-V ISA simulator, also known as the RISC-V ISA simulator (riscv-isa-sim). It is commonly used as a reference model in RISCOF for compliance testing.
+
+### Installation Steps for Spike
+
+```bash
+$ sudo apt-get install device-tree-compiler
+$ git clone https://github.com/riscv-software-src/riscv-isa-sim.git
+$ cd riscv-isa-sim
+$ mkdir build
+$ cd build
+$ ../configure --prefix=/path/to/install
+$ make
+$ [sudo] make install
+```
+Note: Use sudo if the installation path requires administrative privileges.
+
+
+### 2. SAIL (SAIL C-emulator)
+
+```bash
+$ sudo apt-get install opam build-essential libgmp-dev z3 pkg-config zlib1g-dev
+$ opam init -y --disable-sandboxing
+$ opam switch create ocaml-base-compiler
+$ opam install sail -y
+$ eval $(opam config env)
+$ git clone https://github.com/riscv/sail-riscv.git
+$ cd sail-riscv
+$ ARCH=RV32 make
+$ ARCH=RV64 make
+$ ln -s sail-riscv/c_emulator/riscv_sim_RV64 /usr/bin/riscv_sim_RV64
+$ ln -s sail-riscv/c_emulator/riscv_sim_RV32 /usr/bin/riscv_sim_RV32
+```
+
+This will create a C simulator in `c_emulator/riscv_sim_RV64` and `c_emulator/riscv_sim_RV32`. You will need to add these paths to your `$PATH` or create an alias to execute them from the command line.
+
+
+## Necessary Env Files
+
+To run tests via RISCOF, you will need to provide the following items:
+
+- **config.ini**: This file is a basic configuration file following the INI syntax. This file will capture information like the name of the DUT/reference plugins, path to the plugins, path to the riscv-config based YAMLs, etc. This file is located at `riscof-plugins/rv32/config.ini` for RV32 and at `riscof-plugins/rv64/config.ini` for `RV64`
+
+- **riscv-test-suite/**: The directory contains the architectural test suites.
+
+- **riscv-config/**: The repository containing the configuration files for various RISC-V implementations. You can clone the required repository using the following commands:
+
+```
+$ git clone https://github.com/riscv/riscv-config.git
+```
+
+## Running the Tests
+Once everything is set up, you can run the tests using the following command:
+
+```
+$ riscof run --config config.ini --suite riscv-test-suite/ --env riscv-test-suite/env
+```
+
+## Running the coverage command
+You can run the coverage using the following command:
+
+```
+$ riscof coverage --config=config.ini --cgf-file covergroups/dataset.cgf --cgf-file covergroups/m/rv32im.cgf --suite /riscv-test-suite/rv32i_m/M --env /riscv-test-suite/env
+```

coverage/README.md

@@ -0,0 +1,13 @@
+## CGF: Cover Group Format
+
+- Uses a simple to use and a human readable YAML format to define cover groups and cover-points for the RISC-V ISA. 
+- Declares datasets separately which can be used across coverpoints:
+  - Operand Addresses for a single instruction
+  - Operand Value for a single instruction
+  - Abstract functions like walking1s and walking0s which get unrolled by the extraction tool
+- Covergroups include multiple datasets
+  - Each coverpoint is defined as a boolean expression which can to be evaluated by the "eval"
+    tool of python.
+  - Coverpoints to use a standard set of keywords like: rs1, rs2, rd, rs1_val, rs2_val, etc
+- Uses Anchors and Aliases to keep the size of the YAML file small
+

coverage/Zifencei/rv32e_fencei.cgf

@@ -0,0 +1,6 @@
+fencei:
+  config: 
+    - check ISA:=regex(.*E.*Zifencei.*) ;def RVTEST_E = True
+  mnemonics: 
+    fence.i: 0
+

coverage/Zifencei/rv32i_fencei.cgf

@@ -0,0 +1,6 @@
+fencei:
+  config: 
+    - check ISA:=regex(.*I.*Zifencei.*)
+  mnemonics: 
+    fence.i: 0
+

coverage/Zifencei/rv64i_fencei.cgf

@@ -0,0 +1,8 @@
+
+fencei:
+  config: 
+    - check ISA:=regex(.*I.*Zifencei.*)
+  mnemonics: 
+    fence.i: 0
+
+