Build failure on Ubuntu 24.10

[NeQuissimus]

Building this encounters the following:

[...]

#########################################
Compiling LLM runtime for Linux...
#########################################

-- The C compiler identification is GNU 14.2.0
-- The CXX compiler identification is GNU 14.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (1.4s)
-- Generating done (0.0s)
-- Build files have been written to: /home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/build/build_linux_aarch64_Release
[ 25%] Building CXX object CMakeFiles/llm_demo.dir/src/llm_demo.cpp.o
[ 50%] Building CXX object CMakeFiles/multimodel_demo.dir/src/multimodel_demo.cpp.o
In file included from /home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/src/llm_demo.cpp:18:
/home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/../../runtime/Linux/librkllm_api/include/rkllm.h:52:5: error: ‘uint8_t’ does not name a type
   52 |     uint8_t      reserved[112];    /**< reserved */
      |     ^~~~~~~
/home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/../../runtime/Linux/librkllm_api/include/rkllm.h:1:1: note: ‘uint8_t’ is defined in header ‘<cstdint>’; this is probably fixable by adding ‘#include <cstdint>’
  +++ |+#include <cstdint>
    1 | #ifndef _RKLLM_H_
In file included from /home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/src/multimodel_demo.cpp:18:
/home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/../../runtime/Linux/librkllm_api/include/rkllm.h:52:5: error: ‘uint8_t’ does not name a type
   52 |     uint8_t      reserved[112];    /**< reserved */
      |     ^~~~~~~
/home/ubuntu/ezrknpu/ezrknpu/ezrknn-llm/rkllm-runtime/examples/rkllm_api_demo/../../runtime/Linux/librkllm_api/include/rkllm.h:1:1: note: ‘uint8_t’ is defined in header ‘<cstdint>’; this is probably fixable by adding ‘#include <cstdint>’
  +++ |+#include <cstdint>
    1 | #ifndef _RKLLM_H_
make[2]: *** [CMakeFiles/llm_demo.dir/build.make:76: CMakeFiles/llm_demo.dir/src/llm_demo.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/llm_demo.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/multimodel_demo.dir/build.make:76: CMakeFiles/multimodel_demo.dir/src/multimodel_demo.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:111: CMakeFiles/multimodel_demo.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[...]

The following fixes the issue but I am not sure if this needs to be conditional for Armbian or Ubuntu < 24.10

diff --git a/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h b/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h
index e565e6c..90a250b 100644
--- a/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h
+++ b/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h
@@ -5,6 +5,8 @@
 extern "C" {
 #endif

+#include <cstdint>
+
 /**
  * @typedef LLMHandle
  * @brief A handle used to manage and interact with the large language model.

[MartynaKowalska]

I confirm both the build failure and the fix proposed by @NeQuissimus, on Ubuntu Rockchip 24.04 LTS for rk3588. With the fix it now installs, however when giving a prompt to a model it's still not working as it should, possibly because I'm using Python 3.12 instead of 3.10 or 3.8, which is on me.

Edit: haven't checked entirely how everything works, but using a qwen model converted with the latest toolkit works, while using the qwens from this repo's page fails altogether. In case anyone needs it, it's the qwen.rkllm from here, it's now talking to itself so I guess it's not an instruct model and I'm abusing it, but it's the most recent one, supposedly the only one publicly available converted with the latest toolkit. I'm still using Python 3.12. The command now requires to specify max new tokens and max context length:

rkllm model_path max_new_tokens max_context_len

I'm a complete noob here so there might be mistakes, but hopefully it helps someone else. A big shoutout to the repo's maintainer Pelochus, and to NeQuissimus for the fix!

[c0zaut]

@MartynaKowalska - any RKLLM models that are converted with 1.1.* are compatible with each other and even the older 0.9.7 kernel module. I have a bunch on Huggingface that I have tested with Armbian Noble on Orange Pi 5 Plus.

Feel free to try them out: https://huggingface.co/c01zaut

[MartynaKowalska]

@c0zaut Thank you for your reply and contribution! I downloaded and tested a model of yours. This is the command I gave and its initial output:

**martyna@martyna-opi5max**:~$ rkllm Qwen2.5-3B-Instruct-rk3588-w8a8_g256-opt-1-hybrid-ratio-1.0.rkllm 4096 8192
rkllm init start
I rkllm: rkllm-runtime version: 1.1.2, rknpu driver version: 0.9.7, platform: RK3588

rkllm init success

After that, a massive text in English and Chinese was outputted, and I could give my input to the LLM. I said "Hello! How are you?", but it then started conversing with itself about artificial intelligence. Did I do something wrong? Did I pick the wrong model?

[c0zaut]

@MartynaKowalska You did not - it has to do with the chat template. Change the prompt prefix and postfix in your script to align - https://huggingface.co/c01zaut/RK3588-Prompt

If you are just using it for general chat, I made a Gradio app that automatically handles chat templates: https://github.com/c0zaut/RKLLM-Gradio

[MartynaKowalska]

@c0zaut Thank you very much, I managed to set everything up properly and I was able to chat with Qwen. Your app is fantastic, keep up the phenomenal work💜!! Can I ask you why each model has so many different files? What should one choose?

[c0zaut]

@MartynaKowalska - Those are just different conversions. RKLLM has a bunch of options for quantization parameters, so I try to do a range for folks to test out performance/accuracy for each. Thank you for enjoying! Once I have some extra time, I plan to finish implementing multi-modal and image generation support (also looking at websearch, since a couple of models support that tool calling.)

[codembly]

I got suggested to add #include under ubuntu 24.04 and it works.

[Pelochus]

I got suggested to add #include under ubuntu 24.04 and it works.

This is the solution. Sorry to reply so late, I haven't checked issues in a while. In the future I will add this so that you don't need to do this by yourself.