update rknn-toolkit2/rknn-toolkit-lite2 to 1.4.0

Signed-off-by: raul.rao <raul.rao@rock-chips.com>
airockchip · Sep 7, 2022 · d8074d7 · d8074d7
1 parent 9ad7934
commit d8074d7
Show file tree

Hide file tree

Showing 48 changed files with 652 additions and 340 deletions.
diff --git a/README.md b/README.md
@@ -3,33 +3,31 @@ RKNN-Toolkit2 is a software development kit for users to perform model conversio
 
 RKNN-Toolkit-Lite2 provides Python programming interfaces for Rockchip NPU platform  (RK3566, RK3568, RK3588, RK3588S) to help users deploy RKNN models and accelerate the implementation of AI applications.
 
-# Download
-- You can also download all packages, docker image, examples, docs and platform-tools from baidu cloud: [RK_NPU_SDK](https://eyun.baidu.com/s/3eTDMk6Y), fetch code: rknn
+Note:
 
-# Notes
-- Currently rknn-toolkit2 is not compatible with [rknn-toolkit](https://github.com/rockchip-linux/rknn-toolkit)
-- Currently only support on Ubuntu 18.04 python 3.6 / Ubuntu 20.04 python 3.8
-- If you have problem about RKNN-Toolkit2, it is recommended to create a issue or get answers from [Issues](https://github.com/rockchip-linux/rknn-toolkit2/issues).
-- Latest version:1.3.0(Release version)
+- For the deployment of the RKNN model, please refer to:
+
+    https://github.com/rockchip-linux/rknpu2
 
-# Checksums
-## MD5
-```
-69ffac3f944cf682acf563245e532405  packages/rknn_toolkit2-1.3.0_11912b58-cp36-cp36m-linux_x86_64.whl
+- For RK1808/RV1109/RV1126/RK3399Pro, please use:
 
-f37e2296e631587b929a75028dd235bf  packages/rknn_toolkit2-1.3.0_11912b58-cp38-cp38-linux_x86_64.whl
+    https://github.com/rockchip-linux/rknn-toolkit
 
-f269a9cf3fa209b2615efdd0711f3f18  docker/rknn-toolkit2-1.3.0-cp36-docker.tar.gz
+    https://github.com/rockchip-linux/rknpu
 
-db40be96541a121b4c77109d829089b6  docker/rknn-toolkit2-1.3.0-cp38-docker.tar.gz
+    https://github.com/airockchip/RK3399Pro_npu
 
-4dff57052f94a9c080774ab0e6af1d56  rknn_toolkit_lite2/packages/rknn_toolkit_lite2-1.3.0-cp37-cp37m-linux_aarch64.whl
+# Download
+- You can also download all packages, docker image, examples, docs and platform-tools from baidu cloud: [RK_NPU_SDK](https://eyun.baidu.com/s/3eTDMk6Y), fetch code: rknn
+
+# Notes
+- Currently rknn-toolkit2 is not compatible with [rknn-toolkit](https://github.com/rockchip-linux/rknn-toolkit)
+- Currently only support on Ubuntu 18.04 python 3.6 / Ubuntu 20.04 python 3.8
+- Latest version:1.4.0(Release version)
 
-1085cd7ee4bb18ef17b03d66e5428d69  rknn_toolkit_lite2/packages/rknn_toolkit_lite2-1.3.0-cp39-cp39-linux_aarch64.whl
-```
 # Feedback and Community Suport
 Two ways are followed:
-- [Issues](https://github.com/rockchip-linux/rknn-toolkit2/issues) (**Feedback recommended**)
+- [Issues](https://github.com/rockchip-linux/rknn-toolkit2/issues) 
 - QQ Group Chat: 1025468710 **welcome for developer**
 <p align="center"><img width="200" height="200"  src="https://github.com/rockchip-linux/rknn-toolkit2/blob/master/QQGroupQRCode.png"/></p>
 <p align="center">QQ Group Chat</p>
diff --git a/Rockchip_Quick_Start_RKNN_SDK_V1.3.0_CN.pdf → Rockchip_Quick_Start_RKNN_SDK_V1.4.0_CN.pdf b/Rockchip_Quick_Start_RKNN_SDK_V1.3.0_CN.pdf → Rockchip_Quick_Start_RKNN_SDK_V1.4.0_CN.pdf
diff --git a/...it2_API_Difference_With_Toolkit1-1.3.0.md → ...it2_API_Difference_With_Toolkit1-1.4.0.md b/...it2_API_Difference_With_Toolkit1-1.3.0.md → ...it2_API_Difference_With_Toolkit1-1.4.0.md
diff --git a/doc/RKNNToolKit2_OP_Support-1.3.0.md b/doc/RKNNToolKit2_OP_Support-1.3.0.md
diff --git a/doc/RKNNToolKit2_OP_Support-1.4.0.md b/doc/RKNNToolKit2_OP_Support-1.4.0.md
diff --git a/doc/Rockchip_Quick_Start_RKNN_Toolkit2_CN-1.3.0.pdf b/doc/Rockchip_Quick_Start_RKNN_Toolkit2_CN-1.3.0.pdf
diff --git a/doc/Rockchip_Quick_Start_RKNN_Toolkit2_CN-1.4.0.pdf b/doc/Rockchip_Quick_Start_RKNN_Toolkit2_CN-1.4.0.pdf
diff --git a/doc/Rockchip_Quick_Start_RKNN_Toolkit2_EN-1.3.0.pdf b/doc/Rockchip_Quick_Start_RKNN_Toolkit2_EN-1.3.0.pdf
diff --git a/doc/Rockchip_Quick_Start_RKNN_Toolkit2_EN-1.4.0.pdf b/doc/Rockchip_Quick_Start_RKNN_Toolkit2_EN-1.4.0.pdf
diff --git a/...hip_User_Guide_RKNN_Toolkit2_CN-1.3.0.pdf → ...hip_User_Guide_RKNN_Toolkit2_CN-1.4.0.pdf b/...hip_User_Guide_RKNN_Toolkit2_CN-1.3.0.pdf → ...hip_User_Guide_RKNN_Toolkit2_CN-1.4.0.pdf
diff --git a/doc/Rockchip_User_Guide_RKNN_Toolkit2_EN-1.3.0.pdf b/doc/Rockchip_User_Guide_RKNN_Toolkit2_EN-1.3.0.pdf
diff --git a/doc/Rockchip_User_Guide_RKNN_Toolkit2_EN-1.4.0.pdf b/doc/Rockchip_User_Guide_RKNN_Toolkit2_EN-1.4.0.pdf
diff --git a/doc/changelog-1.3.0.txt → doc/changelog-1.4.0.txt b/doc/changelog-1.3.0.txt → doc/changelog-1.4.0.txt
@@ -1,3 +1,25 @@
+2022-8-20
+版本: v1.4.0:
+更新内容:
+1. 升级相关依赖包到主流版本
+2. 添加更多2/3/5维度的Op支持
+3. 更新config/init_runtime等接口
+4. 更新LSTM等Op支持
+5. 添加yuv输入支持
+6. 更新QAT模型支持
+
+2022-7-2
+版本: v1.3.4b5:
+更新内容:
+1. rknn-toolkit2:
+    1) optimize_onnx接口
+        a. 在设置optimization_level=2时，关闭conv+add融合。
+        b. 保留BatchNormalize算子带的量化参数。
+    2) RK3588屏蔽NPU直接输出NHWC layout的支持， RK3566/RV1106保留该功能。
+2.  C API:
+    1) RK3588/RK3566/RV1106支持传入一个包含rknn模型的大文件路径，rknn_init接口设置包含偏移和真实rknn模型大小的rknn_init_extend结构体指针。
+
+
 2021-4-22
 版本: v1.3.0:
 更新内容:

diff --git a/doc/requirements_cp36-1.3.0.txt b/doc/requirements_cp36-1.3.0.txt
diff --git a/doc/requirements_cp36-1.4.0.txt b/doc/requirements_cp36-1.4.0.txt
@@ -0,0 +1,23 @@
+# if install failed, please change the pip source to 'https://mirror.baidu.com/pypi/simple'
+
+# base deps
+numpy==1.19.5
+protobuf==3.12.2
+flatbuffers==1.12
+
+# utils
+requests==2.27.1
+psutil==5.9.0
+ruamel.yaml==0.17.4
+scipy==1.5.4
+tqdm==4.64.0
+bfloat16==1.1
+opencv-python==4.5.5.64
+
+# base
+onnx==1.9.0
+onnxoptimizer==0.2.7
+onnxruntime==1.10.0
+torch==1.10.1
+torchvision==0.11.2
+tensorflow==2.6.2
diff --git a/doc/requirements_cp38-1.3.0.txt b/doc/requirements_cp38-1.3.0.txt
diff --git a/doc/requirements_cp38-1.4.0.txt b/doc/requirements_cp38-1.4.0.txt
@@ -0,0 +1,23 @@
+# if install failed, please change the pip source to 'https://mirror.baidu.com/pypi/simple'
+
+# base deps
+numpy==1.19.5
+protobuf==3.12.2
+flatbuffers==1.12
+
+# utils
+requests==2.27.1
+psutil==5.9.0
+ruamel.yaml==0.17.4
+scipy==1.5.4
+tqdm==4.64.0
+bfloat16==1.1
+opencv-python==4.5.5.64
+
+# base
+onnx==1.9.0
+onnxoptimizer==0.2.7
+onnxruntime==1.10.0
+torch==1.10.1
+torchvision==0.11.2
+tensorflow==2.6.2
diff --git a/examples/caffe/mobilenet_v2/dataset.txt b/examples/caffe/mobilenet_v2/dataset.txt
diff --git a/examples/caffe/mobilenet_v2/mobilenet_v2.caffemodel b/examples/caffe/mobilenet_v2/mobilenet_v2.caffemodel
diff --git a/examples/caffe/mobilenet_v2/mobilenet_v2.prototxt b/examples/caffe/mobilenet_v2/mobilenet_v2.prototxt
diff --git a/examples/caffe/mobilenet_v2/test.py b/examples/caffe/mobilenet_v2/test.py
diff --git a/examples/darknet/yolov3_416x416/dataset.txt b/examples/darknet/yolov3_416x416/dataset.txt
diff --git a/examples/darknet/yolov3_416x416/dog_bike_car_416x416.jpg b/examples/darknet/yolov3_416x416/dog_bike_car_416x416.jpg
diff --git a/examples/darknet/yolov3_416x416/test.py b/examples/darknet/yolov3_416x416/test.py
diff --git a/examples/functions/hybrid_quant/step1.py b/examples/functions/hybrid_quant/step1.py
@@ -38,11 +38,11 @@
     print('')
     print('For example:')
     print('    custom_quantize_layers:')
-    print('        Conv__344:0: float16')
+    print('        FeatureExtractor/MobilenetV2/expanded_conv/depthwise/BatchNorm/batchnorm/add_1:0: float16')
     print('        FeatureExtractor/MobilenetV2/expanded_conv/depthwise/Relu6:0: float16')
     print('Or:')
     print('    custom_quantize_layers: {')
-    print('        Conv__344:0: float16,')
+    print('        FeatureExtractor/MobilenetV2/expanded_conv/depthwise/BatchNorm/batchnorm/add_1:0: float16,')
     print('        FeatureExtractor/MobilenetV2/expanded_conv/depthwise/Relu6:0: float16,')
     print('    }')
     print('==================================================================================================')

diff --git a/examples/onnx/resnet50v2/dataset.txt b/examples/onnx/resnet50v2/dataset.txt
diff --git a/examples/onnx/resnet50v2/dog_224x224.jpg b/examples/onnx/resnet50v2/dog_224x224.jpg
diff --git a/examples/onnx/yolov5/test.py b/examples/onnx/yolov5/test.py
@@ -14,8 +14,8 @@
 
 QUANTIZE_ON = True
 
-BOX_THESH = 0.5
-NMS_THRESH = 0.6
+OBJ_THRESH = 0.25
+NMS_THRESH = 0.45
 IMG_SIZE = 640
 
 CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light",
@@ -82,13 +82,22 @@ def filter_boxes(boxes, box_confidences, box_class_probs):
         classes: ndarray, classes for boxes.
         scores: ndarray, scores for boxes.
     """
-    box_classes = np.argmax(box_class_probs, axis=-1)
-    box_class_scores = np.max(box_class_probs, axis=-1)
-    pos = np.where(box_confidences[..., 0] >= BOX_THESH)
+    boxes = boxes.reshape(-1, 4)
+    box_confidences = box_confidences.reshape(-1)
+    box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])
 
-    boxes = boxes[pos]
-    classes = box_classes[pos]
-    scores = box_class_scores[pos]
+    _box_pos = np.where(box_confidences >= OBJ_THRESH)
+    boxes = boxes[_box_pos]
+    box_confidences = box_confidences[_box_pos]
+    box_class_probs = box_class_probs[_box_pos]
+
+    class_max_score = np.max(box_class_probs, axis=-1)
+    classes = np.argmax(box_class_probs, axis=-1)
+    _class_pos = np.where(class_max_score >= OBJ_THRESH)
+
+    boxes = boxes[_class_pos]
+    classes = classes[_class_pos]
+    scores = (class_max_score* box_confidences)[_class_pos]
 
     return boxes, classes, scores
 
@@ -236,7 +245,7 @@ def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):
 
     # Load ONNX model
     print('--> Loading model')
-    ret = rknn.load_onnx(model=ONNX_MODEL, outputs=['378', '439', '500'])
+    ret = rknn.load_onnx(model=ONNX_MODEL)
     if ret != 0:
         print('Load model failed!')
         exit(ret)

diff --git a/examples/onnx/yolov5/yolov5s.onnx b/examples/onnx/yolov5/yolov5s.onnx
diff --git a/examples/pytorch/resnet18_export_onnx/test.py b/examples/pytorch/resnet18_export_onnx/test.py
@@ -81,6 +81,6 @@ def forward(self, x):
     model = RestNet18()
     model.eval()
 
-    # export onnx (rknn-toolkit2 only support opset_version=12)
+    # export onnx (rknn-toolkit2 only support to opset_version=12)
     x = torch.randn((1, 3, 224, 224))
     torch.onnx.export(model, x, './resnet18.onnx', opset_version=12, input_names=['input'], output_names=['output'])
diff --git a/examples/tensorflow/ssd_mobilenet_v1/test.py b/examples/tensorflow/ssd_mobilenet_v1/test.py
@@ -115,6 +115,7 @@ def load_box_priors():
     outputClasses = outputs[1].reshape((1, NUM_RESULTS, NUM_CLASSES))
     np.save('./tensorflow_ssd_mobilenet_v1_1.npy', outputs[0])
     candidateBox = np.zeros([2, NUM_RESULTS], dtype=int)
+    classScore = [-1000.0] * NUM_RESULTS
     vaildCnt = 0
 
     box_priors = load_box_priors()
@@ -136,6 +137,7 @@ def load_box_priors():
         if topClassScore > 0.4:
             candidateBox[0][vaildCnt] = i
             candidateBox[1][vaildCnt] = topClassScoreIndex
+            classScore[vaildCnt] = topClassScore
             vaildCnt += 1
 
     # calc position
@@ -198,7 +200,7 @@ def load_box_priors():
         xmax = max(0.0, min(1.0, predictions[0][n][3])) * INPUT_SIZE
         ymax = max(0.0, min(1.0, predictions[0][n][2])) * INPUT_SIZE
 
-        print("%d @ (%d, %d) (%d, %d) score=%f" % (topClassScoreIndex, xmin, ymin, xmax, ymax, topClassScore))
+        print("%d @ (%d, %d) (%d, %d) score=%f" % (candidateBox[1][i], xmin, ymin, xmax, ymax, classScore[i]))
         cv2.rectangle(orig_img, (int(xmin), int(ymin)), (int(xmax), int(ymax)),
                       (random.random()*255, random.random()*255, random.random()*255), 3)
 

diff --git a/examples/tflite/mobilenet_v1/test.py b/examples/tflite/mobilenet_v1/test.py
diff --git a/examples/tflite/mobilenet_v1_qat/test.py b/examples/tflite/mobilenet_v1_qat/test.py
@@ -28,7 +28,7 @@ def show_outputs(outputs):
 
     # Pre-process config
     print('--> Config model')
-    rknn.config(mean_values=[0, 0, 0], std_values=[1, 1, 1])
+    rknn.config(mean_values=[128, 128, 128], std_values=[128, 128, 128])
     print('done')
 
     # Load model

diff --git a/packages/md5sum.txt b/packages/md5sum.txt
@@ -0,0 +1,2 @@
+58c3b61e199911146c81f3c85a4c686d  rknn_toolkit2-1.4.0_22dcfef4-cp36-cp36m-linux_x86_64.whl
+c702ff263b54fa12b9044b1d334ce28a  rknn_toolkit2-1.4.0_22dcfef4-cp38-cp38-linux_x86_64.whl
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		58c3b61e199911146c81f3c85a4c686d rknn_toolkit2-1.4.0_22dcfef4-cp36-cp36m-linux_x86_64.whl
		c702ff263b54fa12b9044b1d334ce28a rknn_toolkit2-1.4.0_22dcfef4-cp38-cp38-linux_x86_64.whl