Coral

[ebgoldstein]

compile for edge TPU: https://coral.ai/docs/edgetpu/compiler/

And see how much faster the model is compared to the fp16 model (4s per inference)

[ebgoldstein]

Steps:

0. Use post-training quantization to produce a model with Full integer quantization — https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization)
1. Compile an edgetpu compliant model from a TFLite model via the compiler (or the colab linked in the compiler docs): https://coral.ai/docs/edgetpu/compiler/
2. Put new model on device
3. Install EdgeTPU runtime on RPi: https://coral.ai/docs/accelerator/get-started/#runtime-on-linux
4. Install PyCoral on RPi: https://coral.ai/docs/accelerator/get-started/#2-install-the-pycoral-library
5. modify script to use pycoral https://coral.ai/docs/edgetpu/tflite-python/

[ebgoldstein]

note that currently our model uses float 16 quantization (and another model uses Dynamics range quantization). Full integer reqs some code tweaks, and a representative dataset...

[ebgoldstein]

one potential idea is the do full integer quant. and just omit quantizing the output.. i..e, include:

converter.inference_input_type = tf.int8 # or tf.uint8
but omit:
converter.inference_output_type = tf.int8 # or tf.uint8

the model would run on the TPU except for the last layer, which would run on the CPU..

https://stackoverflow.com/questions/66984379/problem-in-conversion-of-pb-to-tflite-int8-for-coral-devboard-coral-ai

[ebgoldstein]

here is the part in the coral docs refering to float input/output (on an integer quantized model) running on CPU: https://coral.ai/docs/edgetpu/models-intro/#quantization

[ebgoldstein]

ok, i got the edge tpu to work. the steps were:

• used quantization aware training (and remove batchnorm from the model).
• quantizing the model (but keeping input and output tensors as floating point, so CPU deals with those ops (edgeTPU deals with the rest)
• use the online Edge TPU compilation colab notebook to fonvert the tflite to the edgetpu.tflite file
• follow the directions (above) on installing the edge tpu + pycoral stuff on the RPi
• changing the inference scripts (minimal example below).
• result was inference times 10x faster (0.4s, vs 4s)

#import packages
import os
import glob
from pycoral.utils import edgetpu
from pycoral.utils import dataset
from pycoral.adapters import common
from pycoral.adapters import classify
import time
import matplotlib.pyplot as plt
from PIL import Image

import numpy as np

path_to_model = "./models/SandCam_QAT_edgetpu.tflite"


# Initialize the TF interpreter
interpreter = edgetpu.make_interpreter(path_to_model)
interpreter.allocate_tensors()
print(interpreter.get_input_details())
print(interpreter.get_output_details())


def TFLitePred():
    crop_img = Image.open('/home/pi/Desktop/crop2.jpg')
    converted_crop = np.array(crop_img, dtype=np.float32)
    r_crop_img = converted_crop/255
    crop_img_exp = np.expand_dims(r_crop_img, axis=0)
    
    common.set_input(interpreter, crop_img_exp)
    interpreter.invoke()
    predictions = common.output_tensor(interpreter, 0)
    print(predictions)
    
t = time.time()
TFLitePred()
elapsed = time.time() - t
print(elapsed)

[ebgoldstein]

edgeTPU support via Quantize aware training has been fully added via #68

[ebgoldstein]

Current setup does not use coral b/c of supply chain limitations... future models could include it, but would likely need to detect its presence.. which was discussed in #98

one idea is to rely on lsusb or something like that to detect the device on the USB.