Inference Time out

Temi

Hi guys,

I have done some finetuning on a yolov5s model to allow training on my own custom dataset. The model is converted to ONNX model and further optimised using the  https://github.com/kneron/ONNX_Convertor.git repo.

i have now now generated a .nef file to run on the device using the toolchain vs 0.22. I have been able to successfully compile the model , run quantization and also pass the IP evaluation check. but however during inference time i get the following error.

 - - Error: inference failed, error = Error raised in function: generic_image_inference_receive. Error code: 103. Description: ApiReturnCode.KP_FW_INFERENCE_TIMEOUT_103

Andy Hsieh

@Temi

Hi Temi,

This is because there are three possible causes for this problem.

1. a problem with the preprocess.

2. a problem with Inference.

3. post-process.

You may need to provide more information, such as whether you are using Kneron PLUS v2.1.1, which example you are using to get the results, or which APIs you are using.

Temi

Hi @Andy Hsieh thank you very much for the response.

Example is provided by @Jeffrey Chen , please see https://drive.google.com/drive/folders/1nqErFuFoJdGDNUpX92P31hujB1VFM3OK?usp=sharing

Actually in the repo, there is yolov5s-noupsample-coco128_simplify_convert.onnx which has gone through all the conversions from torch to optimised onnx file. I converted this to bie model using the latest docker tool chain. Please see the code snippet below ,i have used to create the .nerf model, i then proceeded to running inference using the exported nef model. i am currently using the Kneron PLUS v2.1.1 and running the default examples.

Running the inference then raises the time out exception issue!.

Andy Hsieh

@Temi

Hi Temi,

If convenient, you can also provide me your platform with a default example of how the nef model runs its reasoning after you have exported the nef model.

For example,

Temi

Hi @Andy Hsieh thanks for the response, i am running on the KL520 device. i have ran the KL520DemoGenericImageInference.py script to test this out, please see output below:

inference_results.py.png

Temi

hi @Andy Hsieh I hope you are well? just following up on this enquiry?

Andy Hsieh

@Temi

Hi Temi,

Sorry for the late reply.

I saw your error message, it may be due to the fact that the image size of the input image does not match with the model you want to infer, you can check that your input image matches with the image size of the model input.

(As a reminder, the size of the input image must be larger than or equal to the model input.)

You can also refer to this link:

https://www.kneron.com/forum/discussion/comment/755/#Comment_755

Temi

hi @Andy Hsieh apologies i had accidentally attached the wrong image in my previous message, i had resolved that error a while back😀. that is not the error i am currently facing. I have made sure the size of input image matches the model., the error i face is with a time out during the inferencing operation. Please see below:

inference_error.png

Andy Hsieh

@Temi

Hi Temi.

Could you please provide us with the nef. model you ran and the onnx model before converting it to .nef model, if it's convenient for you?

Temi

models_kneron.zip

@Andy Hsieh Hi please see attached the onnx file and the .nef model as requested. this is from the yolov5 model without no up sampling operations.

Andy Hsieh

@Temi

Hi Temi,

Seeing that you are using an .onnx model, I found that the reason for this is that your onnx model has Sigmoid operators at the end. (e.g. onnx::Sigmoid_442, onnx::Sigmoid_444, onnx::Sigmoid_446)

When using KL520 platform you can remove the sigmoid operators first and then use Kneron Toolchain to convert to .nef model, then it should work fine.

Temi

@Andy Hsieh thanks for this, so to confirm i need to remove the sigmoid operators and then implement this in pytorch at the end to get my prediction?

Temi

@Andy Hsieh could i also confirm i guess Upsampling operations are also not supported by the device right?

Andy Hsieh

@Temi

Hi Temi,

You don't have to remove it from pytorch, you can use Kneron Toolchain's Model Editor function or a third party model editor.

Because KL520 doesn't support Sigmoid, it may cause your error problem.

If you want to modify the onnx model, you can refer to our documentation center.

1. opertaors supported by each platform, refer to:

https://doc.kneron.com/docs/#toolchain/appendix/operators/

2. model editing, refer to:

https://doc.kneron.com/docs/#toolchain/appendix/converters/#7-model-editor