Inference with not image data
Hello,
I would post a model into KL720 and the input data is not image format,
the input data is 1x27x29x3 dimension of FLOAT data.
As the PLUS examples, they are almost image of input data.
As not image format of data, how can I program PLUS to apply inference?
As far as I understand, the NEF format of model handles fixed point of input data.
If the input is floating point of data, how does the pre-process handle?
Thanks!
The discussion has been closed due to inactivity. To continue with the topic, please feel free to post a new discussion.
Comments
Hi Jack,
To inference a non-image format:
If you were using Kneron PLUS v1.3.0, you could use the function kp_generic_raw_inference_bypass_pre_proc_send to bypass the pre-processing and inference your input. You would also need to be aware of the NPU Data format (please refer to the link: http://doc.kneron.com/docs/#plus_c/appendix/supported_npu_data_layout_format/) when you load the non-image input.
Reference link: Inference API - Document Center (kneron.com)
If you were using Kneron PLUS v2.0.0, you could also use the function kp_generic_data_inference_send to inference a non-image input.
Reference link: Inference API - Document Center (kneron.com)
You could also refer to a previous question here when you format your data (it's in Chinese though)
https://www.kneron.com/forum/discussion/comment/987/#Comment_987
Hi Maria,
If my model is as the below format, the input data are the set of 4 bytes of float data w/ dimension 27x29x3, liked as width 29, height 27 & channel 3
For the 1W16C8B format, how should I pre-process the input data?
Thanks!
Hi Jack,
Our pre-processing only works for 4W4C8B format, so you would need to use kp_generic_raw_inference_bypass_pre_proc_send (for Kneron PLUS v1.3.0) or kp_generic_data_inference_send (for Kneron PLUS v2.0.0) to directly send the data for inferencing without any pre-processing. Please feel free to take a look at the reference links in my reply above for further information.
You could also refer to this link as well: https://www.kneron.com/forum/discussion/comment/790/#Comment_790
Hi Maria,
For the NPU data layout format, it's hard to understand to me.
Do you have some example to explain how to arrange this kind of format?
For the input data of the model, a frame has 3 channels, and its row and column denote distance and speed, separately.
As the NPU data layout format "1W16C8B", because the data type of a element is FLOAT, for 1W, how should I arrange the input data? Or, if a element is more than one byte, how to arrange?
Thanks!
Hi Jack,
Sorry for the confusion, I'll try my best to explain this. Since you're bypassing the preprocess, you could convert your frames into an array first. Then, you would need to know your model's normalization, and write a function that converts your input data based on the normalization. You then could arrange the format by padding 0, because the NPU accepts 4 channels. This whole process would be equivalent to preprocessing.
For arranging the format, for example, if your input was an array of [0, 1, 2, 3, 4, 5, 6, 7, ...], and assuming that 0, 1, 2 is a frame and 3, 4, 5 is the next frame, you could arrange your input into something like [0, 1, 2, 0, 3, 4, 5, 0, 6, 7, 8, 0, ...]
That would format a 3-channel data to something the NPU can accept. So you could input your frame with 3 channels, then pad a 0 at the end of each frame.
If your data only had 1 channel, then the array [0, 1, 2, 3, ...] would be arranged like [0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, ...]
Please let me know if you need a more detailed explanation, thank you!
Hi Jack,
There's also an example talking about inferencing none image data without built-in hardware image pre-processing here: Data Inference - Document Center (kneron.com)
I hope this helps.
Hi Maria,
NPU data layout format, 就我認知, 它應該是以byte(8Bits)為單位來處理input data, 若原始data是3個channel, 要丟進1W16C8B input format的model, 4th channel就要塞0; 而現在要丟進model的data卻是float(32 bits)的資料型態, 要如何做前置處理來符合model的input format, 還是無法理解該如何來處理?
我想還是以實際例子來解釋遇到的問題, as the attached ktcKL720Inf.py @nonimageInf.zip.
之前有透過Kneron toolchain的simulator(ktc.kneron_inference)跑過相同的model和input data, EXEC_OPT = 1 @ktcKL720Inf.py, 其得到的結果似乎就是我想要的
但, 實際用KL720來跑model, 以相同的方式丟進去inference, 得到的結果跟simulator跑出來的就不一樣, EXEC_OPT = 2 @ktcKL720Inf.py. 推測問題點應該是丟進去model的資料不對, 似乎要重新排列...
若以attached的model & data, 這data要如何處理它再丟進model?
Thanks!
Hi Jack,
要inference之前，我們會把float (32 bits)去做浮點到定點(8-bit fixed point)的轉換來量化:
然後這裡(上面回覆的連結)有說明要如何排列資料，您的layout會需要用KP_MODEL_TENSOR_DATA_LAYOUT_1W16C8B:
我們其他部門的同事有對您的程式做一些改動，有加上面的步驟來處理data，建議您使用最新的Kneron PLUS 2.0.0來跑跑看(需要注意程式的路徑)，也可以參考Kneron PLUS 2.0.0裡面的KL720DemoGenericDataInference.py範例。
如果還有任何疑問，請再跟我們說，謝謝!
Hi Maria,
取得model input radix/scale, input data layout and input size等相關資料時, 出現下列錯誤訊息,
(AttributeError: module 'kp' has no attribute 'v1' &)
AttributeError: 'SingleModelDescriptor' object has no attribute 'input_nodes'
是否是我的環境需要更新Kneron Plus(Python kp package)?
Thanks!
Hi Maria,
從餵進model前的前置處理動作看來, 是把維度(27, 29, 3)的float轉成(27, 29, 16)的int8, 然後float值乘128
這樣的話, 我的model是否需要依照這樣的轉換修改train model的test data來重train?
因為原model依照這樣的轉換, 得到的結果都是(0.9921875, 0.0078125), 跟正確預測出來的結果不同
Thanks!
Hi Jack,
第一個問題(module 'kp' has no attribute 'v1')，是因為版本不同(v1.3.0/v2.0.0)，所以需要再改程式裡的一些地方才能成功地跑。
第二個問題，轉成float的動作是只要用bypass_preprocess，都會需要去做。
下面的圖，Data Inference的地方有幾個步驟:
一般用model去inference的時候，會做Software Pre-Processing，再直接餵進model。用KL720時，Convert to 8-bit跟Convert to NPU Data Layout的步驟是為了讓資料可以符合NPU需要的格式，所以才會去做轉換。您不需要重train，沒關係的。
可以請您提供其他的程式(像是有理想的結果的Kneron toolchain的simulator)嗎? 有完整的資料，我們會比較可以重現您的狀況，謝謝!
Hi Jack,
這是我們同事跑出來的數據，請問一下這個有符合您預期的結果嗎?
Hi Maria,
跑出來的數據(result.txt)還是跟ktc跑出來的(as attached ktc_result.txt)有些差距, 應該會有很多frames有TRI的結果, ex. no.51~62, predict[1]值大於0.5.
不過, 我用修改過的跑出來的結果跟你們跑出來的數據不同, 會跟KL720內的SCPU F/W有關嗎? 我是load Kneron PLUS內的SCPU & NPU F/W.
Thanks!
Hi Jack,
謝謝您提供的結果! 我們的同事有再調整一下您的 code ，趨近您理想的結果。
數據不同的原因主要是您的inference header要送的buffer size沒有填正確 (原本送的是沒有alined過的data size)
名字裡有"refined"的檔案是修正過的版本，希望這個能幫助您!
Hi Maria,
謝謝您的幫忙, 這就是我要的結果.
瞭解怎麼轉換float data
Thanks!!