Inquiry on Implementing Real-Time Classification with Kneron 630 and Python

Guest

Hello. I'm a developer working with the Kneron 630.

I'm encountering difficulties in development using Kneron plus Python.

What I'm trying to do is receive input from a camera sensor, pass it through NPU inference, and then retrieve the inferred value on Kneron plus.

The camera KL630 board application uses kp_firmware_hico_mipi.

I’ll share the necessary file contents. Below is the start.sh file:

#/bin/sh

mount -t configfs configfs /sys/kernel/config

mkdir -p /tmp/venc/c0/;

mkdir -p /tmp/aenc/c0/;

mkdir -p /tmp/playback/c0/;

mkdir -p /tmp/twoway/c0/;

mkdir -p /tmp/sr/c0/;

export LD_LIBRARY_PATH=$(pwd)/lib;

#./streamer_msg_sender -C 1 -s 0 -d 2 -q 10

./rtsps -c stream_server_config.ini &

#./kp_firmware_host_stream &

./kp_firmware_hico_mipi &

sleep 10;

./uvc_service &

### Notice : Codec type AAC4, sample rate 8000, bit rate 32000, raw format, stereo, mic_in ###

#./tinyaenc_mmap -d "hw:0,0" -r 8000 -C 0 -b 32000 -t 0 -S 0 -i 0 &

#sleep 2;

#./volume_ctrl -i 0 -M 0 -v 96;

#./playback_example_mmap -d "hw:0,0" -r 8000 -c 2 -C 0 &

===============================================

Below is the driver.sh file.

I'm alternating between USB companion mode and UVC mode by commenting and uncommenting these four files: usb_f_uvc.ko, usb_f_uac.ko, g_webcam.ko, 8192eu.ko.

#!/bin/sh

insmod vpl_edmc.ko

insmod vpl_dmac.ko

insmod vma_ifpe.ko

insmod vma_jdbe.ko

insmod vma_jebe.ko

insmod vma_ispe.ko

insmod vma_h4cde.ko

insmod vma_meae.ko

insmod vma_rs.ko

insmod vma_dce.ko

#insmod vpl_voc.ko

#insmod tp2912FA.ko

#insmod tp2912FB.ko

#echo 1920x1080 > /sys/class/vpl_voc/curt_mode

#insmod phy-vatics-usb.ko #Built-in

#insmod dwc2.ko #Built-in

#insmod libcomposite.ko #Built-in

insmod usb_f_uvc.ko

insmod usb_f_uac.ko

insmod g_webcam.ko

insmod 8192eu.ko

#insmod usb_f_mass_storage.ko #USB Storage mode

#insmod g_mass_storage.ko file=/tmp removable=1 #USB Storage mode

sh ./vma_h5cde.sh

sh sensor.sh

==============================================

Ultimately, I want to implement a model that uses the KL630 for real-time classification and sends the results to a host PC.

Thank you in advance for your help.

Nick Wang

Hi,

For instructions on using hico_cam, please refer to the following steps:

1. Download the Kneron PLUS - enterprise package from the Kneron website.
2. For details on compiling the hico_cam plus example, refer to the link: https://doc.kneron.com/docs/#plus_c/
3. For KL630 setup, refer to Chapter 1.9, "USB Device – PC (Kneron Plus)" in the KL630_96board_User_Manual.
4. Once the above steps are complete, connect the KL630 to the PC via USB and confirm it is recognized. You can then directly run the example kl630_demo_hico_cam_inference on the PC, which will transfer the sensor image and inference results from the KL630 to the PC.

Additionally, if you want to port a customized model to the KL630, please follow the steps below to verify that the model conversion is accurate at each stage.

1. Verify the E2E simulator results for NEF in Kneron toolchain
2. Confirm the results for KL630 companion mode in Kneron PLUS (python code)
3. Confirm the results for KL630 companion mode in Kneron PLUS (C code)
4. Move the C code to KL630

ref: https://doc.kneron.com/docs/#toolchain/manual_1_overview/