at .bie and .nef Map dropping 42% around

MilanMilan
in KL730 related discussion

Running YOLOv11m on KL730 with toolchain v0.17.2. Seeing a large accuracy drop after BIE conversion.

Pipeline: .pt → ONNX → kneronnxopt optimize → remove output sigmoid → BIE (via km.analysis) → NEF

Accuracy at each stage:

  • Original PyTorch model: 89% mAP@0.50
  • BIE evaluation: 54% mAP@0.50 (−35%)
  • NEF evaluation: 44–46% mAP@0.50

Configuration used:

  • Input: 640×640, float32 [0.0–1.0], CHW
  • Calibration: 500 images from train set
  • datapath_bitwidth_mode = "mix balance"
  • Sigmoid removed from ONNX before BIE creation


why this is occurring?

Tagged:

Comments

  • Maria_KneronMaria_Kneron

    Hi,

    That is occurring because the toolchain version 0.17.2 is outdated. Please update it to our latest toolchain version 0.32.1 by running this command: docker pull kneron/toolchain:latest

  • MilanMilan

    sorry i think i write wrong version actually

    i tested on both ,

    kneron/toolchain:latest and kneron/toolchain:v0.25.0

    still yolo11m 'Map' on 11 class object detection became half

     

  • Maria_KneronMaria_Kneron

    Hi,

    No worries, and thank you for letting us know.

    I see that you've removed sigmoid from the ONNX model before generating BIE? It might affect the BIE and NEF results.

    Could you try evaluating the optimized ONNX (before removing the sigmoid) as well?

    If that works, could you also try converting the model with sigmoid? Either that, or please add the sigmoid back in your postprocess for BIE and NEF so they're consistent with your original model.

    Also, setting datapath_bitwidth_mode to int16, mix light, or mixbw could help as well: 3 Optimizing Quantization Modes - Document Center

  • MilanMilan


    • in removed_sigmoid.onnx i do postprocessing and then get same result
    • but same process i do in nef & bie , postprocessing then it droping map
    • should i share code or model?


    Could you try evaluating the optimized ONNX (before removing the sigmoid) as well?

    ans- no Map drop

  • Maria_KneronMaria_Kneron

    Hi,

    If your optimized ONNX's result has no mAP drop, it shouldn't drop too much in your BIE and NEF models if you converted it without removing the sigmoid.

    Yes, please share the following so we could look into it. Thank you!

    -The original model

    -The optimized model (before removing the sigmoid) and the model after removing the sigmoid

    -The data (images) used for quantization

    -The python script for converting the model and measuring mAP

  • MilanMilan

    original model .pt has map50:0.89 , we have 10 class for ppe kit detection

    https://drive.google.com/drive/folders/1remmxsMhMHR0MMuEKjJYGatGZG1Fv4R_?usp=sharing

  • Maria_KneronMaria_Kneron

    Hi,

    Unfortunately, we can't access Google Drive. Could you please directly upload a zip file here, or use another way to share the files? Sorry for the inconvenience, and thank you for your understanding!

  • MilanMilan

    The upload limit is only 50 MB, so I uploaded the model as a .pt file. I converted it to ONNX using my script, then optimized the ONNX model, removed the Sigmoid layer, and converted it to BIE and NEF formats. After these conversions, the mAP dropped by about half.


    read info.txt i have added steps there

  • Maria_KneronMaria_Kneron

    Hi,

    Thank you for providing the files! We're currently looking into it.

    Meanwhile, I remember you said that you removed_sigmoid.onnx, then did postprocessing and then got the accurate result, but after converting that model into BIE and NEF files, the results after postprocessing had a lower mAP. Could you make sure the preprocessing and postprocessing functions are consistent for all onnx, BIE, and NEF models? We also recommend you to use around 100 images when doing quantization.

Sign In or Register to comment.

© Kneron, Inc.