BodyPix tflite model support

See original GitHub issue

@Volcomix , your work is really impressive! Thank you so much for this work.

I saw this comment from @PINTO0309 in this issue https://github.com/Volcomix/virtual-background/issues/2#issuecomment-829730723

I don’t know if it will be useful for you, but I have converted and quantized it for various frameworks and committed it to my repository.

TFLite Float32/Float16/INT8, TFJS, TF-TRT, ONNX, CoreML, OpenVINO IR FP32/FP16, Myriad Inference Blob

https://github.com/PINTO0309/PINTO_model_zoo https://github.com/PINTO0309/PINTO_model_zoo/tree/main/109_Selfie_Segmentation

And I’m able to easily play with the different models. Thank you both for your hardwork.

I noticed @PINTO0309 also has bodypix tflite models here but they don’t seem to work with @Volcomix pipeline. Getting some logs I noticed the output image has the following:

console.log({
  inputHeight: this._tflite._getInputHeight(),
  inputWidth: this._tflite._getInputWidth(),
  inputChannelCount: this._tflite._getInputChannelCount(),
  outputHeight: this._tflite._getOutputHeight(),
  outputWidth: this._tflite._getOutputWidth(),
  outputChannelCount: this._tflite._getOutputChannelCount(),
});
// Outputs an image with 10x8 resolution, 17 channels

As you can see, the output seems wrong. Any advice on what to adjust? My intention is I want to run bodypix model in wasm hoping to gain more performance than the tfjs one. Is this something you can help with? Thank you in advance!

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:12 (5 by maintainers)

github_iconTop GitHub Comments

6reactions
PINTO0309commented, May 24, 2021

All the models were re-transformed and totally replaced with the same model structure of TensorFlow.js. https://github.com/PINTO0309/PINTO_model_zoo/tree/main/035_BodyPix

2reactions
Volcomixcommented, May 30, 2021

After loading the new tflite models and trying with the fastest one (mobilenet050, stride16, 240x320, float16 quantization), the performance seems equivalent/worse with tflite on wasm than with tfjs on webgl. Moreover the tfjs one works on a higher resolution:

image

image

After checking other projects code and the tfjs implementation of BodyPix more closely, I would have to handle padding and scaling implied by the strides which is implemented using tensorflow in all the reference projects but would require a lot of extra work in this demo. As the performance results are not that good with the tflite model of BodyPix, I don’t wish to invest more time on this specific experiment.

I’m very sorry PINTO for making you spend time on it without ending.

P.S.: There is a work in progress in TF.js to handle tflite models. Maybe is there a chance that BodyPix would work on it when it will be ready: https://github.com/tensorflow/tfjs/tree/master/tfjs-tflite

Read more comments on GitHub >

github_iconTop Results From Across the Web

[Updated] BodyPix: Real-time Person Segmentation in the ...
We are excited to announce the release of BodyPix, an open-source machine learning model which allows for person and body-part segmentation in ...
Read more >
Converting TensorflowJS's BodyPix model to TensorFlow Lite
I have come across googles new BodyPix TensorflowJS model today, and I want to get it running on Android and iOS, but using...
Read more >
tafsiri - npm
It currently supports the following models from our [models](https://github.com/tensorflow/tfjs-models) repo: - BlazeFace - BodyPix - CocoSSD - Face ...
Read more >
Segmentation using TensorFlow - Lens Studio Community
I wanted to use the library js of tensor flow BodyPix. ... layer types are supported by Lens Studio, so if your model...
Read more >
tflite | Flutter Package - Pub.dev
Supports image classification, object detection (SSD and YOLO), Pix2Pix and Deeplab and PoseNet ... loadModel( model: "assets/mobilenet_v1_1.0_224.tflite", ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found