DJI Mobile SDK real-time RTSP video streaming

Hello all, I am also trying to stream the raw drone data to an RTSP server.

However, I am having difficulty regarding the keyframes of the video data. As per above, the condition if (naluType == 5 && !rtspClient.isStreaming()) is never true for me. videoBuffer[0] is always 0 for me.

Does anyone know how to set the naluType or any way of getting the keyframe? Appreciate any advice! My code is the same as above and am using a Mavic 2 Pro.

Hello,

Let me explain the reason about nalutype.

H264 get video info using sps and pps (nalu type 7 and 8) and frecuently is also included in keyframes (nalu type 5). This info (sps and pps) are necessary to start stream in RTMP and RTSP so you need to provide it. This is the reason about the nalutype problem. Normally you should get keyframes (nalutype 5) in a consistent interval of X seconds (X value is configured in encoder side in my case it is iFrameInterval value of prepareVideo method).

I was looking a bit in DJI SDK code and I did an alternative way re encoding h264 buffer because you are not getting video info. This is the example code. Maybe bugged (I can’t test it). Test it and let me know if you have any problem (read header comments to get more info about it):

import android.content.Context;
import android.os.Build;

import androidx.annotation.RequiresApi;

import com.pedro.encoder.video.FormatVideoEncoder;
import com.pedro.encoder.video.GetVideoData;
import com.pedro.encoder.video.VideoEncoder;
import com.pedro.rtplibrary.view.GlInterface;
import com.pedro.rtplibrary.view.OffScreenGlThread;

import dji.sdk.camera.VideoFeeder;
import dji.sdk.codec.DJICodecManager;

/**
 * NOTE: This is not tested.
 *
 * Example code of use for DJI to generate valid h264 to send t RtspClient or RtmpClient classes.
 * We are doing the following:
 * - Get frames from DJI device in h264.
 * - Decode this frames into GlInterface class using DJICodecManager.
 * - Copy frames from GlInterface to VideoEncoder surface.
 * - VideoEncoder detect that copy automatically and re encode it to h264 that should be valid to stream.
 *
 * After that, using GetVideoData provided in start method, you should get sps and pps in the onSpsPpsVps callback where you 
 * can start your stream and send data to stream using getVideoData callback.
 *
 * Extra info:
 * GlInterface is an off screen thread that provide a surfaceTexture of OpenGl that you can render. Also,
 * allow you to copy data of that surfaceTexture to other and adding filters, take photo, etc.
 * Keep in mind that onSpsPpsVps could be called multiple times so use a conditional is recommended.
 */
@RequiresApi(api = Build.VERSION_CODES.JELLY_BEAN_MR2)
public class DJIExample implements VideoFeeder.VideoDataListener {

  private DJICodecManager codecManager;
  private VideoEncoder videoEncoder;
  private GlInterface glInterface;
  //encoder and decoder configuration
  private final int width = 640;
  private final int height = 480;
  private final int fps = 30;
  private final int bitrate = 1200 * 1000;
  private final int iFrameInterval = 2;
  private boolean init = false;


  public void start(Context context, GetVideoData getVideoData) {
    if (init) { //maybe we should init this only one time. I'm not sure
      try {
        VideoFeeder.getInstance().getPrimaryVideoFeed().addVideoDataListener(this);
        init = true;
      } catch (Exception ignored) { }
    }
    videoEncoder = new VideoEncoder(getVideoData);
    videoEncoder.prepareVideoEncoder(width, height, fps, bitrate, 0, iFrameInterval, FormatVideoEncoder.SURFACE);
    glInterface = new OffScreenGlThread(context);
    glInterface.init();
    glInterface.setEncoderSize(width, height);
    glInterface.start();
    videoEncoder.start();
    glInterface.addMediaCodecSurface(videoEncoder.getInputSurface());
    codecManager = new DJICodecManager(context, glInterface.getSurfaceTexture(), width, height);
  }

  public void stop() {
    glInterface.removeMediaCodecSurface();
    videoEncoder.stop();
    glInterface.stop();
    codecManager.cleanSurface();
    videoEncoder = null;
    glInterface = null;
    codecManager = null;
  }

  @Override
  public void onReceive(byte[] bytes, int i) {
    if (codecManager != null) {
      codecManager.sendDataToDecoder(bytes, i);
    }
  }
}

Hi @pedroSG94 I have a problem: On SDK, I use getVideoStartCodeSize() process the h264 data, get nalutype, but only has naluType 1/7/9, never get 8 or 5 ,

how can I push this data use your library by rtmp? Thank you very much!