Android平台外部编码数据实时预览播放SDK

Android平台除RTMP、RTSP直播播放外,有些场景可输出编码后(视频:H.264/H.265,音频:AAC/PCMA/PCMU)的数据,比如无人机或类似智能硬件设备,回调出来的H.264/H.265数据,编码后的H.264/H.265数据除了正常转推到RTMP、轻量级RTSP服务或GB28181外,还需要本地预览甚至对数据做二次处理(视频分析、实时水印字符叠加等),大概流程如下:

基于这样的场景诉求,我们开发了Android平台外部编码数据实时预览播放模块(以RTSP拉流,然后回调上来编码后数据,再投递到Android平台编码数据实时预览播放SDK为例)。

Android平台外部编码数据实时预览模块可实现本地低延迟的预览播放,支持软解码和特定机型硬解码,支持等比例或铺满显示,感兴趣的开发者可参看:Android平台如何实现第三方模块编码后(H.264/H.265/AAC/PCMA/PCMU)数据实时预览播放和播放

Android平台国网B接口接入SDK

电网视频监控系统是智能电网的一个重要组成部分,广泛应用于电网的建设、生产、运行、经营等方面。由于视频监控系统在不同的建设时期选用了不同的技术和不同厂家的产品,导致了标准不统一、技术路线不一致。目前国家电网公司智能电网建设,对视频监控系统提出了新的要求,因此实现统一监控、统一存储、分级控制、分域管理,使不同的视频监视系统能够互联互通,满足视频监控系统全局化、整体化的发展需求,已成为亟待解决的问题。

为此,大牛直播SDK推出的Android平台国网B接口接入SDK,可实现不具备国网B接口音视频能力的 Android终端,通过平台注册接入到现有的平台。

Android终端除支持常规的音视频数据接入外,还可以支持移动设备位置(MobilePosition)订阅和通知、语音广播,支持对接数据类型如下:

  1. 编码前数据(目前支持的有YV12/NV21/NV12/I420/RGB24/RGBA32/RGB565等数据类型),其中,Android平台前后摄像头数据,或者屏幕数据,或者Unity拿到的数据,均属编码前数据;
  2. 编码后数据(如无人机等264/HEVC数据,或者本地解析的MP4音视频数据);
  3. 拉取RTSP或RTMP流并接入至GB28181平台(比如其他IPC的RTSP流,可通过Android平台GB28181接入到国标平台)。

技术特点和优势:

  1. 全自研框架,易于扩展,自适应算法让延迟更低、采集编码传输效率更高;
  2. 所有功能以SDK接口形式提供,支持状态反馈;
  3. 可同时运行RTMP直播推送SDK、轻量级RTSP服务SDK和录像SDK;
  4. 支持外部YUV/RGB/H.264/H.265/AAC数据源接入;
  5. 所有参数均可通过SDK接口单独设置,亦可通过默认参数,傻瓜式设置。

功能支持:

  •  [视频格式]H.264/H.265(Android H.265硬编码);
  •  [音频格式]G.711 A律、AAC;
  •  [音量调节]Android平台采集端支持实时音量调节;
  •  [H.264硬编码]支持H.264特定机型硬编码;
  •  [H.265硬编码]支持H.265特定机型硬编码;
  •  [软硬编码参数配置]支持gop间隔、帧率、bit-rate设置;
  •  [软编码参数配置]支持软编码profile、软编码速度、可变码率设置;
  •  支持横屏、竖屏推流;
  •  Android平台支持后台service推送屏幕(推送屏幕需要5.0+版本);
  • 支持RTP OVER UDP和RTP OVER TCP被动模式;
  • 支持信令通道网络传输协议TCP/UDP设置;
  • 支持注册、注销,支持注册刷新及注册有效期设置;
  • 支持前端资源上报(Push_Resourse);
  • 支持请求获取资源(Request_Resource)应答;
  • 支持移动设备位置(MobilePosition)订阅和通知;
  • 支持语音广播;
  •  [实时水印]支持动态文字水印、png水印;
  •  [镜像]Android平台支持前置摄像头实时镜像功能;
  •  [实时静音]支持实时静音/取消静音;
  •  [实时快照]支持实时快照;
  •  [降噪]支持环境音、手机干扰等引起的噪音降噪处理、自动增益、VAD检测;
  •  [外部编码前视频数据对接]支持YUV数据对接;
  •  [外部编码前音频数据对接]支持PCM对接;
  •  [外部编码后视频数据对接]支持外部H.264数据对接;
  •  [外部编码后音频数据对接]外部AAC数据对接;
  •  [扩展录像功能]支持和录像SDK组合使用,录像相关功能。

对应Demo:

  •  Android工程:SmartPublisherV2、Camera2Demo

Android平台GB28181接入SDK

大牛直播SDK推出的Android平台GB28181接入SDK(SmartGBD),可实现不具备国标音视频能力的 Android终端,通过平台注册接入到现有的GB/T28181—2016服务,可用于如执法记录仪、智能安全帽、智能监控、智慧零售、智慧教育、远程办公、明厨亮灶、智慧交通、智慧工地、雪亮工程、平安乡村、生产运输、车载终端等场景,可能是业内为数不多功能齐全性能优异的商业级水准GB28181接入SDK。

Android终端除支持常规的音视频数据接入外,还可以支持移动设备位置(MobilePosition)订阅和通知、图像抓拍、语音广播和语音对讲、历史视音频下载和回放,支持对接数据类型如下:

  1. 编码前数据(目前支持的有YV12/NV21/NV12/I420/RGB24/RGBA32/RGB565等数据类型),其中,Android平台前后摄像头数据,或者屏幕数据,或者Unity拿到的数据,均属编码前数据;
  2. 编码后数据(如无人机等264/HEVC数据,或者本地解析的MP4音视频数据);
  3. 拉取RTSP或RTMP流并接入至GB28181平台(比如其他IPC的RTSP流,可通过Android平台GB28181接入到国标平台)。

技术特点和优势:

  1. 全自研框架,易于扩展,自适应算法让延迟更低、采集编码传输效率更高;
  2. 所有功能以SDK接口形式提供,支持状态反馈;
  3. 可同时运行RTMP直播推送SDK、轻量级RTSP服务SDK和录像SDK;
  4. 支持外部YUV/RGB/H.264/H.265/AAC数据源接入;
  5. 所有参数均可通过SDK接口单独设置,亦可通过默认参数,傻瓜式设置。

功能支持:

  •  [视频格式]H.264/H.265(Android H.265硬编码);
  •  [音频格式]G.711 A律、AAC;
  •  [音量调节]Android平台采集端支持实时音量调节;
  •  [H.264硬编码]支持H.264特定机型硬编码;
  •  [H.265硬编码]支持H.265特定机型硬编码;
  •  [软硬编码参数配置]支持gop间隔、帧率、bit-rate设置;
  •  [软编码参数配置]支持软编码profile、软编码速度、可变码率设置;
  •  支持横屏、竖屏推流;
  •  Android平台支持后台service推送屏幕(推送屏幕需要5.0+版本);
  • 支持纯视频、音视频PS打包传输;
  • 支持RTP OVER UDP和RTP OVER TCP被动模式(TCP媒体流传输客户端);
  • 支持信令通道网络传输协议TCP/UDP设置;
  • 支持注册、注销,支持注册刷新及注册有效期设置;
  • 支持设备目录查询应答;
  • 支持心跳机制,支持心跳间隔、心跳检测次数设置;
  • 支持移动设备位置(MobilePosition)订阅和通知;
  •  适用国家标准:GB/T 28181—2016;
  • 支持语音广播;
  • 支持语音对讲;
  • 支持图像抓拍;
  • 支持历史视音频文件检索;
  • 支持历史视音频文件下载;
  • 支持历史视音频文件回放;
  • 支持云台控制和预置位查询;
  •  [实时水印]支持动态文字水印、png水印;
  •  [镜像]Android平台支持前置摄像头实时镜像功能;
  •  [实时静音]支持实时静音/取消静音;
  •  [实时快照]支持实时快照;
  •  [降噪]支持环境音、手机干扰等引起的噪音降噪处理、自动增益、VAD检测;
  •  [外部编码前视频数据对接]支持YUV数据对接;
  •  [外部编码前音频数据对接]支持PCM对接;
  •  [外部编码后视频数据对接]支持外部H.264数据对接;
  •  [外部编码后音频数据对接]外部AAC数据对接;
  •  [扩展录像功能]支持和录像SDK组合使用,录像相关功能。

对应Demo:

  •  Android工程:SmartPublisherV2、Camera2Demo;

技术对接文档:

相关博客:

Unity环境下RTMP推流|轻量级RTSP服务+RTMP|RTSP播放低延迟解决方案

除了Windows/Linux/Android/iOS Native SDK,大牛直播SDK发布了Unity环境下的RTMP推流|轻量级RTSP服务(Windows平台+Linux平台+Android平台)和RTMP|RTSP直播播放(Windows、Linux、Android和iOS平台全覆盖)低延迟的解决方案。

目前,大牛直播SDK的Unity3D环境下,已覆盖以下SDK:

  •  Windows平台RTMP直播推送SDK(采集Unity窗体、摄像头或屏幕);
  •  Windows平台轻量级RTSP服务SDK(采集Unity窗体、摄像头或屏幕);
  •  Windows平台RTMP|RTSP直播播放SDK;
  •  Linux平台RTMP直播推送SDK(采集Unity窗体、Unity声音);
  •  Linux平台RTMP|RTSP直播播放SDK;
  •  Android平台RTMP直播推送SDK(采集Unity窗体、摄像头、麦克风或Unity声音);
  •  Android平台轻量级RTSP服务SDK(采集Unity窗体、摄像头、麦克风或Unity声音);
  •  Android平台RTMP|RTSP直播播放SDK;
  •  iOS平台RTMP|RTSP直播播放SDK。

1. Unity环境下RTMP推流、轻量级RTSP服务模块

Unity环境下,不管是camera还是窗体数据也好,主要是高效率的拿到原始数据,采集端可用的数据格式是RGB的,拿到之后,通过高效率的数据传递,发给封装后的原生SDK,完成数据编码和RTMP推送。

需要注意的地方有几点:

1. 数据采集投递,确保高效率;

2. 屏幕分辨率发生变化,可实时适配;

3. Unity和原生SDK之间通信,比如event回调等;

4. 屏幕数据如有水平或垂直翻转,需要有一定的矫正。

2. Unity环境下RTMP|RTSP播放器

Unity环境下RTMP或RTSP直播播放我们前几年就有发布,并已应用在好多传统行业领域,比如教育或工业仿真或一些低延迟的控制场景。

相关实现逻辑如下:

1. Native RTSP或RTSP直播播放SDK回调RGB/YUV420/NV12等其中的一种未压缩的图像格式;

2. Unity3D创建相应的RGB/YUV420等Shader;

3.Unity3D从各个平台获取图像数据来填充纹理即可;

需要注意的有几点:

1. 多实例支持:播放端和推送不一样,比如智慧城市,播放端有多路场景,所以多实例支持是必备功能,多实例环境下,需要能有好的区分event状态回调等;

2. 尽可能高效率的数据传递,确保资源占有最小化;

3. 视频分辨率变化后,能自动适配;

4. Unity和原生SDK之间通信,比如event回调等;

5. 长时间运行稳定性。

相关SDK文档及视频

大牛直播SDK Unity3D接口调用SDK说明

Unity3d RTSP/RTMP直播播放端SDK视频演示1

Unity3d RTSP/RTMP直播播放端SDK视频演示2

相关博客

Windows平台Unity3d下如何同时播放多路RTSP或RTMP流

如何在Unity3d平台下低延迟播放RTMP或RTSP流

Windows平台实现Unity下窗体|摄像头|屏幕采集推送

Android平台实现Unity3D下RTMP推送

Unity3D平台实现全景实时RTMP|RTSP流渲染

Unity3D下Linux平台播放RTSP或RTMP流

Android平台实现VR头显Unity下音视频数据RTMP推送

Unity实现Camera和Audio数据的低延迟RTMP推送技术探讨

Android平台Unity下如何通过WebCamTexture采集摄像头数据并推送至RTMP服务器或轻量级RTSP服务

Windows平台实现Unity下窗体|摄像头|屏幕采集推送

技术背景

随着Unity3D的应用范围越来越广,越来越多的行业开始基于Unity3D开发产品,如传统行业中虚拟仿真教育、航空工业、室内设计、城市规划、工业仿真等领域。

基于此,好多开发者苦于在Unity环境下,没有低延迟的推拉流解决方案,前几年,我们在Unity环境下推出了跨平台低延迟的RTMP|RTSP直播播放器,很好的解决了好多对延迟要求苛刻的使用场景。

随着时间的推移,越来越多的开发者联系我们,希望我们能推出Unity环境下的RTMP推送模块,获取到unity的实时数据,更低延迟更高效率的实现数据传输推送,基于此,我们发布了Unity环境下的RTMP推送模块。

本文以Windows平台为例,数据源分别为Unity的窗口、摄像头或整个屏幕,编码传输模块,还是调用大牛直播SDK(官方)的原生接口,简单界面先睹为快:

技术实现

1. 基础初始化


        private bool InitSDK()
        {
            if (!is_pusher_sdk_init_)
            {
                // 设置日志路径(请确保目录存在)
                String log_path = "D:\\pulisherlog";
                NTSmartLog.NT_SL_SetPath(log_path);

                UInt32 isInited = NTSmartPublisherSDK.NT_PB_Init(0, IntPtr.Zero);

                if (isInited != 0)
                {
                    Debug.Log("调用NT_PB_Init失败..");
                    return false;
                }

                is_pusher_sdk_init_ = true;
            }

            return true;
        }

2. 调用Open()接口,获取推送实例

       public bool OpenPublisherHandle(uint video_option, uint audio_option)
        {
            if (publisher_handle_ != IntPtr.Zero)
            {
                return true;
            }

            publisher_handle_count_ = 0;

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_Open(out publisher_handle_,
                video_option, audio_option, 0, IntPtr.Zero))
            {
                return false;
            }

            if (publisher_handle_ != IntPtr.Zero)
            {
                pb_event_call_back_ = new NT_PB_SDKEventCallBack(PbEventCallBack);

                NTSmartPublisherSDK.NT_PB_SetEventCallBack(publisher_handle_, IntPtr.Zero, pb_event_call_back_);

                return true;
            }
            else
            {
                return false;
            }
        }

3. 初始化参数配置

这里需要注意下,如果要采集unity窗口,需要设置图层模式,先填充一层RGBA黑色背景,然后再添加一层,用于叠加外部数据。

       private void SetCommonOptionToPublisherSDK()
        {
            if (!IsPublisherHandleAvailable())
            {
                Debug.Log("SetCommonOptionToPublisherSDK, publisher handle with null..");
                return;
            }

            NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0,
                            0, IntPtr.Zero);

            if (video_option == NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_LAYER)
            {
                // 第0层填充RGBA矩形, 目的是保证帧率, 颜色就填充全黑
                int red = 0;
                int green = 0;
                int blue = 0;
                int alpha = 255;

                NT_PB_RGBARectangleLayerConfig rgba_layer_c0 = new NT_PB_RGBARectangleLayerConfig();

                rgba_layer_c0.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE;
                rgba_layer_c0.base_.index_ = 0;
                rgba_layer_c0.base_.enable_ = 1;
                rgba_layer_c0.base_.region_.x_ = 0;
                rgba_layer_c0.base_.region_.y_ = 0;
                rgba_layer_c0.base_.region_.width_ = video_width_;
                rgba_layer_c0.base_.region_.height_ = video_height_;

                rgba_layer_c0.base_.offset_ = Marshal.OffsetOf(rgba_layer_c0.GetType(), "base_").ToInt32();
                rgba_layer_c0.base_.cb_size_ = (uint)Marshal.SizeOf(rgba_layer_c0);

                rgba_layer_c0.red_ = System.BitConverter.GetBytes(red)[0];
                rgba_layer_c0.green_ = System.BitConverter.GetBytes(green)[0];
                rgba_layer_c0.blue_ = System.BitConverter.GetBytes(blue)[0];
                rgba_layer_c0.alpha_ = System.BitConverter.GetBytes(alpha)[0];

                IntPtr rgba_conf = Marshal.AllocHGlobal(Marshal.SizeOf(rgba_layer_c0));

                Marshal.StructureToPtr(rgba_layer_c0, rgba_conf, true);

                UInt32 rgba_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                rgba_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE,
                                0, IntPtr.Zero);

                Marshal.FreeHGlobal(rgba_conf);

                NT_PB_ExternalVideoFrameLayerConfig external_layer_c1 = new NT_PB_ExternalVideoFrameLayerConfig();

                external_layer_c1.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_EXTERNAL_VIDEO_FRAME;
                external_layer_c1.base_.index_ = 1;
                external_layer_c1.base_.enable_ = 1;
                external_layer_c1.base_.region_.x_ = 0;
                external_layer_c1.base_.region_.y_ = 0;
                external_layer_c1.base_.region_.width_ = video_width_;
                external_layer_c1.base_.region_.height_ = video_height_;

                external_layer_c1.base_.offset_ = Marshal.OffsetOf(external_layer_c1.GetType(), "base_").ToInt32();
                external_layer_c1.base_.cb_size_ = (uint)Marshal.SizeOf(external_layer_c1);

                IntPtr external_layer_conf = Marshal.AllocHGlobal(Marshal.SizeOf(external_layer_c1));

                Marshal.StructureToPtr(external_layer_c1, external_layer_conf, true);

                UInt32 external_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                external_layer_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_EXTERNAL_VIDEO_FRAME,
                                0, IntPtr.Zero);

                Marshal.FreeHGlobal(external_layer_conf);

            }
            else if (video_option == NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_CAMERA)
            {
                CameraInfo camera = cameras_[cur_sel_camera_index_];
                NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];

                SetVideoCaptureDeviceBaseParameter(camera.id_.ToString(), (UInt32)cap.width_, (UInt32)cap.height_);
            }

            SetFrameRate((UInt32)CalBitRate(edit_key_frame_, video_width_, video_height_));

            Int32 type = 0;   //软编码
            Int32 encoder_id = 1;
            UInt32 codec_id = (UInt32)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264;
            Int32 param1 = 0;

            SetVideoEncoder(type, encoder_id, codec_id, param1);

            SetVideoQualityV2(CalVideoQuality(video_width_, video_height_, is_h264_encoder));

            SetVideoMaxBitRate((CalMaxKBitRate(edit_key_frame_, video_width_, video_height_, false)));

            SetVideoKeyFrameInterval((edit_key_frame_));

            if (is_h264_encoder)
            {
                SetVideoEncoderProfile(1);
            }

            SetVideoEncoderSpeed(CalVideoEncoderSpeed(video_width_, video_height_, is_h264_encoder));

            // 音频相关设置

            SetAuidoInputDeviceId(0);
            SetPublisherAudioCodecType(1);
            SetPublisherMute(is_mute);
            SetEchoCancellation(0, 0);
            SetNoiseSuppression(0);
            SetAGC(0);
            SetVAD(0);
            SetInputAudioVolume(Convert.ToSingle(edit_audio_input_volume_));
        }

4. 数据采集

摄像头和屏幕的数据采集,还是调用原生的SDK接口,本文不再赘述,如果需要采集Unity窗体的数据,可以用参考以下代码:

        if ( texture_ == null || video_width_ != Screen.width || video_height_ != Screen.height)
        {
            Debug.Log("OnPostRender screen changed++ scr_width: " + Screen.width + " scr_height: " + Screen.height);

            if (screen_image_ != IntPtr.Zero)
            {
                Marshal.FreeHGlobal(screen_image_);
                screen_image_ = IntPtr.Zero;
            }

            if (texture_ !=  null)
            {
                UnityEngine.Object.Destroy(texture_);
                texture_ = null;
            }

            video_width_ = Screen.width;
            video_height_ = Screen.height;

            texture_ = new Texture2D(video_width_, video_height_, TextureFormat.BGRA32, false);

            screen_image_ = Marshal.AllocHGlobal(video_width_ * 4 * video_height_);

            Debug.Log("OnPostRender screen changed--");

            return;
        }

        texture_.ReadPixels(new Rect(0, 0, video_width_, video_height_), 0, 0, false);
        texture_.Apply();

从 texture里面,通过调用 GetRawTextureData(),获取到原始数据。

5. 数据对接

数据对接,通过调用以下接口:

       public void OnPostRGBAData(IntPtr data, int length, int stride, int width, int height)
        {
            NT_PB_Image pb_image = new NT_PB_Image();
            
            pb_image.format_ = (int)NTSmartPublisherDefine.NT_PB_E_IMAGE_FORMAT.NT_PB_E_IMAGE_FORMAT_RGB32;
            pb_image.width_ = width;
            pb_image.height_ = height;
            pb_image.timestamp_ = 0;
            pb_image.cb_size_ = (UInt32)Marshal.SizeOf(pb_image);   

            pb_image.stride_ = new Int32[16];
            pb_image.stride_[0] = stride;

            pb_image.plane_size_ = new Int32[16];
            pb_image.plane_size_[0] = pb_image.stride_[0] * pb_image.height_;         

            pb_image.plane_ = new IntPtr[16];
            pb_image.plane_[0] = data;

            IntPtr image_data = Marshal.AllocHGlobal(Marshal.SizeOf(pb_image));

            Marshal.StructureToPtr(pb_image, image_data, true);

            NTSmartPublisherSDK.NT_PB_PostLayerImage(publisher_handle_, 0,
                            1, image_data, 0, IntPtr.Zero);

            Marshal.FreeHGlobal(image_data);
        }

6. 本地数据预览

        public bool StartPreview()
        {
            if(CheckPublisherHandleAvailable() == false)
                return false;

            video_preview_image_callback_ = new NT_PB_SDKVideoPreviewImageCallBack(SDKVideoPreviewImageCallBack);

            NTSmartPublisherSDK.NT_PB_SetVideoPreviewImageCallBack(publisher_handle_, (int)NTSmartPublisherDefine.NT_PB_E_IMAGE_FORMAT.NT_PB_E_IMAGE_FORMAT_RGB32, IntPtr.Zero, video_preview_image_callback_);

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPreview(publisher_handle_, 0, IntPtr.Zero))
            {
                if (0 == publisher_handle_count_)
                {
                    NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                    publisher_handle_ = IntPtr.Zero;
                }

                return false;
            }

            publisher_handle_count_++;

            is_previewing_ = true;

            return true;
        }

设置preview后,处理preview的数据回调

        //预览数据回调
        public void SDKVideoPreviewImageCallBack(IntPtr handle, IntPtr user_data, IntPtr image)
        {
            NT_PB_Image pb_image = (NT_PB_Image)Marshal.PtrToStructure(image, typeof(NT_PB_Image));

            NT_VideoFrame pVideoFrame = new NT_VideoFrame();

            pVideoFrame.width_ = pb_image.width_;
            pVideoFrame.height_ = pb_image.height_;

            pVideoFrame.stride_ = pb_image.stride_[0];

            Int32 argb_size = pb_image.stride_[0] * pb_image.height_;

            pVideoFrame.plane_data_ = new byte[argb_size];
            
            if (argb_size > 0)
            {
                Marshal.Copy(pb_image.plane_[0],pVideoFrame.plane_data_,0, argb_size);
            }

            {
                cur_image_ = pVideoFrame;
            }
        }      

7. 相关event回调处理

        private void PbEventCallBack(IntPtr handle, IntPtr user_data, 
            UInt32 event_id,
            Int64 param1,
            Int64 param2,
            UInt64 param3,
            UInt64 param4,
            [MarshalAs(UnmanagedType.LPStr)] String param5,
            [MarshalAs(UnmanagedType.LPStr)] String param6,
            IntPtr param7)
        {
            String event_log = "";

            switch (event_id)
            {
                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_CONNECTING:
                    event_log = "连接中";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_CONNECTION_FAILED:
                    event_log = "连接失败";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_CONNECTED:
                    event_log = "已连接";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_DISCONNECTED:
                    event_log = "断开连接";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                default:
                    break;
            }

            if(OnLogEventMsg != null) OnLogEventMsg.Invoke(event_id, event_log);
        }

8. 开始推送、停止推送

       public bool StartPublisher(String url)
        {
            if (CheckPublisherHandleAvailable() == false) return false;

            if (publisher_handle_ == IntPtr.Zero)
            {
                return false;
            }
            if (!String.IsNullOrEmpty(url))
            {
                NTSmartPublisherSDK.NT_PB_SetURL(publisher_handle_, url, IntPtr.Zero);
            }

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPublisher(publisher_handle_, IntPtr.Zero))
            {
                if (0 == publisher_handle_count_)
                {
                    NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                    publisher_handle_ = IntPtr.Zero;
                }

                is_publishing_ = false;

                return false;
            }

            publisher_handle_count_++;

            is_publishing_ = true;

            return true;
        }

        public void StopPublisher()
        {
            if (is_publishing_ == false) return;

            publisher_handle_count_--;
            NTSmartPublisherSDK.NT_PB_StopPublisher(publisher_handle_);

            if (0 == publisher_handle_count_)
            {
                NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                publisher_handle_ = IntPtr.Zero;
            }

            is_publishing_ = false;
        }

9. 关闭实例

        public void Close()
        {
            if (0 == publisher_handle_count_)
            {
                NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                publisher_handle_ = IntPtr.Zero;
            }
        }

总结

经测试,Unity环境下,通过高效率的数据采集、编码和推送,配合SmartPlayer播放器播放,整体延迟可控制在毫秒级,可适用于大多数Unity环境下对延迟和稳定性要求苛刻的场景。

Unity3D RTMP直播推流SDK

好多开发者苦于很难在unity3d下实现RTMP直播推送,本次以大牛直播SDK的Windows平台RTMP推送模块(以推摄像头为例,如需推屏幕数据,设置相关参数即可)为例,介绍下unity3d的RTMP推送集成。

简单来说,Unity3D环境下,可以直接调用C#的接口封装,针对此,我们先做了一层封装 (nt_publisher_wrapper.cs),核心代码如下:

初始化和基础参数设置:

       private bool InitSDK()
        {
            if (!is_pusher_sdk_init_)
            {
                // 设置日志路径(请确保目录存在)
                String log_path = "D:\\pulisherlog";
                NTSmartLog.NT_SL_SetPath(log_path);

                UInt32 isInited = NTSmartPublisherSDK.NT_PB_Init(0, IntPtr.Zero);

                if (isInited != 0)
                {
                    Debug.Log("调用NT_PB_Init失败..");
                    return false;
                }

                is_pusher_sdk_init_ = true;
            }

            return true;
        }

        public bool OpenPublisherHandle(uint video_option, uint audio_option)
        {
            if (publisher_handle_ != IntPtr.Zero)
            {
                return true;
            }

            publisher_handle_count_ = 0;

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_Open(out publisher_handle_,
                video_option, audio_option, 0, IntPtr.Zero))
            {
                return false;
            }

            if (publisher_handle_ != IntPtr.Zero)
            {
                pb_event_call_back_ = new NT_PB_SDKEventCallBack(PbEventCallBack);

                NTSmartPublisherSDK.NT_PB_SetEventCallBack(publisher_handle_, IntPtr.Zero, pb_event_call_back_);

                return true;
            }
            else
            {
                return false;
            }
        }

        private void SetCommonOptionToPublisherSDK()
        {
            if (!IsPublisherHandleAvailable())
            {
                Debug.Log("SetCommonOptionToPublisherSDK, publisher handle with null..");
                return;
            }

            CameraInfo camera = cameras_[cur_sel_camera_index_];
            NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];

            SetVideoCaptureDeviceBaseParameter(camera.id_.ToString(), (UInt32)cap.width_, (UInt32)cap.height_);

            SetFrameRate((UInt32)CalBitRate(edit_key_frame_, cap.width_, cap.height_));

            SetVideoEncoderType(is_h264_encoder ? 1 : 2);

            SetVideoQualityV2(CalVideoQuality(cap.width_, cap.height_, is_h264_encoder));

            SetVideoMaxBitRate((CalMaxKBitRate(edit_key_frame_, cap.width_, cap.height_, false)));

            SetVideoKeyFrameInterval((edit_key_frame_));

            if (is_h264_encoder)
            {
                SetVideoEncoderProfile(1);

            }

            SetVideoEncoderSpeed(CalVideoEncoderSpeed(cap.width_, cap.height_, is_h264_encoder));

            // 音频相关设置

            SetAuidoInputDeviceId(0);

            SetPublisherAudioCodecType(1);

            SetPublisherMute(is_mute);

            SetInputAudioVolume(Convert.ToSingle(edit_audio_input_volume_));
        }

预览、停止预览:

       public bool StartPreview()
        {
            if(CheckPublisherHandleAvailable() == false)
                return false;

            video_preview_image_callback_ = new NT_PB_SDKVideoPreviewImageCallBack(SDKVideoPreviewImageCallBack);

            NTSmartPublisherSDK.NT_PB_SetVideoPreviewImageCallBack(publisher_handle_, (int)NTSmartPublisherDefine.NT_PB_E_IMAGE_FORMAT.NT_PB_E_IMAGE_FORMAT_RGB32, IntPtr.Zero, video_preview_image_callback_);

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPreview(publisher_handle_, 0, IntPtr.Zero))
            {
                if (0 == publisher_handle_count_)
                {
                    NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                    publisher_handle_ = IntPtr.Zero;
                }

                return false;
            }

            publisher_handle_count_++;

            is_previewing_ = true;

            return true;
        }

        public void StopPreview()
        {
            if (is_previewing_ == false) return;

            is_previewing_ = false;

            publisher_handle_count_--;
            NTSmartPublisherSDK.NT_PB_StopPreview(publisher_handle_);

            if (0 == publisher_handle_count_)
            {
                NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                publisher_handle_ = IntPtr.Zero;
            }
        }

开始推送、停止推送:

        public bool StartPublisher(String url)
        {
            if (CheckPublisherHandleAvailable() == false) return false;

            if (publisher_handle_ == IntPtr.Zero)
            {
                return false;
            }
            if (!String.IsNullOrEmpty(url))
            {
                NTSmartPublisherSDK.NT_PB_SetURL(publisher_handle_, url, IntPtr.Zero);
            }

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPublisher(publisher_handle_, IntPtr.Zero))
            {
                if (0 == publisher_handle_count_)
                {
                    NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                    publisher_handle_ = IntPtr.Zero;
                }

                is_publishing_ = false;

                return false;
            }

            publisher_handle_count_++;

            is_publishing_ = true;

            return true;
        }

        public void StopPublisher()
        {
            if (is_publishing_ == false) return;

            publisher_handle_count_--;
            NTSmartPublisherSDK.NT_PB_StopPublisher(publisher_handle_);

            if (0 == publisher_handle_count_)
            {
                NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                publisher_handle_ = IntPtr.Zero;
            }

            is_publishing_ = false;
        }

相关event事件回调:

        private void PbEventCallBack(IntPtr handle, IntPtr user_data, 
            UInt32 event_id,
            Int64 param1,
            Int64 param2,
            UInt64 param3,
            UInt64 param4,
            [MarshalAs(UnmanagedType.LPStr)] String param5,
            [MarshalAs(UnmanagedType.LPStr)] String param6,
            IntPtr param7)
        {
            String event_log = "";

            switch (event_id)
            {
                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_CONNECTING:
                    event_log = "连接中";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_CONNECTION_FAILED:
                    event_log = "连接失败";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_CONNECTED:
                    event_log = "已连接";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                case (uint)NTSmartPublisherDefine.NT_PB_E_EVENT_ID.NT_PB_E_EVENT_ID_DISCONNECTED:
                    event_log = "断开连接";
                    if (!String.IsNullOrEmpty(param5))
                    {
                        event_log = event_log + " url:" + param5;
                    }
                    break;

                default:
                    break;
            }

            if(OnLogEventMsg != null) OnLogEventMsg.Invoke(event_id, event_log);
        }

SmartPublishWinMono.cs 调用上述封装的代码即可,本地预览的话,拿到回调的RGB数据,在unity3d上层刷下即可,如下图:

经测试,unity3d下,RTMP推送,配合RTMP播放端,依然可以实现毫秒级延迟的推拉流体验。

Windows平台RTMP直播推送集成简要说明

好多开发者在集成大牛直播SDK (官方)的Windows平台RTMP推送模块时吓一跳,怎么这么多接口?本文做个简单的拆分:

初始化

初始化之前,如需设置日志路径,调用NTSmartLog.NT_SL_SetPath(log_path); 设置日志存放路径。

设置过后,调用NT_PB_Init()接口,完成SDK初始化动作,注意,哪怕多实例推送,Init()接口也仅需调一次,同理,UnInit()接口也是。

然后,代码会判断系统是不是支持WR模式采集窗口,WR这种只有Win10高版本的才支持,如果不需要用到采集窗口,这个接口可忽略。

        /*
		 * 检查是否支持WR方式采集窗口
         * is_supported: 输出参数, 输出1表示支持, 0表示不支持
         * 注意:这个需要win10较高版本才支持
         * 成功返回 NT_ERC_OK
		 */
        [DllImport(@"SmartPublisherSDK.dll")]
		public static extern UInt32 NT_PB_IsWRCaptureWindowSupported(ref Int32 is_supported);

再往下,是遍历系统支持的硬解、摄像头等信息,比如LoadHWVideoEncoderInfos():

        private void LoadHWVideoEncoderInfos()
        {            
	        hw_video_encoder_infos_.Clear();

	        Int32 count = 0;
            UInt32 ret = NTSmartPublisherSDK.NT_PB_GetHWVideoEncoderInfoCount(ref count);
	        
            if (NTBaseCodeDefine.NT_ERC_OK == ret && count > 0)
	        {
                IntPtr ptr_hw_video_encoder_infos = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(NT_PB_HWVideoEncoderInfo)) * count);
                
                Int32 out_count = 0;

                ret = NTSmartPublisherSDK.NT_PB_GetHWVideoEncoderInfos(ptr_hw_video_encoder_infos, count, ref out_count);

                if (ret != NTBaseCodeDefine.NT_ERC_OK || out_count < 1)
                {
                    hw_video_encoder_infos_.Clear();
                }
                else
                {
                    for (int i = 0; i < out_count; i++)
                    {
                        NT_PB_HWVideoEncoderInfo hw_video_encoder_info = (NT_PB_HWVideoEncoderInfo)Marshal.PtrToStructure(ptr_hw_video_encoder_infos + i * Marshal.SizeOf(typeof(NT_PB_HWVideoEncoderInfo)), typeof(NT_PB_HWVideoEncoderInfo));
                        
                        hw_video_encoder_infos_.Add(hw_video_encoder_info);
                    }
                }

               Marshal.FreeHGlobal(ptr_hw_video_encoder_infos);
	        }
        }

            if (hw_video_encoder_infos_.Count > 0)
            {
                EnableHWVideoEncoderControls(true);
                FillVideoEncodersControl((uint)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264);
            }

紧接着是Audio和camera相关:

            int auido_devices = 0;

            if (NTBaseCodeDefine.NT_ERC_OK == NTSmartPublisherSDK.NT_PB_GetAuidoInputDeviceNumber(ref auido_devices))
            {
                if (auido_devices > 0)
                {
                    btn_check_auido_mic_input_.Enabled = true;

                    for (int i = 0; i < auido_devices; ++i)
                    {
                        byte[] deviceNameBuffer = new byte[512];

                        string name = "";

                        if (NTBaseCodeDefine.NT_ERC_OK == NTSmartPublisherSDK.NT_PB_GetAuidoInputDeviceName((uint)i, deviceNameBuffer, 512))
                        {
                            int count = 0;
                            for (int j = 0; j < deviceNameBuffer.Length; ++j )
                            {
                                if ( deviceNameBuffer[j] != 0 )
                                {
                                    count++;
                                }
                                else
                                {
                                    break;
                                }
                            }

                            if ( count > 0 )
                            {
                                name = Encoding.UTF8.GetString(deviceNameBuffer, 0, count);
                            }                    
                        }

                        var audio_name = "";

                        if (name.Length == 0)
                        {
                            audio_name = "音频采集设备-";
                        }
                        else
                        {
                            audio_name = name + "-";
                        }

                        audio_name = audio_name + (i + 1);

                        combox_auido_input_devices_.Items.Add(name);
                    }
                    combox_auido_input_devices_.SelectedIndex = 0;
                }
            }

            publisher_handle_ = new IntPtr();

            region_choose_tool_handle_ = new IntPtr();

            win_form_wnd_ = GetForegroundWindow();

            cameras_ = new List<CameraInfo>();

            btn_check_video_bitrate_.CheckState = CheckState.Checked;

            if (IsCanCaptureSpeaker())
            {
                btn_check_auido_speaker_input_.Enabled = true;
            }
            else
            {
                btn_check_auido_speaker_input_.Enabled = false;
            }

            if (btn_check_auido_mic_input_.Checked
                  || btn_check_auido_speaker_input_.Checked)
            {
                btn_check_speex_encoder_.Enabled = true;
                edit_speex_quality_.Enabled = true;
                btn_check_noise_suppression_.Enabled = true;
                btn_check_agc_.Enabled = true;
                btn_check_vad_.Enabled = true;
            }

            if ( btn_check_auido_mic_input_.Checked
                && btn_check_auido_speaker_input_.Checked)
            {
                btn_check_echo_cancel_.Enabled = false;
                edit_echo_delay_.Enabled = false;
            }

            edit_audio_input_volume_.Text = "1.0";
            edit_audio_speaker_input_volume_.Text = "1.0";

            FillCameraInfo();
            InitCameraControl();

OpenPublisherHandle()

OpenPublisherHandle()主要是确认选择数据源类型,然后获取推送句柄,等待做下一步的操作。

选择video option和 audio option

            // 视频
            UInt32 video_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_NO_VIDEO;

            if (btn_desktop_camera_switch.Checked
                || btn_camera_overlay_to_desktop.Checked
                || btn_desktop_overlay_to_camera.Checked)
            {
                // 使用叠加模式
                video_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_LAYER;
            }
            else if (btn_check_window_input_.Checked)
            {
                video_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_WINDOW;
            }
            else if (btn_check_desktop_input_.Checked && btn_check_scale_desktop_.Checked)
            {
                // 使用叠加模式来实现缩放
                video_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_LAYER;
            }
            else if (btn_check_desktop_input_.Checked && !btn_check_scale_desktop_.Checked)
            {
                //屏幕模式
                video_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_SCREEN;
            }
            else if (btn_check_camera_input_.Checked)
            {
                //摄像头模式
                video_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_CAMERA;
            }

            // 音频
            UInt32 audio_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_AUDIO_OPTION.NT_PB_E_AUDIO_OPTION_NO_AUDIO;

            if (btn_check_auido_mic_input_.Checked
                && btn_check_auido_speaker_input_.Checked)
            {
                audio_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_AUDIO_OPTION.NT_PB_E_AUDIO_OPTION_CAPTURE_MIC_SPEAKER_MIXER;
            }
            else if (btn_check_auido_mic_input_.Checked)
            {
                //麦克风模式
                audio_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_AUDIO_OPTION.NT_PB_E_AUDIO_OPTION_CAPTURE_MIC;
            }
            else if (btn_check_auido_speaker_input_.Checked)
            {
                //扬声器模式
                audio_option = (UInt32)NTSmartPublisherDefine.NT_PB_E_AUDIO_OPTION.NT_PB_E_AUDIO_OPTION_CAPTURE_SPEAKER;
            }

调用Open接口获取publisher handle,然设置event callback

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_Open(out publisher_handle_,
                video_option, audio_option, 0, IntPtr.Zero))
            {
                MessageBox.Show("Call open failed!");
                return false;
            }

            if (publisher_handle_ != IntPtr.Zero)
            {
                pb_event_call_back_ = new NT_PB_SDKEventCallBack(PbSDKEventCallBack);

                NTSmartPublisherSDK.NT_PB_SetEventCallBack(publisher_handle_, win_form_wnd_, pb_event_call_back_);
                return true;
            }
            else
            {
                return false;
            }

event callback相关ID

        /*事件ID*/
        public enum NT_PB_E_EVENT_ID : uint
        {
            NT_PB_E_EVENT_ID_BASE = NTBaseCodeDefine.NT_EVENT_ID_SMART_PUBLISHER_SDK,

	        NT_PB_E_EVENT_ID_CONNECTING			= NT_PB_E_EVENT_ID_BASE | 0x2,	/*连接中, param5表示推送URL */
	        NT_PB_E_EVENT_ID_CONNECTION_FAILED	= NT_PB_E_EVENT_ID_BASE | 0x3,	/*连接失败, param5表示推送URL*/
	        NT_PB_E_EVENT_ID_CONNECTED			= NT_PB_E_EVENT_ID_BASE | 0x4,	/*已连接, param5表示推送URL*/
	        NT_PB_E_EVENT_ID_DISCONNECTED		= NT_PB_E_EVENT_ID_BASE | 0x5,	/*断开连接, param5表示推送URL*/
	
	        NT_PB_E_EVENT_ID_RECORDER_START_NEW_FILE    = NT_PB_E_EVENT_ID_BASE | 0x7,	/*录像写入新文件, param5表示录像文件名*/
	        NT_PB_E_EVENT_ID_ONE_RECORDER_FILE_FINISHED = NT_PB_E_EVENT_ID_BASE | 0x8,	/*一个录像文件完成, param5表示录像文件名*/

            NT_PB_E_EVENT_ID_CAPTURE_WINDOW_INVALID = NT_PB_E_EVENT_ID_BASE | 0xd, /*捕获窗口时,如果窗口句柄无效则通知用户, param1为窗口句柄*/

            NT_PB_E_EVENT_ID_RTSP_URL = NT_PB_E_EVENT_ID_BASE | 0xe, /* 通知rtsp url, param1表示rtsp server handle, param5 表示rtsp url */
            NT_PB_E_EVENT_ID_PUSH_RTSP_SERVER_RESPONSE_STATUS_CODE = NT_PB_E_EVENT_ID_BASE | 0xf,  /* 推送rtsp时服务端相应的status code上报,目前只上报401, param1表示status code,  param5表示推送URL */
            NT_PB_E_EVENT_ID_PUSH_RTSP_SERVER_NOT_SUPPORT = NT_PB_E_EVENT_ID_BASE | 0x10,  /* 推送rtsp时服务器不支持rtsp推送,  param5表示推送URL */
        }

SetCommonOptionToPublisherSDK()

SetCommonOptionToPublisherSDK()主要是指定具体采集的音视频数据类型,比如摄像头数据、屏幕数据、摄像头和屏幕叠加后的数据(以层级模式实现)、窗口等,这块比较复杂,好在作为SDK调用者,你只要搞清楚你需要采集的类型,直接移植就可以了。

           // 视频相关设置
            if (btn_desktop_camera_switch.Checked
                || btn_camera_overlay_to_desktop.Checked
                || btn_desktop_overlay_to_camera.Checked
                || btn_check_desktop_input_.Checked
                || btn_check_window_input_.Checked
                || btn_check_camera_input_.Checked)
            {
                if (btn_desktop_camera_switch.Checked)
                {
                    //摄像头和屏幕相互切换
                    int left = Int32.Parse(edit_clip_left_.Text);
                    int top = Int32.Parse(edit_clip_top_.Text);
                    int w = Int32.Parse(edit_clip_width_.Text);
                    int h = Int32.Parse(edit_clip_height_.Text);

                    // 有一个是0, 就使用全屏
                    if (w == 0 || h == 0)
                    {
                        left = 0;
                        top = 0;
                        w = screenArea_.Width;
                        h = screenArea_.Height;
                    }
                    else
                    {
                        // 保证4字节对齐
                        w = NT_ByteAlign(w, 4);
                        h = NT_ByteAlign(h, 4);
                    }


                    NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0,
                                    0, IntPtr.Zero);

                    // 第0层填充RGBA矩形, 目的是保证帧率, 颜色就填充全黑
                    int red = 0;
                    int green = 0;
                    int blue = 0;
                    int alpha = 255;

                    NT_PB_RGBARectangleLayerConfig rgba_layer_c0 = new NT_PB_RGBARectangleLayerConfig();

                    rgba_layer_c0.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE;
                    rgba_layer_c0.base_.index_ = 0;
                    rgba_layer_c0.base_.enable_ = 1;
                    rgba_layer_c0.base_.region_.x_ = left;
                    rgba_layer_c0.base_.region_.y_ = top;
                    rgba_layer_c0.base_.region_.width_ = w;
                    rgba_layer_c0.base_.region_.height_ = h;

                    rgba_layer_c0.base_.offset_ = Marshal.OffsetOf(rgba_layer_c0.GetType(), "base_").ToInt32();
                    rgba_layer_c0.base_.cb_size_ = (uint)Marshal.SizeOf(rgba_layer_c0);

                    rgba_layer_c0.red_ = System.BitConverter.GetBytes(red)[0];
                    rgba_layer_c0.green_ = System.BitConverter.GetBytes(green)[0];
                    rgba_layer_c0.blue_ = System.BitConverter.GetBytes(blue)[0];
                    rgba_layer_c0.alpha_ = System.BitConverter.GetBytes(alpha)[0];

                    IntPtr rgba_conf = Marshal.AllocHGlobal(Marshal.SizeOf(rgba_layer_c0));

                    Marshal.StructureToPtr(rgba_layer_c0, rgba_conf, true);

                    UInt32 rgba_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                    rgba_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE,
                                    0, IntPtr.Zero);

                    Console.WriteLine("[摄像头和屏幕相互切换] NT_PB_AddLayerConfig, rgba: " + rgba_r + Environment.NewLine);

                    Marshal.FreeHGlobal(rgba_conf);

                    //第一层:摄像头
                    NT_PB_CameraLayerConfigV2 camera_layer_c1 = new NT_PB_CameraLayerConfigV2();

                    CameraInfo camera = cameras_[cur_sel_camera_index_];
                    NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];

                    camera_layer_c1.device_unique_id_utf8_ = camera.id_;

                    camera_layer_c1.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_CAMERA;
                    camera_layer_c1.base_.index_ = 1;
                    camera_layer_index_ = camera_layer_c1.base_.index_;
                    camera_layer_c1.base_.enable_ = 1;
                    camera_layer_c1.base_.region_.x_ = left;
                    camera_layer_c1.base_.region_.y_ = top;
                    camera_layer_c1.base_.region_.width_ = w;
                    camera_layer_c1.base_.region_.height_ = h;

                    if (btn_check_flip_horizontal_camera_.Checked)
                    {
                        camera_layer_c1.is_flip_horizontal_ = 1;
                    }
                    else
                    {
                        camera_layer_c1.is_flip_horizontal_ = 0;
                    }

                    if (btn_check_flip_vertical_camera_.Checked)
                    {
                        camera_layer_c1.is_flip_vertical_ = 1;
                    }
                    else
                    {
                        camera_layer_c1.is_flip_vertical_ = 0;
                    }

                    // 这种叠加模式下不要旋转,否则变形厉害, 要么就定好一个角度,调整宽高,但不要动态旋转
                    camera_layer_c1.rotate_degress_ = 0;

                    camera_layer_c1.base_.offset_ = Marshal.OffsetOf(camera_layer_c1.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                    camera_layer_c1.base_.cb_size_ = (uint)Marshal.SizeOf(camera_layer_c1);

                    IntPtr cmr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(camera_layer_c1));

                    Marshal.StructureToPtr(camera_layer_c1, cmr_conf, true);

                    UInt32 c_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                cmr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_CAMERA,
                                0, IntPtr.Zero);

                    Marshal.FreeHGlobal(cmr_conf);

                    //第二层
                    NT_PB_ScreenLayerConfig screen_layer_c2 = new NT_PB_ScreenLayerConfig();

                    screen_layer_c2.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN;
                    screen_layer_c2.base_.index_ = 2;
                    screen_layer_index_ = screen_layer_c2.base_.index_;
                    screen_layer_c2.base_.enable_ = 1;
                    screen_layer_c2.base_.region_.x_ = left;
                    screen_layer_c2.base_.region_.y_ = top;
                    screen_layer_c2.base_.region_.width_ = w;
                    screen_layer_c2.base_.region_.height_ = h;

                    screen_layer_c2.base_.offset_ = Marshal.OffsetOf(screen_layer_c2.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                    screen_layer_c2.base_.cb_size_ = (uint)Marshal.SizeOf(screen_layer_c2);

                    screen_layer_c2.clip_region_.x_ = left;
                    screen_layer_c2.clip_region_.y_ = top;
                    screen_layer_c2.clip_region_.width_ = w;
                    screen_layer_c2.clip_region_.height_ = h;

                    screen_layer_c2.reserve_ = IntPtr.Zero;

                    IntPtr scr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(screen_layer_c2));

                    Marshal.StructureToPtr(screen_layer_c2, scr_conf, true);

                    UInt32 s_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                scr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN,
                                0, IntPtr.Zero);

                    Marshal.FreeHGlobal(scr_conf);

                    // 第三层填充RGBA矩形, 目的是保证帧率, 颜色就填充全黑
                    red = Int32.Parse(edit_rgba_rect_layer_red_.Text);
                    red = ClipIntValue(red, 0, 255);

                    green = Int32.Parse(edit_rgba_rect_layer_green_.Text);
                    green = ClipIntValue(green, 0, 255);

                    blue = Int32.Parse(edit_rgba_rect_layer_blue_.Text);
                    blue = ClipIntValue(blue, 0, 255);

                    alpha = Int32.Parse(edit_rgba_rect_layer_alpha_.Text);
                    alpha = ClipIntValue(alpha, 0, 255);

                    NT_PB_RGBARectangleLayerConfig rgba_layer_c3 = new NT_PB_RGBARectangleLayerConfig();

                    rgba_layer_c3.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE;
                    rgba_layer_c3.base_.index_ = 3;
                    rgba_layer_index_ = rgba_layer_c3.base_.index_;
                    rgba_layer_c3.base_.enable_ = 1;
                    rgba_layer_c3.base_.region_.x_ = left;     //这个只是demo演示,实际以需要遮盖位置为准
                    rgba_layer_c3.base_.region_.y_ = top;
                    rgba_layer_c3.base_.region_.width_ = 160;
                    rgba_layer_c3.base_.region_.height_ = 160;

                    rgba_layer_c3.base_.offset_ = Marshal.OffsetOf(rgba_layer_c3.GetType(), "base_").ToInt32();
                    rgba_layer_c3.base_.cb_size_ = (uint)Marshal.SizeOf(rgba_layer_c3);

                    rgba_layer_c3.red_ = System.BitConverter.GetBytes(red)[0];
                    rgba_layer_c3.green_ = System.BitConverter.GetBytes(green)[0];
                    rgba_layer_c3.blue_ = System.BitConverter.GetBytes(blue)[0];
                    rgba_layer_c3.alpha_ = System.BitConverter.GetBytes(alpha)[0];

                    IntPtr rgba_conf_3 = Marshal.AllocHGlobal(Marshal.SizeOf(rgba_layer_c3));

                    Marshal.StructureToPtr(rgba_layer_c3, rgba_conf_3, true);

                    UInt32 rgba_r_3 = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                    rgba_conf_3, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE,
                                    0, IntPtr.Zero);

                    Console.WriteLine("NT_PB_AddLayerConfig, rgba: " + rgba_r_3 + Environment.NewLine);

                    Marshal.FreeHGlobal(rgba_conf_3);
                    
                    // 第四层填充png水印(注意,实时开启、关闭水印,是根据图层的index来的,如此demo,png水印的index为4)
                    // 如果有图片,增加图片层
                    if (!String.IsNullOrEmpty(image_layer_file_name_utf8_)
                        && image_layer_width_ > 0
                        && image_layer_height_ > 0)
                    {
                        NT_PB_ImageLayerConfig image_layer_c4 = new NT_PB_ImageLayerConfig();

                        image_layer_c4.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_IMAGE;
                        image_layer_c4.base_.index_ = 4;
                        image_layer_index_ = image_layer_c4.base_.index_;
                        image_layer_c4.base_.enable_ = 1;
                        image_layer_c4.base_.region_.x_ = image_layer_left_;
                        image_layer_c4.base_.region_.y_ = image_layer_top_;
                        image_layer_c4.base_.region_.width_ = image_layer_width_;
                        image_layer_c4.base_.region_.height_ = image_layer_height_;

                        image_layer_c4.base_.offset_ = Marshal.OffsetOf(image_layer_c4.GetType(), "base_").ToInt32();
                        image_layer_c4.base_.cb_size_ = (uint)Marshal.SizeOf(image_layer_c4);

                        byte[] buffer1 = Encoding.Default.GetBytes(image_layer_file_name_utf8_);
                        byte[] buffer2 = Encoding.Convert(Encoding.UTF8, Encoding.Default, buffer1, 0, buffer1.Length);
                        string strBuffer = Encoding.Default.GetString(buffer2, 0, buffer2.Length);

                        image_layer_c4.file_name_utf8_ = strBuffer;

                        image_layer_c4.is_setting_background_ = 0;
                        image_layer_c4.bk_red_ = 0;
                        image_layer_c4.bk_green_ = 0;
                        image_layer_c4.bk_blue_ = 0;
                        image_layer_c4.reserve_ = 0;

                        IntPtr image_conf = Marshal.AllocHGlobal(Marshal.SizeOf(image_layer_c4));

                        Marshal.StructureToPtr(image_layer_c4, image_conf, true);

                        UInt32 image_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                        image_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_IMAGE,
                                        0, IntPtr.Zero);

                        Console.WriteLine("NT_PB_AddLayerConfig, image: " + image_r + Environment.NewLine);

                        Marshal.FreeHGlobal(image_conf);

                        NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, UInt32.Parse(edit_frame_rate_.Text));
                    }
                }
                else if (btn_camera_overlay_to_desktop.Checked)
                {
                    //摄像头overlay到桌面
                    int left = Int32.Parse(edit_clip_left_.Text);
                    int top = Int32.Parse(edit_clip_top_.Text);
                    int w = Int32.Parse(edit_clip_width_.Text);
                    int h = Int32.Parse(edit_clip_height_.Text);

			        // 有一个是0, 就使用全屏
			        if ( w == 0 || h == 0 )
			        {
				        left = 0;
				        top = 0;
                        w = screenArea_.Width;
                        h = screenArea_.Height;
			        }
			        else
			        {
				        // 保证4字节对齐
                        w = NT_ByteAlign(w, 4);
                        h = NT_ByteAlign(h, 4);
			        }

                    //第一层:屏幕
                    NT_PB_ScreenLayerConfig screen_layer_c0 = new NT_PB_ScreenLayerConfig();

                    screen_layer_c0.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN;
                    screen_layer_c0.base_.index_ = 0;
                    screen_layer_index_ = screen_layer_c0.base_.index_;
                    screen_layer_c0.base_.enable_ = 1;
                    screen_layer_c0.base_.region_.x_ = left;
                    screen_layer_c0.base_.region_.y_ = top;
                    screen_layer_c0.base_.region_.width_ = w;
                    screen_layer_c0.base_.region_.height_ = h;

                    screen_layer_c0.base_.offset_ = Marshal.OffsetOf(screen_layer_c0.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                    screen_layer_c0.base_.cb_size_ = (uint)Marshal.SizeOf(screen_layer_c0);

                    screen_layer_c0.clip_region_.x_ = left;
                    screen_layer_c0.clip_region_.y_ = top;
                    screen_layer_c0.clip_region_.width_ = w;
                    screen_layer_c0.clip_region_.height_ = h;

                    screen_layer_c0.reserve_ = IntPtr.Zero;

                    NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0,
                            0, IntPtr.Zero);
                    
                    IntPtr scr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(screen_layer_c0));

                    Marshal.StructureToPtr(screen_layer_c0, scr_conf, true);

                    UInt32 s_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                scr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN,
                                0, IntPtr.Zero);

                    Marshal.FreeHGlobal(scr_conf);

                    //第二层:摄像头
                    if (-1 != cur_sel_camera_index_)
                    {
                        int c_l = Int32.Parse(edit_camera_overlay_left_.Text);
                        int c_t = Int32.Parse(edit_camera_overlay_top_.Text);

                        int c_w = Int32.Parse(edit_camera_overlay_width_.Text);
                        int c_h = Int32.Parse(edit_camera_overlay_height_.Text);

                        if (c_w == 0)
                        {
                            c_w = w / 2;
                        }

                        if (c_h == 0)
                        {
                            c_h = h / 2;
                        }

                        ctos_camera_layer_c1_ = new NT_PB_CameraLayerConfigV2();

                        CameraInfo camera = cameras_[cur_sel_camera_index_];
                        NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];

                        ctos_camera_layer_c1_.device_unique_id_utf8_ = camera.id_;

                        ctos_camera_layer_c1_.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_CAMERA;
                        ctos_camera_layer_c1_.base_.index_ = 1;
                        camera_layer_index_ = ctos_camera_layer_c1_.base_.index_;
                        ctos_camera_layer_c1_.base_.enable_ = 1;
                        ctos_camera_layer_c1_.base_.region_.x_ = c_l;
                        ctos_camera_layer_c1_.base_.region_.y_ = c_t;
                        ctos_camera_layer_c1_.base_.region_.width_ = c_w;
                        ctos_camera_layer_c1_.base_.region_.height_ = c_h;

                        if (btn_check_flip_horizontal_camera_.Checked)
                        {
                            ctos_camera_layer_c1_.is_flip_horizontal_ = 1;
                        }
                        else
                        {
                            ctos_camera_layer_c1_.is_flip_horizontal_ = 0;
                        }

                        if (btn_check_flip_vertical_camera_.Checked)
                        {
                            ctos_camera_layer_c1_.is_flip_vertical_ = 1;
                        }
                        else
                        {
                            ctos_camera_layer_c1_.is_flip_vertical_ = 0;
                        }

                        ctos_camera_layer_c1_.rotate_degress_ = GetCameraRotateDegress();

                        ctos_camera_layer_c1_.base_.offset_ = Marshal.OffsetOf(ctos_camera_layer_c1_.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                        ctos_camera_layer_c1_.base_.cb_size_ = (uint)Marshal.SizeOf(ctos_camera_layer_c1_);

                        IntPtr cmr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(ctos_camera_layer_c1_));

                        Marshal.StructureToPtr(ctos_camera_layer_c1_, cmr_conf, true);

                        UInt32 c_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                    cmr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_CAMERA,
                                    0, IntPtr.Zero);

                        Marshal.FreeHGlobal(cmr_conf);
                    }
                   
                    NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, UInt32.Parse(edit_frame_rate_.Text));
                }
                else if (btn_desktop_overlay_to_camera.Checked)
                {
                    //桌面overlay到摄像头

                    //第一层:摄像头
                    if (-1 != cur_sel_camera_index_
                           && -1 != cur_sel_camera_resolutions_index_
                           && -1 != cur_sel_camera_frame_rate_index_)
                    {
                        NT_PB_CameraLayerConfigV2 camera_layer_c0 = new NT_PB_CameraLayerConfigV2();

                        CameraInfo camera = cameras_[cur_sel_camera_index_];
                        NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];

                        camera_layer_c0.device_unique_id_utf8_ = camera.id_;

                        camera_layer_c0.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_CAMERA;
                        camera_layer_c0.base_.index_ = 0;
                        camera_layer_index_ = camera_layer_c0.base_.index_;
                        camera_layer_c0.base_.enable_ = 1;
                        camera_layer_c0.base_.region_.x_ = 0;
                        camera_layer_c0.base_.region_.y_ = 0;
                        camera_layer_c0.base_.region_.width_ = cap.width_;
                        camera_layer_c0.base_.region_.height_ = cap.height_;

                        if (btn_check_flip_horizontal_camera_.Checked)
                        {
                            camera_layer_c0.is_flip_horizontal_ = 1;
                        }
                        else
                        {
                            camera_layer_c0.is_flip_horizontal_ = 0;
                        }

                        if (btn_check_flip_vertical_camera_.Checked)
                        {
                            camera_layer_c0.is_flip_vertical_ = 1;
                        }
                        else
                        {
                            camera_layer_c0.is_flip_vertical_ = 0;
                        }

                        // 这种叠加模式下不要旋转,否则变形厉害, 要么就定好一个角度,调整宽高,但不要动态旋转
                        camera_layer_c0.rotate_degress_ = 0;

                        camera_layer_c0.base_.offset_ = Marshal.OffsetOf(camera_layer_c0.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                        camera_layer_c0.base_.cb_size_ = (uint)Marshal.SizeOf(camera_layer_c0);

                        NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0,
                                        0, IntPtr.Zero);

                        IntPtr cmr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(camera_layer_c0));

                        Marshal.StructureToPtr(camera_layer_c0, cmr_conf, true);

                        UInt32 r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                    cmr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_CAMERA,
                                    0, IntPtr.Zero);

                        Marshal.FreeHGlobal(cmr_conf);

                        //第二层:屏幕
                        NT_PB_ScreenLayerConfig screen_layer_c1 = new NT_PB_ScreenLayerConfig();

                        screen_layer_c1.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN;
                        screen_layer_c1.base_.index_ = 1;
                        screen_layer_index_ = screen_layer_c1.base_.index_;
                        screen_layer_c1.base_.enable_ = 1;
                        screen_layer_c1.base_.region_.x_ = 0;
                        screen_layer_c1.base_.region_.y_ = 0;
                        screen_layer_c1.base_.region_.width_ = cap.width_ / 2;
                        screen_layer_c1.base_.region_.height_ = cap.height_ / 2;

                        screen_layer_c1.base_.offset_ = Marshal.OffsetOf(screen_layer_c1.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                        screen_layer_c1.base_.cb_size_ = (uint)Marshal.SizeOf(screen_layer_c1);

                        screen_layer_c1.clip_region_.x_ = 0;
                        screen_layer_c1.clip_region_.y_ = 0;
                        screen_layer_c1.clip_region_.width_ = cap.width_ / 2;
                        screen_layer_c1.clip_region_.height_ = cap.height_ / 2;

                        screen_layer_c1.reserve_ = IntPtr.Zero;

                        IntPtr scr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(screen_layer_c1));

                        Marshal.StructureToPtr(screen_layer_c1, scr_conf, true);

                        UInt32 s_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                    scr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN,
                                    0, IntPtr.Zero);

                        Marshal.FreeHGlobal(scr_conf);
                    }

                    NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, (uint)(cur_sel_camera_frame_rate_index_ + 1));
                }
                else if (btn_check_desktop_input_.Checked && btn_check_scale_desktop_.Checked)
                {
                    int left = 0;
                    int top = 0;
                    int w = 0;
                    int h = 0;
                    int scale_w = 0;
                    int scale_h = 0;

                    GetScreenScaleConfigInfo(ref left, ref top, ref w, ref h, ref scale_w, ref scale_h);

                    NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0,
                    0, IntPtr.Zero);

                    // 第0层填充RGBA矩形, 目的是保证帧率, 颜色就填充全黑
                    int red = 0;
                    int green = 0;
                    int blue = 0;   
                    int alpha = 255;

                    NT_PB_RGBARectangleLayerConfig rgba_layer_c0 = new NT_PB_RGBARectangleLayerConfig();

                    rgba_layer_c0.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE;
                    rgba_layer_c0.base_.index_ = 0;
                    rgba_layer_index_ = rgba_layer_c0.base_.index_;
                    rgba_layer_c0.base_.enable_ = 1;
                    rgba_layer_c0.base_.region_.x_ = 0;
                    rgba_layer_c0.base_.region_.y_ = 0;
                    rgba_layer_c0.base_.region_.width_ = scale_w;
                    rgba_layer_c0.base_.region_.height_ = scale_h;

                    rgba_layer_c0.base_.offset_ = Marshal.OffsetOf(rgba_layer_c0.GetType(), "base_").ToInt32();
                    rgba_layer_c0.base_.cb_size_ = (uint)Marshal.SizeOf(rgba_layer_c0);

                    rgba_layer_c0.red_   = 0;
                    rgba_layer_c0.green_ = 0;
                    rgba_layer_c0.blue_  = 0;
                    rgba_layer_c0.alpha_ = 255;

                    IntPtr rgba_conf_0 = Marshal.AllocHGlobal(Marshal.SizeOf(rgba_layer_c0));

                    Marshal.StructureToPtr(rgba_layer_c0, rgba_conf_0, true);

                    UInt32 rgba_r_0 = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                    rgba_conf_0, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE,
                                    0, IntPtr.Zero);

                    Console.WriteLine("NT_PB_AddLayerConfig, rgba: " + rgba_r_0 + Environment.NewLine);

                    Marshal.FreeHGlobal(rgba_conf_0);

                    //第1层
                    NT_PB_ScreenLayerConfigV2 screen_layer_c1 = new NT_PB_ScreenLayerConfigV2();

                    screen_layer_c1.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN;
                    screen_layer_c1.base_.index_ = 1;
                    screen_layer_index_ = screen_layer_c1.base_.index_;
                    screen_layer_c1.base_.enable_ = checkbox_black_screen_.Checked?0:1;
                    screen_layer_c1.base_.region_.x_ = left;
                    screen_layer_c1.base_.region_.y_ = top;
                    screen_layer_c1.base_.region_.width_ = scale_w;
                    screen_layer_c1.base_.region_.height_ = scale_h;

                    screen_layer_c1.base_.offset_ = Marshal.OffsetOf(screen_layer_c1.GetType(), "base_").ToInt32(); //offsetof(T, base_);
                    screen_layer_c1.base_.cb_size_ = (uint)Marshal.SizeOf(screen_layer_c1);

                    screen_layer_c1.clip_region_.x_ = left;
                    screen_layer_c1.clip_region_.y_ = top;
                    screen_layer_c1.clip_region_.width_ = w;
                    screen_layer_c1.clip_region_.height_ = h;

                    screen_layer_c1.reserve1_ = IntPtr.Zero;
                    screen_layer_c1.reserve2_ = 0;
                    screen_layer_c1.scale_filter_mode_ = 3;
                    
                    IntPtr scr_conf = Marshal.AllocHGlobal(Marshal.SizeOf(screen_layer_c1));

                    Marshal.StructureToPtr(screen_layer_c1, scr_conf, true);

                    UInt32 s_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,
                                scr_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_SCREEN,
                                0, IntPtr.Zero);

                    Marshal.FreeHGlobal(scr_conf);

                    NTSmartPublisherSDK.NT_PB_SetSleepMode(publisher_handle_, checkbox_black_screen_.Checked ? 1 : 0, 0);

                    NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, UInt32.Parse(edit_frame_rate_.Text));
                }
                else if (btn_check_desktop_input_.Checked && !btn_check_scale_desktop_.Checked)
                {
                    //桌面
                    NTSmartPublisherSDK.NT_PB_SetScreenClip(publisher_handle_,
                    UInt32.Parse(edit_clip_left_.Text),
                    UInt32.Parse(edit_clip_top_.Text),
                    UInt32.Parse(edit_clip_width_.Text),
                    UInt32.Parse(edit_clip_height_.Text));

                    NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, UInt32.Parse(edit_frame_rate_.Text));
                }
                else if (btn_check_window_input_.Checked)
                {
                    if (IntPtr.Zero != cur_sel_capture_window_)
                    {
                        NTSmartPublisherSDK.NT_PB_SetCaptureWindow(publisher_handle_, cur_sel_capture_window_);

                        NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, UInt32.Parse(edit_frame_rate_.Text));

                        NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0, 0, IntPtr.Zero);
                    }
                }
                else if (btn_check_camera_input_.Checked)
                {
                    //摄像头
                    if (-1 != cur_sel_camera_index_
                        && -1 != cur_sel_camera_resolutions_index_
                        && -1 != cur_sel_camera_frame_rate_index_)
                    {
                        CameraInfo camera = cameras_[cur_sel_camera_index_];
                        NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];

                        NTSmartPublisherSDK.NT_PB_SetVideoCaptureDeviceBaseParameter(publisher_handle_,
                            camera.id_.ToString(), (UInt32)cap.width_, (UInt32)cap.height_);

                        NTSmartPublisherSDK.NT_PB_SetFrameRate(publisher_handle_, (UInt32)(cur_sel_camera_frame_rate_index_ + 1));

                        if (btn_check_flip_vertical_camera_.Checked)
                        {
                            NTSmartPublisherSDK.NT_PB_FlipVerticalCamera(publisher_handle_, 1);
                        }
                        else
                        {
                            NTSmartPublisherSDK.NT_PB_FlipVerticalCamera(publisher_handle_, 0);
                        }

                        if (btn_check_flip_horizontal_camera_.Checked)
                        {
                            NTSmartPublisherSDK.NT_PB_FlipHorizontalCamera(publisher_handle_, 1);
                        }
                        else
                        {
                            NTSmartPublisherSDK.NT_PB_FlipHorizontalCamera(publisher_handle_, 0);
                        }

                        Int32 degress = GetCameraRotateDegress();
                        NTSmartPublisherSDK.NT_PB_RotateCamera(publisher_handle_, degress);
                    }
                }

音视频参数设定

其他音视频相关接口参数设定,比是否启用DXGI, Aero模式,软硬编码模式,帧率关键帧间隔码率等设定。

                if (btn_check_dxgi_screen_capturer_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_EnableDXGIScreenCapturer(publisher_handle_, 1);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_EnableDXGIScreenCapturer(publisher_handle_, 0);
                }

                if (check_capture_layered_window_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_EnableScreenCaptureLayeredWindow(publisher_handle_, 1);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_EnableScreenCaptureLayeredWindow(publisher_handle_, 0);
                }

                if (btn_check_capturer_disable_aero_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_DisableAeroScreenCapturer(publisher_handle_, 1);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_DisableAeroScreenCapturer(publisher_handle_, 0);
                }

                if (btn_check_wr_way_capture_window_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetCaptureWindowWay(publisher_handle_, 2);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetCaptureWindowWay(publisher_handle_, 1);
                }

                int cur_video_codec_id = (int)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264;

                if (btn_check_h265_encoder_.Checked)
                {
                    cur_video_codec_id = (int)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H265;
                }

                bool is_hw_encoder = false;

                if ( btn_check_video_hardware_encoder_.Checked)
                {
                    is_hw_encoder = true;
                }

                Int32 cur_sel_encoder_id = 0;
                Int32 cur_sel_gpu = 0;


                if (is_hw_encoder)
                {
                    int cur_sel_hw = combobox_video_encoders_.SelectedIndex;
                    if (cur_sel_hw >= 0)
                    {
                        cur_sel_encoder_id = Convert.ToInt32(combobox_video_encoders_.SelectedValue);
                        cur_sel_gpu = -1;

                        int cur_sel_hw_dev = combobox_video_hardware_encoder_devices_.SelectedIndex;
                        if (cur_sel_hw_dev >= 0)
                        {
                            cur_sel_gpu = Convert.ToInt32(combobox_video_hardware_encoder_devices_.SelectedValue);
                        }
                    }
                    else
                    {
                        is_hw_encoder = false;
                    }
                }

                if (!is_hw_encoder)
                {
                    if ((int)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264 == cur_video_codec_id)
                    {
                        cur_sel_encoder_id = btn_check_openh264_encoder_.Checked ? 1 : 0;
                    }
                }

                NTSmartPublisherSDK.NT_PB_SetVideoEncoder(publisher_handle_, (int)(is_hw_encoder ? 1 : 0), (int)cur_sel_encoder_id, (uint)cur_video_codec_id, (int)cur_sel_gpu);

                if (!btn_check_window_input_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetVideoBitRate(publisher_handle_, Int32.Parse(edit_bit_rate_.Text));
                }
                else
                {
                    // 窗口的分辨率会变, 所以设置一组码率下去
                    Int32 frame_rate = Int32.Parse(edit_bit_rate_.Text);
                    SetBitrateGroup(publisher_handle_, frame_rate);
                }

                NTSmartPublisherSDK.NT_PB_SetVideoQualityV2(publisher_handle_, Int32.Parse(edit_video_quality_.Text));

                NTSmartPublisherSDK.NT_PB_SetVideoMaxBitRate(publisher_handle_, Int32.Parse(edit_video_max_bitrate_.Text));

                NTSmartPublisherSDK.NT_PB_SetVideoKeyFrameInterval(publisher_handle_, Int32.Parse(edit_key_frame_.Text));

                if (cur_video_codec_id == (int)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264)
                {
                    int profile_sel = combox_h264_profile_.SelectedIndex;

                    if (profile_sel != -1)
                    {
                        NTSmartPublisherSDK.NT_PB_SetVideoEncoderProfile(publisher_handle_, profile_sel + 1);
                    }
                }

                NTSmartPublisherSDK.NT_PB_SetVideoEncoderSpeed(publisher_handle_, Int32.Parse(edit_video_encode_speed_.Text));

                // 清除编码器所有的特定的参数
                NTSmartPublisherSDK.NT_PB_ClearVideoEncoderSpecialOptions(publisher_handle_);

                if (cur_sel_encoder_id == 1)
                {
                    // qp_max 和 qp_min 当前只对openh264有效, 这里也就只在openh264使用的场景下设置配置值
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderQPMax(publisher_handle_, Int32.Parse(edit_qp_max_.Text));
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderQPMin(publisher_handle_, Int32.Parse(edit_qp_min_.Text));

                    // openh264 配置特定参数
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderSpecialInt32Option(publisher_handle_, "usage_type", btn_check_openh264_ppt_usage_type_.Checked ? 1 : 0);
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderSpecialInt32Option(publisher_handle_, "rc_mode", btn_check_openh264_rc_bitrate_mode_.Checked ? 1 : 0);
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderSpecialInt32Option(publisher_handle_, "enable_frame_skip", btn_check_openh264_frame_skip_.Checked ? 1 : 0);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderQPMax(publisher_handle_, -1);
                    NTSmartPublisherSDK.NT_PB_SetVideoEncoderQPMin(publisher_handle_, -1);
                }

                // 音频相关设置
                if (btn_check_auido_mic_input_.Checked)
                {
                    int count = combox_auido_input_devices_.Items.Count;
                    if (count != -1 && count > 0)
                    {
                        int cur_sel = combox_auido_input_devices_.SelectedIndex;
                        if (cur_sel != -1)
                        {
                            NTSmartPublisherSDK.NT_PB_SetAuidoInputDeviceId(publisher_handle_, (uint)cur_sel);
                        }
                    }
                }

                // 只采集扬声器时做静音补偿
                if (!btn_check_auido_mic_input_.Checked
                    && btn_check_auido_speaker_input_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetCaptureSpeakerCompensateMute(publisher_handle_, 1);
                }

                if (btn_check_speex_encoder_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetPublisherAudioCodecType(publisher_handle_, 2);

                    NTSmartPublisherSDK.NT_PB_SetPublisherSpeexEncoderQuality(publisher_handle_, Int32.Parse(edit_speex_quality_.Text));
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetPublisherAudioCodecType(publisher_handle_, 1);
                }

                if (btn_check_auido_mic_input_.Checked
                    || btn_check_auido_speaker_input_.Checked)
                {
                    if (btn_check_set_mute_.Checked)
                    {
                        NTSmartPublisherSDK.NT_PB_SetMute(publisher_handle_, 1);
                    }
                }

                if (btn_check_echo_cancel_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetEchoCancellation(publisher_handle_, 1, Int32.Parse(edit_echo_delay_.Text));
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetEchoCancellation(publisher_handle_, 0, 0);
                }

                if (btn_check_noise_suppression_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetNoiseSuppression(publisher_handle_, 1);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetNoiseSuppression(publisher_handle_, 0);
                }

                if (btn_check_agc_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetAGC(publisher_handle_, 1);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetAGC(publisher_handle_, 0);
                }

                if (btn_check_vad_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetVAD(publisher_handle_, 1);
                }
                else
                {
                    NTSmartPublisherSDK.NT_PB_SetVAD(publisher_handle_, 0);
                }

                if (btn_check_auido_mic_input_.Checked
                    && btn_check_auido_speaker_input_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetInputAudioVolume(publisher_handle_, 0, Convert.ToSingle(edit_audio_input_volume_.Text));
                    NTSmartPublisherSDK.NT_PB_SetInputAudioVolume(publisher_handle_, 1, Convert.ToSingle(edit_audio_speaker_input_volume_.Text));
                }
                else if (btn_check_auido_mic_input_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetInputAudioVolume(publisher_handle_, 0, Convert.ToSingle(edit_audio_input_volume_.Text));
                }
                else if (btn_check_auido_speaker_input_.Checked)
                {
                    NTSmartPublisherSDK.NT_PB_SetInputAudioVolume(publisher_handle_, 0, Convert.ToSingle(edit_audio_speaker_input_volume_.Text));
                }

获取视频码率默认值,不是每个开发者都有音视频开发背景,如果不想自行设置码率等一些参数,可参考我们的码率设定。

       private void FillBitrateControlDefValue()
        {
            int w = 640, h = 480;
            int frame_rate = 5;
            bool is_var_bitrate = false;

            GetVideoConfigInfo(ref w, ref h, ref frame_rate, ref is_var_bitrate);

            if (btn_check_openh264_encoder_.Checked)
            {
                is_var_bitrate = false;
            }

            int kbit_rate = CalBitRate(frame_rate, w, h);
            int max_kbit_rate = CalMaxKBitRate(frame_rate, w, h, is_var_bitrate);

            if (is_var_bitrate)
            {
                btn_check_video_bitrate_.CheckState = CheckState.Unchecked;
            }
            else
            {
                btn_check_video_bitrate_.CheckState = CheckState.Checked;
            }

            if (is_var_bitrate)
            {
                edit_bit_rate_.Enabled = false;
                edit_video_quality_.Enabled = true;
            }
            else
            {
                edit_bit_rate_.Enabled = true;
                edit_video_quality_.Enabled = false;
            }

            if (btn_check_video_bitrate_.Checked)
            {
                edit_bit_rate_.Text = kbit_rate.ToString();
                edit_video_max_bitrate_.Text = max_kbit_rate.ToString();
            }
            else
            {
                edit_bit_rate_.Text = "0";
                edit_video_max_bitrate_.Text = max_kbit_rate.ToString();
            }

            bool is_h264 = false;

            if (btn_check_h265_encoder_.Checked)
            {
                is_h264 = false;
            }
            else
            {
                is_h264 = true;
            }

            edit_video_quality_.Text = CalVideoQuality(w, h, is_h264).ToString();

            combox_h264_profile_.SelectedIndex = 2;

            edit_video_encode_speed_.Text = CalVideoEncoderSpeed(w, h, is_h264).ToString();

            // 默认关键帧间隔设置为帧率的2倍
            edit_key_frame_.Text = (frame_rate * 2).ToString();
        }

开始推送

设置推送URL后,调用StartPublisher接口开始推流,如需发送扩展SEI用户数据,推送之前设置下数据发送对接大小。

            if (publisher_handle_ == IntPtr.Zero)
            {
                MessageBox.Show("[publish] handle with null");
            }
            if (!String.IsNullOrEmpty(url))
	        {
                NTSmartPublisherSDK.NT_PB_SetURL(publisher_handle_, url, IntPtr.Zero);
	        }

            //设置用户数据发送队列大小
            NTSmartPublisherSDK.NT_PB_SetPostUserDataQueueMaxSize(publisher_handle_, 3, 0);

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPublisher(publisher_handle_, IntPtr.Zero))
            {
                if (0 == publisher_handle_count_)
                {
                    NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                    publisher_handle_ = IntPtr.Zero;
                }

                is_publishing_ = false;

                MessageBox.Show("调用推流接口失败");

                return;
            }

停止推送

调用NT_PB_StopPublisher()即可,停止推送后,如果没有录像等,可调用NT_PB_Close()接口,关掉实例,并把handle置 IntPtr.Zero。

        private void btn_stop_publish_Click(object sender, EventArgs e)
        {
            publisher_handle_count_--;
            NTSmartPublisherSDK.NT_PB_StopPublisher(publisher_handle_);

            rtmp_play_urls_.Clear();
            UpdateDisplayURLs();

            if (0 == publisher_handle_count_)
            {
                NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                publisher_handle_ = IntPtr.Zero;
            }

            btn_publish.Enabled = true;
            btn_stop_publish.Enabled = false;

            is_publishing_ = false;

            if (0 == publisher_handle_count_)
            {
                if (btn_check_desktop_input_.Checked)
                {
                    btn_choose_screen_region_.Text = "选择屏幕区域";
                }

                btn_check_dxgi_screen_capturer_.Enabled = true;
                check_capture_layered_window_.Enabled = true;

                btn_check_wr_way_capture_window_.Enabled = true;

                btn_desktop_camera_switch_.Text = "切换到摄像头";
                btn_disable_image_watermark_.Text = "停止水印";
                btn_disable_camera_overlay_.Text = "停止叠加摄像头";
                btn_disable_desktop_overlay_.Text = "停止叠加屏幕";

                btn_desktop_camera_switch.Enabled = true;
                btn_camera_overlay_to_desktop.Enabled = true;
                btn_desktop_overlay_to_camera.Enabled = true;
                btn_desktop_camera_switch.Enabled = true;

                btn_check_desktop_input_.Enabled = true;
                btn_check_scale_desktop_.Enabled = true;
                edit_desktop_scale_.Enabled = true;
                btn_check_camera_input_.Enabled = true;

                btn_add_image_watermark_.Enabled = true;

                timer_clock_.Enabled = false;

                if (btn_desktop_camera_switch.Checked
                    || btn_camera_overlay_to_desktop.Checked
                    || btn_desktop_overlay_to_camera.Checked)
                {
                    btn_check_desktop_input_.CheckState = CheckState.Checked;
                    btn_check_camera_input_.CheckState = CheckState.Checked;
                }
                else
                {
                }

                EnableAuidoInputControl();
            }
        }

预览推送数据

设置NT_PB_SetVideoPreviewImageCallBack(),调用NT_PB_StartPreview()接口即可。

        private void btn_preview_Click(object sender, EventArgs e)
        {
            if (btn_check_window_input_.Checked)
            {
                if (IntPtr.Zero == cur_sel_capture_window_)
                {
                    MessageBox.Show("请先下拉选择采集窗口");
                    return;
                }
            }

            if (publisher_handle_ == IntPtr.Zero)
            {
                if (!OpenPublisherHandle())
                {
                    return;
                }
            }

            if (publisher_handle_count_ < 1)
            {
                SetCommonOptionToPublisherSDK();
            }

            video_preview_image_callback_ = new NT_PB_SDKVideoPreviewImageCallBack(SDKVideoPreviewImageCallBack);

            NTSmartPublisherSDK.NT_PB_SetVideoPreviewImageCallBack(publisher_handle_, (int)NTSmartPublisherDefine.NT_PB_E_IMAGE_FORMAT.NT_PB_E_IMAGE_FORMAT_RGB32, IntPtr.Zero, video_preview_image_callback_);

            if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPreview(publisher_handle_, 0, IntPtr.Zero))
            {
                if (0 == publisher_handle_count_)
                {
                    NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                    publisher_handle_ = IntPtr.Zero;
                }

                 MessageBox.Show("预览失败, 请确保选择了视频采集选项");
                return;
            }

            publisher_handle_count_++;

            btn_preview.Enabled = false;
            btn_stop_preview.Enabled = true;

            if (1 == publisher_handle_count_)
            {
                if (btn_check_desktop_input_.Checked)
                {
                    btn_choose_screen_region_.Text = "移动屏幕区域";
                }

                btn_check_dxgi_screen_capturer_.Enabled = false;
                check_capture_layered_window_.Enabled = false;

                btn_check_wr_way_capture_window_.Enabled = false;

                btn_desktop_camera_switch.Enabled = false;
                btn_camera_overlay_to_desktop.Enabled = false;
                btn_desktop_overlay_to_camera.Enabled = false;

                btn_add_image_watermark_.Enabled = false;

                if (btn_desktop_camera_switch.Checked
                    || btn_camera_overlay_to_desktop.Checked
                    || btn_desktop_overlay_to_camera.Checked)
                {

                }
                else
                {
                    btn_check_desktop_input_.Enabled = false;
                    btn_check_camera_input_.Enabled = false;
                }

                DisableAuidoInputControl();
            }

            if (ui_preview_wnd_ == null)
            {
                ui_preview_wnd_ = new nt_pb_ui_preview_wnd();
            }

            ui_preview_wnd_.Show();
        }

        public void VideoPreviewImageCallBack(NT_VideoFrame frame)
        {
            if (cur_image_.plane_ != IntPtr.Zero)
            {
                Marshal.FreeHGlobal(cur_image_.plane_);
                cur_image_.plane_ = IntPtr.Zero;
            }

            cur_image_ = frame;

            if (ui_preview_wnd_ != null)
            {
                ui_preview_wnd_.OnRGBXImage(cur_image_);
            }
        }

        public void SDKVideoPreviewImageCallBack(IntPtr handle, IntPtr user_data, IntPtr image)
        {
            NT_PB_Image pb_image = (NT_PB_Image)Marshal.PtrToStructure(image, typeof(NT_PB_Image));

            NT_VideoFrame pVideoFrame = new NT_VideoFrame();

            pVideoFrame.width_ = pb_image.width_;
            pVideoFrame.height_ = pb_image.height_;

            pVideoFrame.stride_ = pb_image.stride_[0];

            Int32 argb_size = pb_image.stride_[0] * pb_image.height_;
            
            pVideoFrame.plane_ = Marshal.AllocHGlobal(argb_size);

            CopyMemory(pVideoFrame.plane_, pb_image.plane_[0], (UInt32)argb_size);

            if (InvokeRequired)
            {
                BeginInvoke(set_video_preview_image_callback_, pVideoFrame);
            }
            else
            {
                set_video_preview_image_callback_(pVideoFrame);
            }
        }

停止预览更简单,调用NT_PB_StopPreview()。

        private void btn_stop_preview_Click(object sender, EventArgs e)
        {
            publisher_handle_count_--;
            NTSmartPublisherSDK.NT_PB_StopPreview(publisher_handle_);

            if (0 == publisher_handle_count_)
            {
                NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);
                publisher_handle_ = IntPtr.Zero;
            }

            btn_preview.Enabled = true;
            btn_stop_preview.Enabled = false;

            if (0 == publisher_handle_count_)
            {
                if (btn_check_desktop_input_.Checked)
                {
                    btn_choose_screen_region_.Text = "选择屏幕区域";
                }

                btn_check_dxgi_screen_capturer_.Enabled = true;
                check_capture_layered_window_.Enabled = true;

                btn_check_wr_way_capture_window_.Enabled = true;

                btn_desktop_camera_switch_.Text = "切换到摄像头";
                btn_disable_image_watermark_.Text = "停止水印";
                btn_disable_camera_overlay_.Text = "停止叠加摄像头";
                btn_disable_desktop_overlay_.Text = "停止叠加屏幕";

                btn_desktop_camera_switch.Enabled = true;
                btn_camera_overlay_to_desktop.Enabled = true;
                btn_desktop_overlay_to_camera.Enabled = true;
                btn_desktop_camera_switch.Enabled = true;

                btn_check_desktop_input_.Enabled = true;
                btn_check_camera_input_.Enabled = true;

                btn_add_image_watermark_.Enabled = true;

                timer_clock_.Enabled = false;

                if (btn_desktop_camera_switch.Checked
                    || btn_camera_overlay_to_desktop.Checked
                    || btn_desktop_overlay_to_camera.Checked)
                {
                    btn_check_desktop_input_.CheckState = CheckState.Checked;
                    btn_check_camera_input_.CheckState = CheckState.Checked;
                }
                else
                {
                }

                EnableAuidoInputControl();
            }

            ui_preview_wnd_.Hide();
            ui_preview_wnd_ = null;
        }

实时截图

            if (String.IsNullOrEmpty(capture_image_path_))
            {
                MessageBox.Show("请先设置保存截图文件的目录! 点击截图左边的按钮设置!");
                return;
            }

            if (publisher_handle_ == IntPtr.Zero)
            {
                return;
            }

            String name = capture_image_path_ + "\\" + DateTime.Now.ToString("hh-mm-ss") + ".png";

            byte[] buffer1 = Encoding.Default.GetBytes(name);
            byte[] buffer2 = Encoding.Convert(Encoding.Default, Encoding.UTF8, buffer1, 0, buffer1.Length);

            byte[] buffer3 = new byte[buffer2.Length + 1];
            buffer3[buffer2.Length] = 0;

            Array.Copy(buffer2, buffer3, buffer2.Length);

            IntPtr file_name_ptr = Marshal.AllocHGlobal(buffer3.Length);
            Marshal.Copy(buffer3, 0, file_name_ptr, buffer3.Length);

            capture_image_call_back_ = new NT_PB_SDKCaptureImageCallBack(SDKCaptureImageCallBack);

            UInt32 ret = NTSmartPublisherSDK.NT_PB_CaptureImage(publisher_handle_, file_name_ptr, IntPtr.Zero, capture_image_call_back_);

            Marshal.FreeHGlobal(file_name_ptr);

            if (NT.NTBaseCodeDefine.NT_ERC_OK == ret)
            {
                // 发送截图请求成功
            }
            else if ((UInt32)NT.NTSmartPublisherDefine.NT_PB_E_ERROR_CODE.NT_ERC_PB_TOO_MANY_CAPTURE_IMAGE_REQUESTS == ret)
            {
                // 通知用户延时
                MessageBox.Show("Too many capture image requests!");
            }
            else
            {
                // 其他失败
            }