UE4 实时AI抠像 -- 不需绿幕

近期，有个活是要用到NVIDIA MAXINE-VFX-SDK的AI实时抠像。以下记录一下使用过程。

具体的效果见：

【UE5 AI实时抠像插件】 https://www.bilibili.com/video/BV1Qt4y1578A/?share_source=copy_web&vd_source=5d1f34294b1a782c02bc9cb606b40872

注意：本插件基于AI模型，不需要绿幕，一般较复杂环境都可以正常抠像。

1.准备源码

https://github.com/NVIDIA/MAXINE-VFX-SDK

2.封装第三方库

 主要包括bin、lib、include三个文件夹，以及一个.build.cs文件。bin存放dll文件、lib里是引导文件和静态库文件，include里存放的是暴露出来的头文件。

3.UE4插件封装

代码实现的头文件如下：

// Fill out your copyright notice in the Description page of Project Settings.

#pragma once

#include "CoreMinimal.h"
#include "GameFramework/Actor.h"
#include "../../NVIDIA_VFX_SDKLIB/include/opencv2/opencv.hpp"
#include "Runtime/Engine/Classes/Engine/Texture2D.h"
#include "NvidiaCutting.generated.h"

UCLASS()
class NVDIA_VFX_SDK_API ANvidiaCutting : public AActor
{
	GENERATED_BODY()
	
public:	
	// Sets default values for this actor's properties
	ANvidiaCutting();

protected:
	// Called when the game starts or when spawned
	virtual void BeginPlay() override;
	virtual void EndPlay(const EEndPlayReason::Type EndPlayReason)override;

public:	
	// Called every frame
	virtual void Tick(float DeltaTime) override;

	UFUNCTION(BlueprintImplementableEvent, Category = Webcam)
		void OnNextVideoFrame();

	void Prepare(const char* inFile, const char* outFile);
	void UpdateFrame();
	void UpdateTexture();

protected:
	void UpdateTextureRegions(UTexture2D* Texture, int32 MipIndex, uint32 NumRegions, FUpdateTextureRegion2D* Regions, uint32 SrcPitch, uint32 SrcBpp, uint8* SrcData, bool bFreeData);
	FUpdateTextureRegion2D* VideoUpdateTextureRegion;

public:
	//
	UPROPERTY(BlueprintReadOnly, VisibleAnywhere, Category = Webcam)
		FVector2D VideoSize;

	//texture
	UPROPERTY(BlueprintReadOnly, VisibleAnywhere, Category = Webcam)
		UTexture2D* VideoTexture;

	//pipe
	UPROPERTY(BlueprintReadOnly, Category = Webcam)
		TArray<FColor> Data;

	//result of nVidia cutting
	const int camIndex = 0;
	cv::Mat result;
	cv::VideoCapture reader;
	cv::VideoWriter writer;
	unsigned frameNum;
	std::mutex lock_ftrame;

	//is normal 
	bool isNorm = true;
};

 4.UE应用示例

调用非常简单：请参考 UE4 UE5 opencv插件 获取摄像头画面_ue4 opencv_东风吹柳的博客-CSDN博客