Discussion on cross-platform low-latency RTSP to RTMP push technology solution

Realize RTSP camera data transfer to RTMP and push to the server, which can be realized by third-party libraries or tools. The overall design structure is as follows:

 

A good forwarding module needs low latency first! Secondly, it is stable enough, flexible, has a stateful feedback mechanism, low resource usage, and is cross-platform. It is best provided in the form of an interface, which is convenient for third-party system integration. The overall function design is as follows:

1. Streaming: Play the data callback interface of the SDK through RTSP live streaming to get the audio and video data;

2. Retweet: Through the coded data input interface of the RTMP live push SDK, pass the callback data to the RTMP live push module to realize the forwarding of the RTSP data stream to the RTMP server;

3. Recording: If you need to record, use the RTSP live broadcast SDK to pull the audio and video data and store the MP4 file directly;

4. Snapshot: If you need a real-time snapshot, after pulling the stream, decode and call the snapshot interface on the player side to generate a snapshot, because the snapshot involves video data decoding, you don’t need to enable it if it is not necessary, otherwise it will consume extra performance.

5. Streaming preview: If you want to preview the streaming data, just call the playback interface on the player side to realize the streaming data preview;

6. Transfer the data to AAC and then forward: Considering that the audio from many monitoring devices may be PCMA/PCMU, if you need a more general audio format, you can convert it to AAC and push it through RTMP;

7. Retweet RTMP real-time mute: only need to add a judgment in the place where the audio data is transmitted;

8. Feedback on streaming speed: Feedback on real-time bandwidth through the real-time bit rate feedback event on the RTSP player;

9. Overall network status feedback: Considering that some cameras may be temporarily or abnormally shut down, and the RTMP server is also the same, you can check the overall network status through the event callback status of the push-pull stream, and define it like this: whether the stream cannot be pulled or pushed to the RTMP server.

The following two technical solutions are introduced respectively:

FFmpeg technical solution

1. Install FFmpeg: First, you need to install FFmpeg. FFmpeg is an open source cross-platform video and audio processing tool that supports converting RTSP streams to RTMP streams. You can download the binary installer for Windows from the official FFmpeg website and follow the instructions to install it.
2. Configure FFmpeg: After installing FFmpeg, you need to configure its command line parameters to convert RTSP streams to RTMP streams and push them to the target server. You can use the following command line arguments:

 ffmpeg -i rtsp://[摄像头地址]/[流媒体地址] -f flv rtmp://[服务器地址]/[直播频道]

Among them, ​rtsp://[摄像头地址]/[流媒体地址]​is the RTSP stream address of the camera, ​rtmp://[服务器地址]/[直播频道]​and is the RTMP stream address of the target server. You can modify these parameters according to actual conditions.

1. Running FFmpeg: After configuring FFmpeg, you can use the command line or script file to run FFmpeg. You can run the above command directly on the command line, or write the command into a script file (such as a bat file), and then run the script file. It should be noted that the camera address, streaming media address, server address and live channel in the above solution need to be replaced with actual addresses and information. Also, you need to make sure that the camera's RTSP stream is publicly accessible and that the destination server's RTMP stream address is configured correctly.
2. Integrate into the application: If you need to implement real-time video streaming in the application, you can integrate FFmpeg into the application. You can use FFmpeg's API or command-line interface to call FFmpeg's functions programmatically, and convert the camera's RTSP stream to RTMP stream, and push it to the target server.

SDK technology solution

Take the demo of RTSP to RTMP push C++ on the Windows platform of Daniu Live SDK as an example:

1. Streaming: Streaming is similar to playing, but it does not need to be played (that is, no decoding is required, and the resource consumption is very low). After basic parameter configuration (corresponding to OpenPullHandle() in the demo), set the audio and video data callback, and then call StartPullStream():

1.1 Basic parameter setting:

bool nt_stream_relay_wrapper::OpenPullHandle(const std::string& url, bool is_rtsp_tcp_mode, bool is_mute)
{
	if ( pull_handle_ != NULL )
		return true;

	if ( url.empty() )
		return false;

	duration_ = 0;

	NT_HANDLE pull_handle = NULL;

	ASSERT( pull_api_ != NULL );
	if (NT_ERC_OK != pull_api_->Open(&pull_handle, render_wnd_, 0, NULL))
	{
		return false;
	}

	ASSERT(pull_handle != NULL);

	pull_api_->SetEventCallBack(pull_handle, this, &NT_Pull_SDKEventHandle);

	pull_api_->SetBuffer(pull_handle, 0);
	pull_api_->SetFastStartup(pull_handle, 1);
	pull_api_->SetRTSPTcpMode(pull_handle, is_rtsp_tcp_mode ? 1 : 0);
	pull_api_->SetMute(pull_handle, is_mute ? 1 : 0);

	if ( NT_ERC_OK != pull_api_->SetURL(pull_handle, url.c_str()) )
	{
		pull_api_->Close(pull_handle);
		pull_handle = NULL;
		return false;
	}

	if ( setting_pos_ >= 0ll )
	{
		pull_api_->SetPos(pull_handle, setting_pos_);
	}

	pull_handle_ = pull_handle;

	return true;
}

1.2 Set audio and video data callback:

pull_api_->SetPullStreamVideoDataCallBack(pull_handle_, this, &SP_SDKPullStreamVideoDataHandle);
pull_api_->SetPullStreamAudioDataCallBack(pull_handle_, this, &SP_SDKPullStreamAudioDataHandle);

1.3 Start streaming:

auto ret = pull_api_->StartPullStream(pull_handle_);
	if ( NT_ERC_OK != ret )
	{
		if ( !is_playing_ )
		{
			pull_api_->Close(pull_handle_);
			pull_handle_ = NULL;
		}
		
		return false;
	}

The overall code for streaming is as follows:

bool nt_stream_relay_wrapper::StartPull(const std::string& url, bool is_rtsp_tcp_mode, bool is_transcode_aac)
{
	if ( is_pulling_ )
		return false;

	if ( !OpenPullHandle(url, is_rtsp_tcp_mode) )
		return false;

	pull_api_->SetPullStreamVideoDataCallBack(pull_handle_, this, &SP_SDKPullStreamVideoDataHandle);

	pull_api_->SetPullStreamAudioDataCallBack(pull_handle_, this, &SP_SDKPullStreamAudioDataHandle);

	pull_api_->SetPullStreamAudioTranscodeAAC(pull_handle_, is_transcode_aac? 1: 0);

	auto ret = pull_api_->StartPullStream(pull_handle_);
	if ( NT_ERC_OK != ret )
	{
		if ( !is_playing_ )
		{
			pull_api_->Close(pull_handle_);
			pull_handle_ = NULL;
		}
		
		return false;
	}

	is_pulling_ = true;

	return true;
}

2. Stop streaming:

It is relatively simple to stop the streaming process. First determine whether it is in the streaming state. If it is pulling the stream, just call StopPullStream(). If there is no preview screen, call the Close() interface to close the streaming instance.

void nt_stream_relay_wrapper::StopPull()
{
	if ( !is_pulling_ )
		return;

	pull_api_->StopPullStream(pull_handle_);

	if ( !is_playing_ )
	{
		pull_api_->Close(pull_handle_);
		pull_handle_ = NULL;
	}

	is_pulling_ = false;
}

3. Stream preview:

Streaming preview, to put it bluntly, is to play streaming data. The process is relatively simple. The demo call is as follows. If you don’t need to play the sound, call SetMute() to turn it on/off in real time:

bool nt_stream_relay_wrapper::StartPlay(const std::string& url, bool is_rtsp_tcp_mode, bool is_mute)
{
	if ( is_playing_ )
		return false;

	if ( !OpenPullHandle(url, is_rtsp_tcp_mode, is_mute) )
		return false;

	pull_api_->SetMute(pull_handle_, is_mute ? 1 : 0);

	auto ret = pull_api_->StartPlay(pull_handle_);
	if ( NT_ERC_OK != ret )
	{
		if ( !is_pulling_ )
		{
			pull_api_->Close(pull_handle_);
			pull_handle_ = NULL;
		}

		return false;
	}

	is_playing_ = true;

	return true;
}

4. Close the preview on the streaming side:

void nt_stream_relay_wrapper::StopPlay()
{
	if ( !is_playing_ )
		return;

	pull_api_->StopPlay(pull_handle_);

	if ( !is_pulling_ )
	{
		pull_api_->Close(pull_handle_);
		pull_handle_ = NULL;
	}

	is_playing_ = false;
}

5. Start streaming to the RTMP server:

The process of pushing the stream, as mentioned before, calls the RTMP push module, and then the data source transmits the encoded audio and video data. The demo source code in the figure below also shows that when the RTSP stream is obtained and the RTMP is retweeted, Data decryption processing:

bool nt_stream_relay_wrapper::StartPush(const std::string& url)
{
	if ( is_pushing_ )
		return false;

	if ( url.empty() )
		return false;

	if ( !OpenPushHandle() )
		return false;

	auto push_handle = GetPushHandle();
	ASSERT(push_handle != nullptr);

	ASSERT(push_api_ != NULL);
	if ( NT_ERC_OK != push_api_->SetURL(push_handle, url.c_str(), NULL) )
	{
		if ( !is_started_rtsp_stream_ )
		{
			push_api_->Close(push_handle);
			SetPushHandle(nullptr);
		}

		return false;
	}

	if ( NT_ERC_OK != push_api_->StartPublisher(push_handle, NULL) )
	{
		if ( !is_started_rtsp_stream_ )
		{
			push_api_->Close(push_handle);
			SetPushHandle(nullptr);
		}

		return false;
	}

	is_pushing_ = true;

	return true;
}

6. Transfer and retweet RTMP data:

void nt_stream_relay_wrapper::OnVideoDataHandle(NT_HANDLE handle, NT_UINT32 video_codec_id, 
	NT_BYTE* data, NT_UINT32 size, NT_SP_PullStreamVideoDataInfo* info)
{
	if (!is_pushing_ && !is_started_rtsp_stream_)
		return;

	if ( pull_handle_ != handle )
		return;

	if (data == NULL)
		return;

	if (size < 1)
		return;

	if (info == NULL)
		return;

	std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);

	if (!is_pushing_ && !is_started_rtsp_stream_)
		return;

	if (push_handle_ == NULL)
		return;

	push_api_->PostVideoEncodedDataV2(push_handle_, video_codec_id,
		data, size, info->is_key_frame_, info->timestamp_, info->presentation_timestamp_);
}

void nt_stream_relay_wrapper::OnAudioDataHandle(NT_HANDLE handle, NT_UINT32 auido_codec_id,
	NT_BYTE* data, NT_UINT32 size, NT_SP_PullStreamAuidoDataInfo* info)
{
	if (!is_pushing_ && !is_started_rtsp_stream_)
		return;

	if (pull_handle_ != handle)
		return;

	if (data == NULL)
		return;

	if (size < 1)
		return;

	if (info == NULL)
		return;

	std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);

	if (!is_pushing_ && !is_started_rtsp_stream_)
		return;

	if (push_handle_ == NULL)
		return;

	push_api_->PostAudioEncodedData(push_handle_, auido_codec_id, data, size,
		info->is_key_frame_, info->timestamp_, 
		info->parameter_info_, info->parameter_info_size_);
}

7. Turn off real-time RTMP retweets

void nt_stream_relay_wrapper::StopPush()
{
	if ( !is_pushing_ )
		return;

	is_pushing_ = false;

	std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);

	if ( nullptr == push_handle_ )
		return;

	push_api_->StopPublisher(push_handle_);

	if ( !is_started_rtsp_stream_ )
	{
		push_api_->Close(push_handle_);

		push_handle_ = nullptr;
	}
}

Summarize

No matter which option you choose, you need to ensure the following:

1. Choose a stable and reliable third-party library or service to ensure the quality and reliability of the conversion;
2. Understand and master related technologies and protocols, such as RTSP and RTMP, and how to use related libraries and tools for conversion and processing;
3. Consider performance and resource issues, especially when dealing with a large number of video streams or high concurrency scenarios. You need to make sure your system has enough processing power and bandwidth to avoid problems like delays or dropped frames.

The RTSP to RTMP push on the Windows platform requires some technical preparation and planning, as well as the understanding and use experience of related protocols and tools. To make a basic demo, you can use FFmpeg, but if it is a product, there are too many points to consider. up.