Introduction:
The first project I did after joining the company was about video. Because I used the SDK provided by others, it was easy to implement the functions. So at the end of the project, I thought that I couldn’t just use it, but at least know how to use it. The principle process! So let me explain some of my explanations about the video connection process of WebRTC:
Regarding the WebRTC library , although it provides point-to-point communication, the premise is that both parties must connect to the server. First, the metadata (actually signaling) exchanged between browsers to establish communication must go through the server. Secondly, The official NAT and firewall also need to go through the server (actually, it can be understood as drilling holes, which is to find a way to establish a connection). As for the server, I don't know much about it and won't say much.
Android side:
Here is a compiled WebRTC project , otherwise it will be difficult for newcomers to compile it themselves. Regarding the android client, you only need to understand the RTCPeerConnection interface, which represents a WebRTC connection from the local computer to the remote end and provides the implementation of methods to create, maintain, monitor, and close the connection. We still need to understand two things: 1. Determine the characteristics of the media stream on this machine, such as resolution, encoding capabilities, etc. (This is actually included in the SDP description, which will be explained later) 2. The network addresses of the hosts at both ends of the connection (In fact, it is ICE Candidate)
Principle (important):
Exchange SDP descriptors through offer and answer: (For example, A initiates a video request to B) For example, A and B need to establish a point-to-point connection. The approximate process is: both ends first establish a PeerConnection instance (here called pc), and A uses pc The provided createOffer() method establishes an offer signaling containing an SDP descriptor. Similarly, A passes A's SDP descriptor to A's pc object through the setLocalDescription() method provided by pc, and A sends the offer signaling through the server. to B. B extracts the SDP descriptor contained in A's offer signaling and hands it to B's pc instance object through the setRemoteDescription() method provided by pc. B uses the createAnswer() method provided by pc to create an SDP containing B. Descriptor answer signaling, B uses the setLocalDescription() method provided by pc to hand over its SDP descriptor to its own pc instance object, and then sends the answer signaling to A through the server. Finally, A receives B's answer signaling. Finally, extract the SDP descriptor and call the setRemoteDescription() method to hand it to A's own pc instance object.
Therefore, the process of video connection at both ends is roughly the above process. Through a series of signaling exchanges, the pc instance objects created by A and B contain the SDP descriptors of A and B, completing the first of the above two things. thing, then the second thing is to obtain the network addresses of the hosts at both ends of the connection, as follows:
Establish a NAT/firewall traversal connection (hole punching) through the ICE framework. This URL should be directly accessible from the outside world. WebRTC uses the ICE framework to obtain this URL. When the PeerConnection is created, the address of the ICE server can be passed in, such as :
private void init(Context context) {
PeerConnectionFactory.initializeAndroidGlobals(context, true, true, true);
this.factory = new PeerConnectionFactory();
this.iceServers.add(new IceServer("turn:turn.realtimecat.com:3478", "learningtech", "learningtech"));
}
注意：“turn:turn.realtimecat.com:3478”这段字符其实就是该ICE服务器的地址。Of course, this address also needs to be exchanged. Taking AB as an example, the exchange process is as follows (PeerConnection is referred to as PC): A and B each create PC instances configured with ICE servers, and add onicecandidate event callbacks to them when network candidates are available. , the onicecandidate function will be called inside the callback function. A or B encapsulates the network candidate message in ICE Candidate signaling, relays it through the server, and passes it to the other party. A or B receives the ICE Candidate signaling sent by the other party through the server relay. When, parse it and obtain the network candidate, add it to the PC instance through the addIceCandidate() method of the PC instance.
In this way, the connection is established, and the stream can be added to RTCPeerConnection through addStream() to transmit media stream data. After adding the stream to the RTCPeerConnection instance, the other party can listen through the callback function bound to onaddstream. Call addStream() before the connection is completed, and after the connection is established, the other party can still monitor the media stream.
The following is the code implementation process I made using SDK:
1. First, in the interface layout, write the GLSurfaceView control in the xml file where the video is to be displayed. Of course, you can also add the control dynamically (I wrote it statically, this random)
2. First initialize the control, that is: (Of course, you can initialize it as soon as you enter the interface, or you can initialize it after connecting to the server later, in any order)
public void initPlayView(GLSurfaceView glSurfaceView) {
VideoRendererGui.setView(glSurfaceView, (Runnable)null);
this.isVideoRendererGuiSet = true;
}This step is to add glSurfaceView to VideoRendererGui as the interface to be displayed.
3. Log in to the video server. This step should actually be the first (the order of steps 2 and 3 is not limited)
public void connect(String url) throws URISyntaxException {
//先初始化配置网络ping的一些信息
this.init(url);
//然后在连接服务器
this.client.connect();
}
private void init(String url) throws URISyntaxException {
if (!this.init) {
Options opts = new Options();
opts.forceNew = true;
opts.reconnection = false;
opts.query = "user_id=" + this.username;
this.client = IO.socket(url, opts);
this.client.on("connect", new Listener() {
public void call(Object... args) {
if (Token.this.mEventHandler != null) {
Message msg = Token.this.mEventHandler.obtainMessage(10010);
Token.this.mEventHandler.sendMessage(msg);
}
}
}).on("disconnect", new Listener() {
public void call(Object... args) {
if (Token.this.mEventHandler != null) {
Message msg = Token.this.mEventHandler.obtainMessage(10014);
Token.this.mEventHandler.sendMessage(msg);
}
}
}).on("error", new Listener() {
public void call(Object... args) {
if (Token.this.mEventHandler != null) {
Error error = null;
if (args.length > 0) {
try {
error = (Error) (new Gson()).fromJson((String) args[0], Error.class);
} catch (Exception var4) {
var4.printStackTrace();
}
}
Message msg = Token.this.mEventHandler.obtainMessage(10013, error);
Token.this.mEventHandler.sendMessage(msg);
}
}
}).on("connect_timeout", new Listener() {
public void call(Object... args) {
if (Token.this.mEventHandler != null) {
Message msg = Token.this.mEventHandler.obtainMessage(10012);
Token.this.mEventHandler.sendMessage(msg);
}
}
}).on("connect_error", new Listener() {
public void call(Object... args) {
if (Token.this.mEventHandler != null) {
Message msg = Token.this.mEventHandler.obtainMessage(10011);
Token.this.mEventHandler.sendMessage(msg);
}
}
}).on("message", new Listener() {
public void call(Object... args) {
try {
Token.this.handleMessage(cn.niusee.chat.sdk.Message.parseMessage((JSONObject) args[0]));
} catch (MessageErrorException var3) {
var3.printStackTrace();
}
}
});
this.init = true;
}
}4. When logging in, set some monitors for the token:
public interface OnTokenCallback {
void onConnected();//视频连接成功的回调
void onConnectFail();
void onConnectTimeOut();
void onError(Error var1);//视频连接错误的回调
void onDisconnect();//视频断开的回调
void onSessionCreate(Session var1);//视频打洞成功的回调
}5. The following is my code for logging in to the server:
public void login(String username) {
try {
SingleChatClient.getInstance(getApplication()).setOnConnectListener(new SingleChatClient.OnConnectListener() {
@Override
public void onConnect() {
// loadDevices();
Log.e(TAG, "连接视频服务器成功");
state.setText("登录视频服务器成功!");
}
@Override
public void onConnectFail(String reason) {
Log.e(TAG, "连接视频服务器失败");
state.setText("登录视频服务器失败!" + reason);
}
@Override
public void onSessionCreate(Session session) {
Log.e(TAG, "来电者名称:" + session.callName);
mSession = session;
accept.setVisibility(View.VISIBLE);
requestPermission(new String[]{Manifest.permission.CAMERA}, "请求设备权限", new GrantedResult() {
@Override
public void onResult(boolean granted) {
if(granted){
createLocalStream();
}else {
Toast.makeText(MainActivity.this,"权限拒绝",Toast.LENGTH_SHORT).show();
}
}
});
mSession.setOnSessionCallback(new OnSessionCallback() {
@Override
public void onAccept() {
Toast.makeText(MainActivity.this, "视频接收", Toast.LENGTH_SHORT).show();
}
@Override
public void onReject() {
Toast.makeText(MainActivity.this, "拒绝通话", Toast.LENGTH_SHORT).show();
}
@Override
public void onConnect() {
Toast.makeText(MainActivity.this, "视频建立成功", Toast.LENGTH_SHORT).show();
}
@Override
public void onClose() {
Log.e(TAG, "onClose 我是被叫方");
hangup();
}
@Override
public void onRemote(Stream stream) {
Log.e(TAG, "onRemote 我是被叫方");
mRemoteStream = stream;
mSingleChatClient.getChatClient().playStream(stream, new Point(0, 0, 100, 100, false));
mSingleChatClient.getChatClient().playStream(mLocalStream, new Point(72, 72, 25, 25, false));
}
@Override
public void onPresence(Message message) {
}
});
}
});
// SingleChatClient.getInstance(getApplication()).connect(UUID.randomUUID().toString(), WEB_RTC_URL);
Log.e("MainActicvity===",username);
SingleChatClient.getInstance(getApplication()).connect(username, WEB_RTC_URL);
} catch (URISyntaxException e) {
e.printStackTrace();
Log.d(TAG, "连接失败");
}
}Notice:
onSessionCreate（Session session）这个回调是当检测到有视频请求来的时候才会触发，所以这里可以设置当触发该回调是显示一个接受按钮，一个拒绝按钮，session中携带了包括对方的userName，以及各种信息（上面所说的SDP描述信息等），这个时候通过session来设置OnSessionCallback的回调信息，public interface OnSessionCallback {
void onAccept();//用户同意
void onReject();//用户拒绝
void onConnect();//连接成功
void onClose();//连接掉开
void onRemote(Stream var1);//当远程流开启的时候，就是对方把他的本地流传过来的时候
void onPresence(Message var1);//消息通道过来的action消息，action是int型，远程控制的时候可以使用这个int型信令发送指令
}Notice:
@Override
public void onRemote(Stream stream) {
Log.e(TAG, "onRemote 我是被叫方");
mRemoteStream = stream;
mSingleChatClient.getChatClient().playStream(stream, new Point(0, 0, 100, 100, false));
mSingleChatClient.getChatClient().playStream(mLocalStream, new Point(72, 72, 25, 25, false));
} Here, when the remote stream callback is executed, the other party's screen can be displayed, and its own local stream small window can be refreshed. (The most important premise is that if you want the other party to receive the local stream you sent, you must first call playStream yourself, so that the other party can receive the local stream you sent through the onRemote callback)
6. When A actively requests B to start a video chat, it needs to be called manually:
private void call() {
try {
Log.e("MainActivity===","对方username："+userName);
mSession = mSingleChatClient.getToken().createSession(userName);
//userName是指对方的用户名,并且这里要新建session对象，因为你是主动发起呼叫的，如果是被呼叫的则在onSessionCreate(Session session)回调中会拿到session对象的。（主叫方和被叫方不太一样）
} catch (SessionExistException e) {
e.printStackTrace();
}
requestPermission(new String[]{Manifest.permission.CAMERA}, "请求设备相机权限", new GrantedResult() {
@Override
public void onResult(boolean granted) {
if(granted){//表示用户允许
createLocalStream();//权限允许之后，首先打开本地流，以及摄像头开启
}else {//用户拒绝
Toast.makeText(MainActivity.this,"权限拒绝",Toast.LENGTH_SHORT).show();
return;
}
}
});
mSession.setOnSessionCallback(new OnSessionCallback() {
@Override
public void onAccept() {
Toast.makeText(MainActivity.this, "通话建立成功", Toast.LENGTH_SHORT).show();
}
@Override
public void onReject() {
Toast.makeText(MainActivity.this, "对方拒绝了您的视频通话请求", Toast.LENGTH_SHORT).show();
}
@Override
public void onConnect() {
}
@Override
public void onClose() {
mSingleChatClient.getToken().closeSession(userName);
Log.e(TAG, "onClose 我是呼叫方");
hangup();
Toast.makeText(MainActivity.this, "对方已中断视频通话", Toast.LENGTH_SHORT).show();
}
@Override
public void onRemote(Stream stream) {
mStream = stream;
Log.e(TAG, "onRemote 我是呼叫方");
Toast.makeText(MainActivity.this, "视频建立成功", Toast.LENGTH_SHORT).show();
mSingleChatClient.getChatClient().playStream(stream, new Point(0, 0, 100, 100, false));
mSingleChatClient.getChatClient().playStream(mLocalStream, new Point(72, 72, 25, 25, false));
}
@Override
public void onPresence(Message message) {
}
});
if (mSession != null) {
mSession.call();//主动开启呼叫对方
}
}Create a local stream:
private void createLocalStream() {
if (mLocalStream == null) {
try {
String camerName = CameraDeviceUtil.getFrontDeviceName();
if(camerName==null){
camerName = CameraDeviceUtil.getBackDeviceName();
}
mLocalStream = mSingleChatClient.getChatClient().createStream(camerName,
new Stream.VideoParameters(640, 480, 12, 25), new Stream.AudioParameters(true, false, true, true), null);
} catch (StreamEmptyException | CameraNotFoundException e) {
e.printStackTrace();
}
} else {
mLocalStream.restart();
}
mSingleChatClient.getChatClient().playStream(mLocalStream, new Point(72, 72, 25, 25, false));
}Summarize:
The above is just a brief introduction to the principles and usage of the SDK (if you want to know more about the SDK, you can leave a message in the comments below and I will send it to you). I will focus on explaining the principles in more detail in the future, but there is one more important point. The problem is about multi-network interoperability, and party A is in China Unicom 4G state, party B is in telecom WIFI state, or party B is in mobile 4G state. There may be problems with interoperability between these different network operators, so test it before At that time, we conducted special packet capture and debugging, and the results showed that when A was using Unicom 4G and initiated video to B (Mobile 4G), A was always in the hole-making state, but the hole was blocked and did not go away. Forwarding (that is, the Internet), theoretically speaking, forwarding is the last situation, that is, all the previous methods fail, then forwarding is definitely possible, but forwarding involves setting up a transfer server, which requires a lot of bandwidth. It can guarantee the video connection, so the current video supports the intranet (under the same wifi) by default, or the interoperability between the same network operators. As for the interoperability between other different network operators, 100% interoperability is not guaranteed, so This is a difficult question.
Author: Ai Shen Accidentally
The original article explains the principles of WebRTC video implementation on the Android side - Nuggets
★The business card at the end of the article allows you to receive free audio and video development learning materials, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmap, etc.
See below! ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓