Button message distribution mechanism developed by flutter tv (1)

In Android development, we know that user messages are divided into key messages and touch messages. For TV applications, we only consider key messages.

Analysis of the source code shows that Android places key data acquisition and message processing in the Native layer, and provides a callback interface to the application layer. Since the Flutter framework is also written by the Google team, the principle of processing key messages is the same, but in order to achieve cross-platform, the original role played by the android native layer has become the callback interface of key messages in the application layer of each platform. On this basis, another layer of message encapsulation is done, and the key event callback interface is provided to the Flutter UI layer.

Taking the android platform as an example, first we start the analysis from MainActivity, which is the main interface of the application. In flutter, this class inherits FlutterActivity, and FlutterActivity exists in a library specially packaged by flutter for the android system. This library is called flutter.jar, which is responsible for establishing contact between flutter and android.

This library also has an important class: FlutterView, flutter cross-platform is the interface, UI drawing does not depend on system components, then this FlutterView is to encapsulate the interface written by dart into a control that can be used by the android platform. In general, this control After completing all the drawing work of the android application interface, ios also has the same set of mechanisms, thus realizing that different platforms share a set of code for drawing the interface.

Continuing to look down, FlutterActivity inherits from Activity and implements Provider, PluginRegistry, and ViewFactory interfaces. The Provider interface has an abstract method getFlutterView(), which returns a FlutterView object. FlutterActivity implements this method. The specific process is:

//实例化一个FlutterActivityDelegate对象并定义一个Provider对象viewProvider
private final FlutterActivityDelegate delegate = new FlutterActivityDelegate(this, this);
private final Provider viewProvider;

//在空构造函数里面对viewProvider初始化
public FlutterActivity() {
    this.eventDelegate = this.delegate;
    this.viewProvider = this.delegate;
    this.pluginRegistry = this.delegate;
}

// 实现getFlutterView()接口
public FlutterView getFlutterView() {
    return this.viewProvider.getFlutterView();
}

Through the getFlutterView() method, the Android platform application gets the interface drawn by Flutter. Next, let's see how this special control FlutterView is created. In the FlutterActivityDelegate class, a ViewFactory interface is defined with two abstract methods:

public interface ViewFactory {
    FlutterView createFlutterView(Context var1);
    FlutterNativeView createFlutterNativeView();
}

We see that although FlutterActivity implements ViewFactory, it does not really implement the interface, and both methods return empty:

public FlutterView createFlutterView(Context context) {
    return null;
}
public FlutterNativeView createFlutterNativeView() {
    return null;
}

So where is the specific implementation? Looking back, we found that the instantiation of the two interface objects, Provider and PluginRegistry, directly assigned the created FlutterActivityDelegate object. Looking at the FlutterActivityDelegate class, we found that this class also implements the FlutterActivityEvents, Provider, and PluginRegistry interfaces. The default constructors are as follows:

public FlutterActivityDelegate(Activity activity, FlutterActivityDelegate.ViewFactory viewFactory) {
    this.activity = (Activity)Preconditions.checkNotNull(activity);
    this.viewFactory = (FlutterActivityDelegate.ViewFactory)Preconditions.checkNotNull(viewFactory);
}

Combined with the process of creating a FlutterActivityDelegate object with FlutterActivity, it is found that both the activity and viewFactory objects in the FlutterActivityDelegate class point to the FlutterActivity object, and a non-empty judgment is made.

FlutterActivity implements the plugin registration interface PluginRegistry, which opens up the communication channel between the Android platform and the flutter interface. The FlutterActivityEvents interface implemented by FlutterActivityDelegate defines a life cycle method similar to the Android system Activity:

public interface FlutterActivityEvents extends ComponentCallbacks2, ActivityResultListener, RequestPermissionResultListener {
    void onCreate(Bundle var1);
    void onNewIntent(Intent var1);
    void onPause();
    void onResume();
    void onPostResume();
    void onDestroy();
    boolean onBackPressed();
    void onUserLeaveHint();
}

Tracking the code, we will see that in the FlutterView class, flutter notifies the life cycle of the Android platform application to the flutter interface drawn by dart by sending a message. Flutter defines a message system specifically for the interaction of application life cycle data on different platforms:

private final BasicMessageChannel<String> mFlutterLifecycleChannel;

Because the key message mechanism is discussed here, it will not be elaborated on the life cycle.

Continuing to analyze, we found that in the onCreate() method overridden by the FlutterActivityDelegate class, the operations of initializing and creating the FlutterView were performed:

    public void onCreate(Bundle savedInstanceState) {
        if(VERSION.SDK_INT >= 21) {
            Window window = this.activity.getWindow();
            window.addFlags(-2147483648);
            window.setStatusBarColor(1073741824);
            window.getDecorView().setSystemUiVisibility(1280);
        }
        String[] args = getArgsFromIntent(this.activity.getIntent());
        FlutterMain.ensureInitializationComplete(this.activity.getApplicationContext(), args);
        this.flutterView = this.viewFactory.createFlutterView(this.activity);
        if(this.flutterView == null) {
            FlutterNativeView nativeView = this.viewFactory.createFlutterNativeView();
            this.flutterView = new FlutterView(this.activity, (AttributeSet)null, nativeView);
            this.flutterView.setLayoutParams(matchParent);
            this.activity.setContentView(this.flutterView);
            this.launchView = this.createLaunchView();
            if(this.launchView != null) {
                this.addLaunchView();
            }
        }
        boolean reuseIsolate = true;
        if(!this.loadIntent(this.activity.getIntent(), true)) {
            String appBundlePath = FlutterMain.findAppBundlePath(this.activity.getApplicationContext());
            if(appBundlePath != null) {
                this.flutterView.runFromBundle(appBundlePath, (String)null, "main", true);
            }
        }
    }

First, the system calls the createFlutterView() interface of viewFactory. If the creation fails, it calls the createFlutterNativeView() method of viewFactory to instantiate a FlutterNativeView object, and then uses the object as a parameter to call the FlutterView constructor for instantiation.

Open the FlutterView class, we can see that if the incoming FlutterNativeView object is empty, the system will call the FlutterView constructor with a Context object to initialize the FlutterNativeView object it holds.

Finally, the heavyweight debut: FlutterView, this class is the core class that implements the Android application interface to the flutter framework for drawing. The interface can be platform-independent because FlutterView inherits SurfaceView.

We know that Android's View is drawn on the "surface", and for SurfaceView, it acts as the surface itself. SurfaceView is to dig a hole in the window, it is displayed in this hole, other Views are displayed on the window, so View can be displayed on top of SurfaceView, you can also add some layers on top of SurfaceView.

It can be seen from the API that SurfaceView is a subclass of View. It is specially created for making games. It is very powerful. The most important thing is that it supports the OpenGL ES library, and both 2D and 3D effects can be achieved. This is why we say that the Flutter application cross-platform mechanism can be regarded as a game App that implements a similar code engine.

Since the flutter interface is directly inherited from SurfaceView, its drawing process no longer depends on the system platform, and the call of the system controls is decoupled, and the interface written by flutter can run on platforms such as Android and IOS.

And, SurfaceView implements a double buffering mechanism. We know that the Android system provides View for drawing processing, and we can meet most of the drawing needs through a custom View. But we usually customize the View for the active update situation, and the user cannot control the speed of its drawing.

Since the View informs the system to call the view.onDraw method to redraw through the invalidate method, and the Android system redraws the screen by sending the VSYNC signal, the refresh time is 16ms. If the View cannot complete the operation within 16ms, Users will look stuck.

For example, when there is too much logic executed in the draw method, and on an interface that needs to be refreshed frequently, such as a game interface, the main thread will be blocked continuously, resulting in screen freezes. The SurfaceView is equivalent to another drawing thread, it will not block the main thread.

After a brief introduction to SurfaceView, we continue to go down. Flutter cross-platform development, in addition to paying attention to the drawing of the interface, is more concerned about how to communicate with different platforms, mainly the communication of data and some business logic.

Speaking of this, we have to mention the message system of flutter, which is currently divided into 7 modules for management: multilingual, routing navigation, key messages, life cycle, system information, user settings and platform plugins, which almost cover different All the most differentiated functions of the platform, these modules are very dependent on the native system.

Let's look at the definition in the code:

    private final MethodChannel mFlutterLocalizationChannel;
    private final MethodChannel mFlutterNavigationChannel;
    private final BasicMessageChannel<Object> mFlutterKeyEventChannel;
    private final BasicMessageChannel<String> mFlutterLifecycleChannel;
    private final BasicMessageChannel<Object> mFlutterSystemChannel;
    private final BasicMessageChannel<Object> mFlutterSettingsChannel;

    //这个插件消息管理对象被定义为局部的变量，上面几个都是在很多地方使用的
    MethodChannel flutterPlatformChannel = new MethodChannel(this, "flutter/platform", JSONMethodCodec.INSTANCE);

Strictly speaking, these 7 message interfaces are also divided into two categories: one is to communicate data through the api of the reflection native system, and the other is to package the data into a special format in the form of messages and use dart to write in the true sense. The flutter control interaction. The message data stream is transmitted in binary form. For platform systems written in different languages, the binary format must be known.

Look at the initialization of these seven message management objects:

this.mFlutterLocalizationChannel = new MethodChannel(this, "flutter/localization", JSONMethodCodec.INSTANCE);
this.mFlutterNavigationChannel = new MethodChannel(this, "flutter/navigation", JSONMethodCodec.INSTANCE);
this.mFlutterKeyEventChannel = new BasicMessageChannel(this, "flutter/keyevent", JSONMessageCodec.INSTANCE);
this.mFlutterLifecycleChannel = new BasicMessageChannel(this, "flutter/lifecycle", StringCodec.INSTANCE);
this.mFlutterSystemChannel = new BasicMessageChannel(this, "flutter/system", JSONMessageCodec.INSTANCE);
this.mFlutterSettingsChannel = new BasicMessageChannel(this, "flutter/settings", JSONMessageCodec.INSTANCE);

//platformPlugin负责处理平台插件消息，并提供回调接口
PlatformPlugin platformPlugin = new PlatformPlugin(activity);
MethodChannel flutterPlatformChannel = new MethodChannel(this, "flutter/platform", JSONMethodCodec.INSTANCE);
//监听平台插件消息回调接口 
flutterPlatformChannel.setMethodCallHandler(platformPlugin);

Here, we only look at the key message processing flow.

The FlutterView class has two interfaces for handling key events, one is onKeyUp() and the other is onKeyDown(), which correspond to key release and press events respectively. The two methods have the same process. In TV interaction, we generally only care about the remote control keys. Press the event, so we only analyze onKeyDown() here.

public boolean onKeyDown(int keyCode, KeyEvent event) {
    if(!this.isAttached()) {
        return super.onKeyDown(keyCode, event);
    } else {
        Map<String, Object> message = new HashMap();
        message.put("type", "keydown");
        message.put("keymap", "android");
        this.encodeKeyEvent(event, message);
        this.mFlutterKeyEventChannel.send(message);
        return super.onKeyDown(keyCode, event);
    }
}

Analysis of the code shows that when FlutterView is not bound to Activity, it directly returns the return value of the onKeyDown() method of the parent class View, otherwise a Map object will be created, the key is String type, and the value is Object, because the type is defined when the message is sent It is generic, and the Object class is the parent class of all classes, so value does not specify a subclass object type.

This map stores the data required for key processing. The first set of key-value pairs is type, and this value has two possible values. If it is onKeyDown(), the value is "keydown". Similarly, onKeyUp(), the value is "keydown". "keyup". The key field of the second set of key-value pairs is keymap, which is used to distinguish which platform the key message is sent from. If the application is running on the Android system, the passed value is "android", and if it is running on the Fuchsia system, the passed value For "fuchsia", currently the flutter api only handles the key messages of these two systems. As for IOS, since I am not familiar with this system framework, maybe there is another mechanism for key processing?

The remaining fields are the same for onKeyUp() and onKeyDown(), so the system encapsulates a method encodeKeyEvent():

private void encodeKeyEvent(KeyEvent event, Map<String, Object> message) {
    message.put("flags", Integer.valueOf(event.getFlags()));
    message.put("codePoint", Integer.valueOf(event.getUnicodeChar()));
    message.put("keyCode", Integer.valueOf(event.getKeyCode()));
    message.put("scanCode", Integer.valueOf(event.getScanCode()));
    message.put("metaState", Integer.valueOf(event.getMetaState()));
}

Are these KeyEvent fields familiar? Let's look at the source code comments of flutter:

  /// See <https://developer.android.com/reference/android/view/KeyEvent.html#getFlags()>;
  final int flags;
  /// See <https://developer.android.com/reference/android/view/KeyEvent.html#getUnicodeChar()>;
  final int codePoint;
  /// See <https://developer.android.com/reference/android/view/KeyEvent.html#getKeyCode()>;
  final int keyCode;
  /// See <https://developer.android.com/reference/android/view/KeyEvent.html#getScanCode()>;
  final int scanCode;
  /// See <https://developer.android.com/reference/android/view/KeyEvent.html#getMetaState()>;
  final int metaState;

The meaning is very clear. These fields are the KeyEvent attributes corresponding to Android. If you don't know what they are doing, let's go to the Android development official website to check the relevant api documents.

At this point, we are familiar with the process of bridging key messages between Flutter and the Android platform, and by the way, we understand the principle of interaction between Flutter and Android native API. Finally, summarize the key class calls of Flutter's key message distribution process from the bottom layer to the UI layer on the Android platform:

1. The Android platform handles keystroke messages

   • Goldfish_event.c

     • in the Linux kernel layer

     • Responsible for driving key messages

   • EventHub.cpp

     • hardware abstraction layer

     • Used to read the RawEvent in the device file

   • com_android_server_KeyInputQueue.cpp

     • JNI native methods

     • Provides the function android_server_KeyInputQueue_readEvent to the Java framework layer for reading input device events

   • WindowManagerService.java

     • Java framework layer

     • Window management service, open a thread-driven InputReader through the interface provided by InputManager to continuously read events from the device file under the /dev/input/ directory, and then distribute it to clients connected to the WindowManagerService service through InputDispatcher.

   • KeyInputQueue.java

     • Java framework layer

     • Create a thread, read events in a loop, and put events into the event queue

   • InputManager.java

     • Java framework layer

     • Monitor key events

   • ViewRootImpl.java

     • Android UI layers

     • Android's Activity registers and destroys the keyboard message receiving channel through the setView() interface of this class

   • DecorView.java

     • Android UI layers

     • Distribute key events, the important method dispatchKeyEvent()

   • FlutterView.java

     • Android and Flutter UI bridge layer

     • Encapsulate key message

2. Flutter intercepts platform key events as messages

   • system_channels.dart

     • dart framework layer

     • The flutter message system is responsible for the delivery of key messages in key events

   • raw_keyboard.dart

     • dart framework layer

     • It defines several classes related to key events, including two abstract classes RawKeyEvent, RawKeyEventData and their subclass implementations, and one RawKeyBoard class that listens to key events. This dart language likes to put several related classes in one file, which is contrary to the rule of one class and one file in Java language, and needs to be adapted.

     • RawKeyEvent has a factory method that returns an object implemented by a subclass, that is, a RawKeyUpEvent or RawKeyDownEvent object that inherits RawKeyEvent. The specific process is to first generate the RawKeyEventData of a specific platform from the received key data of different platforms, and distinguish the platform according to the keymap field in the message. For example, on the Android platform, the data is instantiated into a RawKeyEventDataAndroid subclass object. At the same time, it will decide whether to return a RawKeyUpEvent object or a RawKeyDownEvent object according to the type field in the message.

       /// Creates a concrete [RawKeyEvent] class from a message in the form received
       /// on the [SystemChannels.keyEvent] channel.
       factory RawKeyEvent.fromMessage(Map<String, dynamic> message) {
       RawKeyEventData data;
       
       final String keymap = message['keymap'];
       switch (keymap) {
         case 'android':
           data = new RawKeyEventDataAndroid(
             flags: message['flags'] ?? 0,
             codePoint: message['codePoint'] ?? 0,
             keyCode: message['keyCode'] ?? 0,
             scanCode: message['scanCode'] ?? 0,
             metaState: message['metaState'] ?? 0,
           );
           break;
         case 'fuchsia':
           data = new RawKeyEventDataFuchsia(
             hidUsage: message['hidUsage'] ?? 0,
             codePoint: message['codePoint'] ?? 0,
             modifiers: message['modifiers'] ?? 0,
           );
           break;
         default:
           // We don't yet implement raw key events on iOS, but we don't hit this
           // exception because the engine never sends us these messages.
           throw new FlutterError('Unknown keymap for key events: $keymap');
       }
       
       final String type = message['type'];
       switch (type) {
         case 'keydown':
           return new RawKeyDownEvent(data: data);
         case 'keyup':
           return new RawKeyUpEvent(data: data);
         default:
           throw new FlutterError('Unknown key event type: $type');
       }
       }
       

     • The RawKeyBoard class is responsible for monitoring the key messages sent by the platform, and implements the message callback. In the message callback, the fromMessage() method of RawKeyEvent is called to instantiate a RawKeyEvent object, and the object is used as a parameter for the control that responds to the key event in the callback method.

       class RawKeyboard {
         RawKeyboard._() {
           SystemChannels.keyEvent.setMessageHandler(_handleKeyEvent);
         }
       
         /// The shared instance of [RawKeyboard].
         static final RawKeyboard instance = new RawKeyboard._();
       
         final List<ValueChanged<RawKeyEvent>> _listeners = <ValueChanged<RawKeyEvent>>[];
       
         /// Calls the listener every time the user presses or releases a key.
         ///
         /// Listeners can be removed with [removeListener].
         void addListener(ValueChanged<RawKeyEvent> listener) {
           _listeners.add(listener);
         }
       
         /// Stop calling the listener every time the user presses or releases a key.
         ///
         /// Listeners can be added with [addListener].
         void removeListener(ValueChanged<RawKeyEvent> listener) {
           _listeners.remove(listener);
         }
       
         Future<dynamic> _handleKeyEvent(dynamic message) async {
           if (_listeners.isEmpty)
             return;
           final RawKeyEvent event = new RawKeyEvent.fromMessage(message);
           if (event == null)
             return;
           for (ValueChanged<RawKeyEvent> listener in new List<ValueChanged<RawKeyEvent>>.from(_listeners))
             if (_listeners.contains(listener))
               listener(event);
         }
       }

   • raw_keyboard_listener.dart

     • flutter ui layers

     • Responsible for defining the control container that responds to keys, listening to focus change events, and providing key callback interfaces

     • This file defines a RawKeyboardListener class, which inherits StatefulWidget, so it is a stateful control. When the state is initialized, it will monitor the focus change of the control:

       @override
       void initState() {
       super.initState();
       widget.focusNode.addListener(_handleFocusChanged);
       }
       
       void _handleFocusChanged() {
       if (widget.focusNode.hasFocus)
       _attachKeyboardIfDetached();
       else
       _detachKeyboardIfAttached();
       }
       

     • Only when the control has the focus will it respond to the key event. The specific method is to listen to the key event and implement the callback when the control has the focus:

       void _attachKeyboardIfDetached() {
        if (_listening)
        return;
        RawKeyboard.instance.addListener(_handleRawKeyEvent);
        _listening = true;
       }
       

     • When losing focus, remove the event listener:

       void _detachKeyboardIfAttached() {
        if (!_listening)
        return;
        RawKeyboard.instance.removeListener(_handleRawKeyEvent);
        _listening = false;
       }
       

     • Let's look at the _handleRawKeyEvent() method, which actually calls back the key event interface through the user's key operation:

       void _handleRawKeyEvent(RawKeyEvent event) {
        if (widget.onKey != null)
        widget.onKey(event);
       }

     • How to use this control? In fact, the biggest feature of the dart language is that everything is a control, so if a basic control wants to expand its function, it must be wrapped by a control with this function, that is, as a child node of others, in the design mode, we call it for decorative patterns. Specifically here, that is, if you want a control to respond to key events, you need to make the parent node of the control RawKeyboardListener.

As for how to implement UI interaction after the Flutter interface receives the key message from the system platform, the next article on the key message distribution mechanism developed by flutter tv (part 2) will introduce in detail.