gstreamer基础教程6-Media formats and Pad Capabilities

官网：https://gstreamer.freedesktop.org/

Demo基础教程：https://gstreamer.freedesktop.org/documentation/tutorials/basic/concepts.html

Demo下载地址：git://anongit.freedesktop.org/gstreamer/gst-docs

Goal

主要讲解pad的能力集，主要是从理论上，一般他们不可见，但他们是element的基础。

Pad Capabilities are a fundamental(基础) element of GStreamer, although most of the time they are invisible because the framework handles them automatically. This somewhat theoretical(理论) tutorial shows:

• What are Pad Capabilities.

• How to retrieve them.

• When to retrieve them.

• Why you need to know about them.

Introduction

Pads

pans可以支持很多能力集，但不支持区间段，只支持固定值。caps的主要作用就是找到两个pad的共同部分，进而可以进行连接。

As it has already been shown, Pads allow information to enter and leave an element. The Capabilities (or Caps, for short) of a Pad, then, specify what kind of information can travel through the Pad. For example, “RGB video with a resolution of 320x200 pixels and 30 frames per second”, or “16-bits per sample audio, 5.1 channels at 44100 samples per second”, or even compressed formats like mp3 or h264.

Pads can support multiple Capabilities (for example, a video sink can support video in different types of RGB or YUV formats) and Capabilities can be specified as ranges (for example, an audio sink can support samples rates from 1 to 48000 samples per second). However, the actual information traveling from Pad to Pad must have only one well-specified type. Through a process known as negotiation(谈判), two linked Pads agree on a common type, and thus the Capabilities of the Pads become fixed(稳固) (they only have one type and do not contain ranges). The walkthrough(演练) of the sample code below should make all this clear.

In order for two elements to be linked together, they must share a common subset of Capabilities (Otherwise they could not possibly understand each other). This is the main goal of Capabilities.

As an application developer, you will usually build pipelines by linking elements together (to a lesser extent if you use all-in-all elements like playbin). In this case, you need to know the Pad Caps (as they are familiarly referred to) of your elements, or, at least, know what they are when GStreamer refuses to link two elements with a negotiation error.

Pad templates

pads是通过模板被创建的，模板会指明可能具有的能力集，如果二个pad的模板里边的能力集没有公共子集的话，他们会提前结束能力集协商。所以pad的模板能力集会受限被对比，模板能力集对比通过后才会实例化，进一步对比具体实例的能力集。比如一个pad是audio，一个pad是video，模板就没有相同的子集，所以就会直接失败。

Pads are created from Pad Templates, which indicate(指明) all possible Capabilities a Pad could ever have. Templates are useful to create several(几个) similar Pads, and also allow early refusal of connections between elements: If the Capabilities of their Pad Templates do not have a common subset (their intersection is empty), there is no need to negotiate further.

Pad Templates can be viewed as the first step in the negotiation process. As the process evolves(进行), actual Pads are instantiated(实例化) and their Capabilities refined(更精炼) until they are fixed (or negotiation fails).

Capabilities examples

SINK template: 'sink'
  Availability: Always
  Capabilities:
    audio/x-raw
               format: S16LE
                 rate: [ 1, 2147483647 ]
             channels: [ 1, 2 ]
    audio/x-raw
               format: U8
                 rate: [ 1, 2147483647 ]
             channels: [ 1, 2 ]

This pad is a sink which is always available on the element (we will not talk about availability for now). It supports two kinds of media, both raw audio in integer format (audio/x-raw): signed, 16-bit little endian and unsigned 8-bit. The square brackets(方括号) indicate a range: for instance, the number of channels varies from 1 to 2.

SRC template: 'src'
  Availability: Always
  Capabilities:
    video/x-raw
                width: [ 1, 2147483647 ]
               height: [ 1, 2147483647 ]
            framerate: [ 0/1, 2147483647/1 ]
               format: { I420, NV12, NV21, YV12, YUY2, Y42B, Y444, YUV9, YVU9, Y41B, Y800, Y8, GREY, Y16 , UYVY, YVYU, IYU1, v308, AYUV, A420 }

video/x-raw indicates that this source pad outputs raw video. It supports a wide range of dimensions(规模) and framerates, and a set of YUV formats (The curly braces(花括号) indicate a list). All these formats indicate different packing and subsampling of the image planes.

Last remarks

可以使用gst-inspect-1.0工具查看element的属性，有些element的属性依赖硬件平台。所以不同平台的element的能力集可能是不同的，极端情况下，统一平台二次执行能力集都有可能发生变化。这个实例将通过两个element的状态变化过程观察他们的caps的变化情况。

You can use the gst-inspect-1.0 tool described in Basic tutorial 10: GStreamer tools to learn about the Caps of any GStreamer element.

Bear(承受) in mind that some elements query(询问) the underlying hardware for supported formats and offer their Pad Caps accordingly (They usually do this when entering the READY state or higher). Therefore, the shown caps can vary from platform to platform, or even from one execution to the next (even though this case is rare).

This tutorial instantiates(实例化) two elements (this time, through their factories), shows their Pad Templates, links them and sets the pipeline to play. On each state change, the Capabilities of the sink element's Pad are shown, so you can observe(观察) how the negotiation proceeds until the Pad Caps are fixed.

A trivial(简单的、不重要的) Pad Capabilities Example

Copy this code into a text file named basic-tutorial-6.c (or find it in your GStreamer installation).

basic-tutorial-6.c

#include <gst/gst.h>

/* Functions below print the Capabilities in a human-friendly format */
static gboolean print_field (GQuark field, const GValue * value, gpointer pfx) {
  gchar *str = gst_value_serialize (value);

  g_print ("%s  %15s: %s\n", (gchar *) pfx, g_quark_to_string (field), str);
  g_free (str);
  return TRUE;
}

static void print_caps (const GstCaps * caps, const gchar * pfx) {
  guint i;

  g_return_if_fail (caps != NULL);

  if (gst_caps_is_any (caps)) {
    g_print ("%sANY\n", pfx);
    return;
  }
  if (gst_caps_is_empty (caps)) {
    g_print ("%sEMPTY\n", pfx);
    return;
  }

  for (i = 0; i < gst_caps_get_size (caps); i++) {
    GstStructure *structure = gst_caps_get_structure (caps, i);

    g_print ("%s%s\n", pfx, gst_structure_get_name (structure));
    gst_structure_foreach (structure, print_field, (gpointer) pfx);
  }
}

/* Prints information about a Pad Template, including its Capabilities */
static void print_pad_templates_information (GstElementFactory * factory) {
  const GList *pads;
  GstStaticPadTemplate *padtemplate;

  g_print ("Pad Templates for %s:\n", gst_element_factory_get_longname (factory));
  if (!gst_element_factory_get_num_pad_templates (factory)) {
    g_print ("  none\n");
    return;
  }

  pads = gst_element_factory_get_static_pad_templates (factory);
  while (pads) {
    padtemplate = pads->data;
    pads = g_list_next (pads);

    if (padtemplate->direction == GST_PAD_SRC)
      g_print ("  SRC template: '%s'\n", padtemplate->name_template);
    else if (padtemplate->direction == GST_PAD_SINK)
      g_print ("  SINK template: '%s'\n", padtemplate->name_template);
    else
      g_print ("  UNKNOWN!!! template: '%s'\n", padtemplate->name_template);

    if (padtemplate->presence == GST_PAD_ALWAYS)
      g_print ("    Availability: Always\n");
    else if (padtemplate->presence == GST_PAD_SOMETIMES)
      g_print ("    Availability: Sometimes\n");
    else if (padtemplate->presence == GST_PAD_REQUEST) {
      g_print ("    Availability: On request\n");
    } else
      g_print ("    Availability: UNKNOWN!!!\n");

    if (padtemplate->static_caps.string) {
      GstCaps *caps;
      g_print ("    Capabilities:\n");
      caps = gst_static_caps_get (&padtemplate->static_caps);
      print_caps (caps, "      ");
      gst_caps_unref (caps);

    }

    g_print ("\n");
  }
}

/* Shows the CURRENT capabilities of the requested pad in the given element */
static void print_pad_capabilities (GstElement *element, gchar *pad_name) {
  GstPad *pad = NULL;
  GstCaps *caps = NULL;

  /* Retrieve pad */
  pad = gst_element_get_static_pad (element, pad_name);
  if (!pad) {
    g_printerr ("Could not retrieve pad '%s'\n", pad_name);
    return;
  }

  /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */
  caps = gst_pad_get_current_caps (pad);
  if (!caps)
    caps = gst_pad_query_caps (pad, NULL);

  /* Print and free */
  g_print ("Caps for the %s pad:\n", pad_name);
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  gst_object_unref (pad);
}

int main(int argc, char *argv[]) {
  GstElement *pipeline, *source, *sink;
  GstElementFactory *source_factory, *sink_factory;
  GstBus *bus;
  GstMessage *msg;
  GstStateChangeReturn ret;
  gboolean terminate = FALSE;

  /* Initialize GStreamer */
  gst_init (&argc, &argv);

  /* Create the element factories */
  source_factory = gst_element_factory_find ("audiotestsrc");
  sink_factory = gst_element_factory_find ("autoaudiosink");
  if (!source_factory || !sink_factory) {
    g_printerr ("Not all element factories could be created.\n");
    return -1;
  }

  /* Print information about the pad templates of these factories */
  print_pad_templates_information (source_factory);
  print_pad_templates_information (sink_factory);

  /* Ask the factories to instantiate actual elements */
  source = gst_element_factory_create (source_factory, "source");
  sink = gst_element_factory_create (sink_factory, "sink");

  /* Create the empty pipeline */
  pipeline = gst_pipeline_new ("test-pipeline");

  if (!pipeline || !source || !sink) {
    g_printerr ("Not all elements could be created.\n");
    return -1;
  }

  /* Build the pipeline */
  gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
  if (gst_element_link (source, sink) != TRUE) {
    g_printerr ("Elements could not be linked.\n");
    gst_object_unref (pipeline);
    return -1;
  }

  /* Print initial negotiated caps (in NULL state) */
  g_print ("In NULL state:\n");
  print_pad_capabilities (sink, "sink");

  /* Start playing */
  ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
  if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr ("Unable to set the pipeline to the playing state (check the bus for error messages).\n");
  }

  /* Wait until error, EOS or State Change */
  bus = gst_element_get_bus (pipeline);
  do {
    msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, GST_MESSAGE_ERROR | GST_MESSAGE_EOS |
        GST_MESSAGE_STATE_CHANGED);

    /* Parse message */
    if (msg != NULL) {
      GError *err;
      gchar *debug_info;

      switch (GST_MESSAGE_TYPE (msg)) {
        case GST_MESSAGE_ERROR:
          gst_message_parse_error (msg, &err, &debug_info);
          g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message);
          g_printerr ("Debugging information: %s\n", debug_info ? debug_info : "none");
          g_clear_error (&err);
          g_free (debug_info);
          terminate = TRUE;
          break;
        case GST_MESSAGE_EOS:
          g_print ("End-Of-Stream reached.\n");
          terminate = TRUE;
          break;
        case GST_MESSAGE_STATE_CHANGED:
          /* We are only interested in state-changed messages from the pipeline */
          if (GST_MESSAGE_SRC (msg) == GST_OBJECT (pipeline)) {
            GstState old_state, new_state, pending_state;
            gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
            g_print ("\nPipeline state changed from %s to %s:\n",
                gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
            /* Print the current capabilities of the sink element */
            print_pad_capabilities (sink, "sink");
          }
          break;
        default:
          /* We should not reach here because we only asked for ERRORs, EOS and STATE_CHANGED */
          g_printerr ("Unexpected message received.\n");
          break;
      }
      gst_message_unref (msg);
    }
  } while (!terminate);

  /* Free resources */
  gst_object_unref (bus);
  gst_element_set_state (pipeline, GST_STATE_NULL);
  gst_object_unref (pipeline);
  gst_object_unref (source_factory);
  gst_object_unref (sink_factory);
  return 0;
}

Walkthrough

print_field, print_caps and print_pad_templates 三个函数仅仅以友好的格式将caps打印出来，详细GstCaps的结构可以参考 GStreamer Documentation中关于 Pad Caps的部分。

The print_field, print_caps and print_pad_templates simply display, in a human-friendly format, the capabilities structures. If you want to learn about the internal organization of the GstCaps structure, read the GStreamer Documentation regarding(关于) Pad Caps.

/* Shows the CURRENT capabilities of the requested pad in the given element */
static void print_pad_capabilities (GstElement *element, gchar *pad_name) {
  GstPad *pad = NULL;
  GstCaps *caps = NULL;

  /* Retrieve pad */
  pad = gst_element_get_static_pad (element, pad_name);
  if (!pad) {
    g_printerr ("Could not retrieve pad '%s'\n", pad_name);
    return;
  }

  /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */
  caps = gst_pad_get_current_caps (pad);
  if (!caps)
    caps = gst_pad_query_caps (pad, NULL);

  /* Print and free */
  g_print ("Caps for the %s pad:\n", pad_name);
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  gst_object_unref (pad);
}

gst_element_get_static_pad() 返回指定element的pad的能力集。gst_pad_get_current_caps()返回当前能力集(是否是最后结果取决于协商是否完成)，gst_pad_get_current_caps()可能为空，此时我们可以通过gst_pad_query_caps() 来获取当前可接受的能力。在NULL状态，可接受的能力集就是模板的能力集，随着状态的变化(NULL->READY->PAUSE->PLAYING)，能力集会发生变化(过程可能依赖硬件)。

gst_element_get_static_pad() retrieves the named Pad from the given element. This Pad is static because it is always present in the element. To know more about Pad availability read the GStreamer documentation about Pads.

Then we call gst_pad_get_current_caps() to retrieve the Pad's current Capabilities, which can be fixed or not, depending on the state of the negotiation process. They could even be non-existent, in which case, we call gst_pad_query_caps() to retrieve the currently acceptable Pad Capabilities. The currently acceptable Caps will be the Pad Template's Caps in the NULL state, but might change in later states, as the actual hardware Capabilities might be queried.

We then print these Capabilities.

/* Create the element factories */
source_factory = gst_element_factory_find ("audiotestsrc");
sink_factory = gst_element_factory_find ("autoaudiosink");
if (!source_factory || !sink_factory) {
  g_printerr ("Not all element factories could be created.\n");
  return -1;
}

/* Print information about the pad templates of these factories */
print_pad_templates_information (source_factory);
print_pad_templates_information (sink_factory);

/* Ask the factories to instantiate actual elements */
source = gst_element_factory_create (source_factory, "source");
sink = gst_element_factory_create (sink_factory, "sink");

之前我们直接调用 gst_element_factory_make() 创建element，这里我们将此函数拆分开来，先通过函数gst_element_factory_find()获取模板，然后用获取到的模板创建element。模板获取以后，对应的pad cap就已经存在了。

In the previous tutorials we created the elements directly using gst_element_factory_make() and skipped talking about factories, but we will do now. A GstElementFactory is in charge of instantiating a particular type of element, identified by its factory name.

You can use gst_element_factory_find() to create a factory of type “videotestsrc”, and then use it to instantiate multiple “videotestsrc” elements using gst_element_factory_create(). gst_element_factory_make() is really a shortcut for gst_element_factory_find()+ gst_element_factory_create().

The Pad Templates can already be accessed through the factories, so they are printed as soon as the factories are created.

We skip the pipeline creation and start, and go to the State-Changed message handling:

case GST_MESSAGE_STATE_CHANGED:
  /* We are only interested in state-changed messages from the pipeline */
  if (GST_MESSAGE_SRC (msg) == GST_OBJECT (pipeline)) {
    GstState old_state, new_state, pending_state;
    gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
    g_print ("\nPipeline state changed from %s to %s:\n",
        gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
    /* Print the current capabilities of the sink element */
    print_pad_capabilities (sink, "sink");
  }
  break;

从状态的变化打印过程，我们可以看到二个element的pad caps在不断变化，知道固定下来。固定以后就没有中括号和大括号这样的可选项，而是固定下来的某个值(我们是否可以干预能力集协商过程呢？？？这里留一个疑问)。

This simply prints the current Pad Caps every time the state of the pipeline changes. You should see, in the output, how the initial caps (the Pad Template's Caps) are progressively refined until they are completely fixed (they contain a single type with no ranges).

Conclusion

This tutorial has shown:

• What are Pad Capabilities and Pad Template Capabilities.

• How to retrieve them with gst_pad_get_current_caps() or gst_pad_query_caps().

• That they have different meaning depending on the state of the pipeline (initially they indicate all the possible Capabilities, later they indicate(指明) the currently negotiated Caps for the Pad).

• That Pad Caps are important to know beforehand if two elements can be linked together.

• That Pad Caps can be found using the gst-inspect-1.0 tool described in Basic tutorial 10: GStreamer tools.

Next tutorial shows how data can be manually injected into and extracted from the GStreamer pipeline.

Remember that attached to this page you should find the complete source code of the tutorial and any accessory files needed to build it. It has been a pleasure having you here, and see you soon!