Linux audio collection and pitfalls encountered in localized platforms (2)

Linux audio collection and pitfalls encountered in localized platforms (2)

ALSA collection is a dead end, so we can only try other methods. Here, through the interface of PulseAudio, we have successfully realized the function of collecting microphone and system sound on the domestic platform.

linux PulseAudio audio collection

First of all, the difference between PulseAudio and ALSA is that ALSA is at the kernel level, while PulseAudio is a user-level service and is used as a Sound Server to manage various audio inputs and outputs of the application, the same as ALSA , most linux distributions have PulseAudio installed by default. Galaxy Kirin, our domestically produced chip platform, is naturally no exception. The structure diagram of PulseAudio looks like this:

As you can see, PulseAudio, as a service, is located on the upper layer of ALSA. It allows multiple applications to call PulseAudio at the same time and uses it as the audio mixer internally. This can avoid the program appearing in different hardware environments due to the exclusivity of ALSA. The situation cannot be used normally. The calling relationship between the application and PulseAudio is as follows:

Normally, the system does not pre-install the PulseAudio development package. At this time, we need to install it so that we can call the interface in the code.

sudo apt-get install libpulse-dev

PulseAudio audio collection is obviously much more complicated than ALSA. Each application is considered to be a PulseAudio client, connecting to the system's PulseAudio service, and needs to maintain a thread as a circular queue for data transfer. Here is a list of several functions used by race:

#include <pulse/pulseaudio.h>

/***
 申请一个包含线程的事件循环
*/
pa_threaded_mainloop* pa_threaded_mainloop_new();

/***
 开启事件循环
 @return: 0表示成功，小于0表示错误码
*/
int pa_threaded_mainloop_start(pa_threaded_mainloop* m);

/***
 终止事件循环，在调用此函数前，必须确保事件循环已经解锁
*/
void pa_threaded_mainloop_stop(pa_threaded_mainloop* m);

/***
 阻塞并等待事件循环中消息被触发，注意，该函数返回并不一定是因为调用了pa_threaded_mainloop_signal()
 需要甄别这一点
*/
void pa_threaded_mainloop_wait(pa_threaded_mainloop* m);

/***
 触发消息
*/
void pa_threaded_mainloop_signal(pa_threaded_mainloop* m, int wait_for_accept);

#include <pulse/pulseaudio.h>

/***
 创建PulseAudio连接上下文
*/
pa_context* pa_context_new(pa_mainloop_api *mainloop, const char *name);

/***
 将context连接到指定的PulseAudio服务，如果server为NULL，则连接到系统默认服务。
 @return: 小于0表示错误
*/
int pa_context_connect(pa_context *c, const char *server, pa_context_flags_t flags, const pa_spawn_api *api);

/***
 终止事件循环，在调用此函数前，必须确保事件循环已经解锁
*/
void pa_context_disconnect(pa_context* c);

/***
 引用计数减1
*/
void pa_context_unref(pa_context* c);

/***
 返回当前上下文状态
*/
pa_context_state_t pa_context_get_state(const pa_context* c);

#include <pulse/pulseaudio.h>

/***
 在当前PulseAudio连接上，创建一个stream，用于输入或输出音频数据
*/
pa_stream* pa_stream_new(pa_context *c, const char *name, const pa_sample_spec *ss, const pa_channel_map *map);

/***
 将context连接到指定的PulseAudio服务，如果server为NULL，则连接到系统默认服务。
 @return: 小于0表示错误
*/
int pa_stream_connect_record(pa_context *c, const char *server, pa_context_flags_t flags, const pa_spawn_api *api);

/***
 从缓冲区中读取下一个采集的音频片段
*/
int pa_stream_peek(pa_stream *p, const void **data, size_t *nbytes);

/***
 放弃当前输入(采集)的音频片段
*/
void pa_stream_drop(pa_stream* s);

/***
 关闭输入输出流
*/
void pa_stream_disconnect(pa_stream* s);

/***
 引用计数减1
*/
void pa_stream_unref(pa_stream* s);

/***
 返回当前stream状态
*/
pa_context_state_t pa_stream_get_state(const pa_stream* s);

Here is a simple example to demonstrate how to call

1. Create an event loop, connect to the PulseAudio server, create a stream and set parameters. In order to make it look more intuitive, I have deleted some incorrectly judged code here.

bool PulseAudioCapture::Start(Observer* ob)
{
    
    
    observer_ = ob;

    SIMPLE_LOG("try open %s\n", device_name_.c_str());

    int ret = 0;
    const char* name = "HbsPulse";
    const char* stream_name = "HbsPulseStream";
    char* device = NULL;
    if (false == device_name_.empty())
    {
    
    
        device = (char*)device_name_.c_str();
    }

    const struct pa_sample_spec *pss = nullptr;

    pa_sample_format_t sam_fmt = AV_NE(PA_SAMPLE_S16BE, PA_SAMPLE_S16LE);
    const pa_sample_spec ss = {
    
     sam_fmt, sample_rate_, channel_count_ };

    pa_buffer_attr attr = {
    
     (uint32_t)-1 };
    pa_channel_map cmap;
    const pa_buffer_attr *queried_attr = nullptr;
    int stream_flag = 0;

    pa_channel_map_init_extend(&cmap, channel_count_, PA_CHANNEL_MAP_WAVEEX);

    mainloop_ = pa_threaded_mainloop_new();

    context_ = pa_context_new(pa_threaded_mainloop_get_api(mainloop_), name);

    pa_context_set_state_callback(context_, context_state_cb, this);

    pa_context_connect(context_, pulse_server_, /*0*/PA_CONTEXT_NOFLAGS, NULL);

    pa_threaded_mainloop_lock(mainloop_);

    pa_threaded_mainloop_start(mainloop_);

    for (;;)
    {
    
    
        pa_context_state_t state = pa_context_get_state(context_);

        if (state == PA_CONTEXT_READY)
            break;

        if (!PA_CONTEXT_IS_GOOD(state))
        {
    
    
            int ec = pa_context_errno(context_);
            SIMPLE_LOG("pulse context state bad: %d, err: %d\n", state, ec);

            goto unlock_and_fail;
        }

        /* Wait until the context is ready */
        pa_threaded_mainloop_wait(mainloop_);
    }

    SIMPLE_LOG("pulse context ready!\n");

    stream_ = pa_stream_new(context_, stream_name, &ss, &cmap);

    pa_stream_set_state_callback(stream_, stream_state_cb, this);
    pa_stream_set_read_callback(stream_, stream_read_cb, this);
    pa_stream_set_write_callback(stream_, stream_write_cb, this);
    pa_stream_set_latency_update_callback(stream_, stream_latency_update_cb, this);

    ret = pa_stream_connect_record(stream_, device, &attr,
        PA_STREAM_ADJUST_LATENCY|PA_STREAM_AUTO_TIMING_UPDATE);

    for (;;)
    {
    
    
        pa_stream_state_t state = pa_stream_get_state(stream_);

        if (state == PA_STREAM_READY)
            break;

        if (!PA_STREAM_IS_GOOD(state))
        {
    
    
            int ec = pa_context_errno(context_);
            SIMPLE_LOG("pulse stream state bad: %d, err: %d\n", state, ec);

            goto unlock_and_fail;
        }

        /* Wait until the stream is ready */
        pa_threaded_mainloop_wait(mainloop_);
    }

    pa_threaded_mainloop_unlock(mainloop_);

    SIMPLE_LOG("pulse audio start ok, fragsize: %d, framesize: %d\n", fragment_size_, pa_frame_size_);

    ThreadStart();

    return true;

unlock_and_fail:
    pa_threaded_mainloop_unlock(mainloop_);

    ClosePulse();
    return false;
}

2. Read audio data

bool PulseAudioCapture::ReadData()
{
    
    
    int ret;
    size_t read_length;
    const void *read_data = NULL;

    pa_usec_t latency;
    int negative;
    ptrdiff_t pos = 0;

    pa_threaded_mainloop_lock(mainloop_);

    if (IsPulseDead())
    {
    
    
        SIMPLE_LOG("pulse is dead\n");
        goto unlock_and_fail;
    }

    while (pos < fragment_size_)
    {
    
    
        int r = pa_stream_peek(stream_, &read_data, &read_length);
        if (r != 0)
        {
    
    
            SIMPLE_LOG("pa_stream_peek: %d\n", r);
            goto unlock_and_fail;
        }

        if (read_length <= 0)
        {
    
    
            pa_threaded_mainloop_wait(mainloop_);
            if (IsPulseDead())
            {
    
    
                SIMPLE_LOG("pulse is dead\n");
                goto unlock_and_fail;
            }
        }
        else if (!read_data)
        {
    
    
            /* There's a hole in the stream, skip it. We could generate
            * silence, but that wouldn't work for compressed streams. */
            r = pa_stream_drop(stream_);
            if (r != 0)
            {
    
    
                SIMPLE_LOG("null data, pa_stream_drop: %d\n", r);
                goto unlock_and_fail;
            }
        }
        else 
        {
    
    
            if (!pos)
            {
    
    
                if (pcm_buf_.empty())
                {
    
    
                    pcm_buf_.resize(fragment_size_);
                }

                //pcm_dts_ = av_gettime();
                pa_operation_unref(pa_stream_update_timing_info(stream_, NULL, NULL));

                if (pa_stream_get_latency(stream_, &latency, &negative) >= 0)
                {
    
    
                    if (negative)
                    {
    
    
                        pcm_dts_ += latency;
                    }
                    else
                        pcm_dts_ -= latency;
                }
                else
                {
    
    
                    SIMPLE_LOG("pa_stream_get_latency() failed\n");
                }
            }

            if (pcm_buf_.size() - pos < read_length)
            {
    
    
                if (pos)
                    break;
                pa_stream_drop(stream_);
                /* Oversized fragment??? */
                SIMPLE_LOG("Oversized fragment\n");
                goto unlock_and_fail;
            }

            memcpy(pcm_buf_.data() + pos, read_data, read_length);
            pos += read_length;
            pa_stream_drop(stream_);
        }
    }

SIMPLE_LOG("read pos: %d\n", pos);

    pa_threaded_mainloop_unlock(mainloop_);

    return true;

unlock_and_fail:
    pa_threaded_mainloop_unlock(mainloop_);
    return false;
}

When selecting an audio device, the audio device name must be queried through the PulseAudio related interface. For audio collection devices, you can call the pa_context_get_source_info_list() function. After experiments, PulseAudio was used to collect audio, and the function of collecting microphone and system sound on the Kirin system of the domestic platform was successfully realized, avoiding various troubles that occurred in the previous use of ALSA code in a multi-sound card environment.

In addition, it should be noted that the size of the data collected through PulseAudio may not be required for encoding, and data buffering is also required.

Please add the author hbstream for cooperation. (Please indicate the author and source when reprinting)

---