Teach you to learn Dapr - 9. Observability

content

introduce

Monitor applications through Tracing, Metrics, Logs, and Health.

Distributed tracing

Dapr uses the Zipkinprotocol for distributed tracing and metrics collection. Due to the ubiquity of the Zipkin protocol, many backends are available out of the box, such as Stackdriver, Zipkin, New Relic, etc. Combined OpenTelemetry Collector, Dapr can export traces to many other backends, including but not limited to Azure Monitor, Datadog, Instana, Jaeger, and SignalFX.

Tracing Design

Dapr adds an HTTP/gRPC middleware to the Dapr Sidecar. Middleware intercepts all Dapr and application traffic and automatically injects correlation IDs to track distributed transactions. This design has several benefits:

No code inspection required. Automatically track all traffic with configurable tracking levels.
Consistent tracking behavior across microservices. Tracking is configured and managed on the Dapr sidecar, so it is consistent across services made by different teams and possibly written in different programming languages.
Configurable and extensible. Using the Zipkin API and OpenTelemetry Collector, Dapr tracing can be configured to work with popular tracing backends, including custom ones that customers may have.
You can define and enable multiple exporters at the same time.

W3C Association ID

Dapr uses the standard W3C trace context header. For HTTP requests, Dapr uses traceparentthe header. For gRPC requests, Dapr uses grpc-trace-binthe header. When a request arrives without a tracking ID, Dapr creates a new one. Otherwise, it passes the tracking ID down the call chain.

configure

Dapr uses probability sampling. The sampling rate defines the probability of sampling the trace span and can have a value between 0 and 1 (inclusive). The default sampling rate is 0.0001 (that is, 1 out of 10,000 spans is sampled).

To change the default tracking behavior, use a configuration file. For example, the following configuration object changes the sampling rate to 1 (i.e. every span is sampled) and uses the Zipkin protocol to send traces to the Zipkin server at http://zipkin.default.svc.cluster.local

yaml file path:%UserProfile%\.dapr\config.yaml

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    zipkin:
      endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"

注: Change the sample rate to 0 for full 禁用tracking.

W3C trace context

Dapr uses the W3C tracing context for distributed tracing of service calls and pub/sub messages. Dapr does all the heavy lifting of generating and propagating trace context information, and only in rare cases do you need to propagate or create a trace context.

The W3C trace context has the following advantages:

Provides a unique identifier for individual traces and requests, allowing trace data from multiple providers to be linked together.
Provides an agreed-upon mechanism for forwarding vendor-specific trace data and avoids trace disruption when multiple tools are involved in a single transaction.
Provides industry standards that middlemen, platforms, and hardware providers can support.

There are two situations where you need to know how to use tracking:

Dapr generates and propagates trace contexts between services.
Dapr generates a trace context and you need to propagate the trace context to another service, or you generate a trace context and Dapr propagates the trace context to the service.

Dapr generates and propagates trace context between services

In some cases, Dapr will do all the work for you. You don't need to create and propagate any tracking headers. Dapr is responsible for creating all tracking headers and propagating them. Let's understand the scenario with an example;

A single service call (Service A -> Service B)

Dapr generates trace headers in service A, which propagate from service A to service B.
Multiple sequential service calls (Service A -> Service B -> Service C)

Dapr generates trace headers at the start of a request in service A, which are propagated from service A -> service B -> service C, etc. to further Dapr-enabled services.
The request is from an external endpoint (e.g. from a gateway service to Dapr-enabled service A)

Dapr Sidecar Health Check

Dapr provides a way /healthzto determine its health using an HTTP endpoint. With this endpoint, the health of a Dapr process or sidecar can be probed to determine its readiness and activity.

The Dapr health endpoint is automatically configured for you when you deploy Dapr to a managed platform such as Kubernetes. You don't need to do any configuration.

Health endpoint: Integration with Kubernetes

Kubernetes uses readiness and liveness probes to determine the health of containers.

The kubelet uses liveness probes to know when to restart containers. For example, liveness probes can catch deadlocks, where an application is running but cannot make progress. Restarting the container in this state helps make the application more usable, albeit with flaws.

The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A pod is considered ready when all its containers are ready. One use of this ready signal is to control which Pods are used as backends for Kubernetes services. When a Pod is not ready, it will be removed from the Kubernetes Service Load Balancer.

When integrated with Kubernetes, the Dapr sidecar is injected with a Kubernetes probe configuration telling it to use the Dapr healthz endpoint. This is done by the Sidecar Injector system service. The integration with the kubelet is shown in the figure below.

How to configure liveness probes in Kubernetes

In the pod configuration file, the liveness probe is added in the container specification section as follows:

livenessProbe:
     httpGet:
       path: /healthz
       port: 8080
     initialDelaySeconds: 3
     periodSeconds: 3

In the above example, the periodSeconds field specifies that the kubelet should perform liveness probes every 3 seconds. The initialDelaySeconds field tells the kubelet that it should wait 3 seconds before performing the first probe.

注: Any code greater than or equal to 200 and less than 400 indicates success. Other codes indicate failure.

How to configure readiness probes in Kubernetes

The configuration of ready probes is similar to that of liveness probes. The only difference is that you use the readinessProbe field instead of the livenessProbe field.

readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3

How to configure Dapr Sidecar health endpoint with Kubernetes

This configuration is done automatically by the Sidecar Injector service. Dapr has its HTTP health endpint on port 3500 /v1.0/healthz, which can be used with Kubernetes for readiness and liveness detection. When injecting the Dapr sidecar, the readiness and liveness probes are configured with the following values in the pod configuration file.

livenessProbe:
      httpGet:
        path: v1.0/healthz
        port: 3500
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds : 5
      failureThreshold : 3
readinessProbe:
      httpGet:
        path: v1.0/healthz
        port: 3500
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds : 5
      failureThreshold: 3

Using Observability in .Net

Create Assignment.Server

Create the 类库project, add Dapr.AspNetCore, OpenTelemetry, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http, OpenTelemetry.Extensions.Hostingand OpenTelemetry.Exporter.ZipkinNuGet package references, and finally modify the program port to 5000.

!!!Note: The version is very important, NuGet should be opened 包含预发行版and the specified version should be used

OpenTelemetry-1.2.0-beta1

OpenTelemetry.Instrumentation.AspNetCore-1.0.0-rc8

OpenTelemetry.Instrumentation.Http-1.0.0-rc8

OpenTelemetry.Exporter.Zipkin-1.2.0-beta1

OpenTelemetry.Extensions.Hosting-1.0.0-rc8

Modify program.cs

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetryTracing((tracerProviderBuilder) =>
    tracerProviderBuilder
        .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("testobservability"))
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddZipkinExporter(zipkinOptions =>
        {
            zipkinOptions.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
        }
    )
);
var app = builder.Build();

app.Map("/Amazing", async (HttpContext context) =>
{
    if (context.Request.Headers.TryGetValue("traceparent", out var traceparent))
    {
        Console.WriteLine($"TraceParent: {traceparent}");
    }
    if (context.Request.Headers.TryGetValue("tracestate", out var tracestate))
    {
        Console.WriteLine($"TraceState: {tracestate}");
    }

    System.Diagnostics.Activity.Current?.SetParentId(traceparent.ToString());
    _ = await new HttpClient().GetStringAsync("https://www.baidu.com");
    Console.WriteLine($"Invoke succeed: traceID:{traceparent}");
});

app.Run();

You can see that we directly demonstrated a fun usage, that is, to enable .Net OpenTelemetry, and then modify Diagnostics.Activityit ParentIdso that the current Tracing is consistent with the TraceId from Dapr Sidecar.

Run Assignment.Server

Use Dapr CLI to start, first use the command line tool to jump to the directory dapr-study-room\Assignment07\Assignment.Server, and then execute the following command

dapr run --app-id testobservability --app-port 5000 --dapr-http-port 3500 --dapr-grpc-port 50001 dotnet run

Use Dapr CLI to issue a command to see

dapr invoke --app-id testobservability --method /Amazing

Open Zipkin, address: http://localhost:9411/, Let's take a look at Zipkin's Tracing, not only Dapr Sidecar's request records come in, but also bundled with HttpClient's, yes, the fun is here.

In addition to tracking HttpClient, EF Core, etc. are integrated.

As for the integration of Metrics and Logs, it is also very simple, and needs to be matched with different backends such as Prometheus, Fluentd, etc. You can even connect to the cloud services of some cloud vendors by customizing the Exporter.

Source code of this chapter

Assignment09

https://github.com/doddgu/dapr-study-room

We are in action, new framework, new ecology

Our goals are 自由的, 易用的, 可塑性强的, 功能丰富的, 健壮的.

So we draw on the design concept of Building blocks and are making a new framework MASA Framework. What are its characteristics?

Natively supports Dapr and allows replacing Dapr with traditional communication methods
Unlimited architecture, monolithic applications, SOA, and microservices are supported
Support .Net native framework, reduce learning burden, and insist not to create new wheels except for concepts that must be introduced in specific fields
Rich ecological support, in addition to the framework, there are a series of products such as component library, authority center, configuration center, troubleshooting center, and alarm center
Unit test coverage of core code base is 90%+
Open source, free, community driven
what else? We are waiting for you, let's discuss together

After several months of production project practice, the POC has been completed, and the previous accumulation is currently being refactored into a new open source project

At present, the source code has begun to be synchronized to Github (the documentation site is under planning and will be gradually improved):

QQ group: 7424099

WeChat group: add technology operation WeChat (MasaStackTechOps), note your intention, invite to join the group