introduce
Through a complete example, add Prometheus monitoring middleware to microservices based on gorilla/mux framework.
What is a Prometheus monitoring interceptor/middleware?
The monitoring interceptor records Prometheus Metrics for each API request.
We will use rk-boot to start the gorilla/mux microservice .
Please visit the following address for the full tutorial: https://github.com/rookie-ninja/rk-mux
Install
go get github.com/rookie-ninja/rk-boot/mux
quick start
1. Create boot.yaml
The boot.yaml file describes the original information of Mux framework startup, rk-boot starts GoFrame by reading boot.yaml.
To verify, we enabled the following options:
- commonService : commonService contains a series of common APIs. Details
- prom : Prometheus (Prometheus) client.
- prometheus middleware : Start the prometheus middleware.
---
mux:
- name: greeter # Required
port: 8080 # Required
enabled: true # Required
prom:
enabled: true # Optional, default: false
commonService:
enabled: true # Optional, default: false
interceptors:
metricsProm:
enabled: true # Optional, default: false
2. Create main.go
Added /v1/greeter API.
// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main
import (
"context"
"fmt"
"github.com/rookie-ninja/rk-boot"
"github.com/rookie-ninja/rk-boot/mux"
"github.com/rookie-ninja/rk-mux/interceptor"
"net/http"
)
func main() {
// Create a new boot instance.
boot := rkboot.NewBoot()
// Register handler
entry := rkbootmux.GetMuxEntry("greeter")
entry.Router.NewRoute().Methods(http.MethodGet).Path("/v1/greeter").HandlerFunc(Greeter)
// Bootstrap
boot.Bootstrap(context.TODO())
boot.WaitForShutdownSig(context.TODO())
}
func Greeter(writer http.ResponseWriter, request *http.Request) {
rkmuxinter.WriteJson(writer, http.StatusOK, &GreeterResponse{
Message: fmt.Sprintf("Hello %s!", request.URL.Query().Get("name")),
})
}
// Response.
type GreeterResponse struct {
Message string
}
3. Folder structure
$ tree
.
├── boot.yaml
├── go.mod
├── go.sum
└── main.go
0 directories, 4 files
4. Start main.go
$ go run main.go
2022-02-09T15:35:02.181+0800 INFO boot/mux_entry.go:643 Bootstrap muxEntry {"eventId": "a35a0331-4311-4057-a399-526c76f79ca9", "entryName": "greeter", "entryType": "Mux"}
------------------------------------------------------------------------
endTime=2022-02-09T15:35:02.181722+08:00
startTime=2022-02-09T15:35:02.181528+08:00
elapsedNano=193785
timezone=CST
ids={"eventId":"a35a0331-4311-4057-a399-526c76f79ca9"}
app={"appName":"rk","appVersion":"","entryName":"greeter","entryType":"Mux"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"192.168.1.102","os":"darwin","realm":"*","region":"*"}
payloads={"commonServiceEnabled":true,"commonServicePathPrefix":"/rk/v1/","muxPort":8080,"promEnabled":true,"promPath":"/metrics","promPort":8080}
counters={}
pairs={}
timing={}
remoteAddr=localhost
operation=Bootstrap
resCode=OK
eventStatus=Ended
EOE
5. Verify
- Send a request to the /rk/v1/healthy API in CommonService.
$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}
- Send a request to the /v1/greeter API.
$ curl -X GET "localhost:8080/v1/greeter?name=rk-dev"
{"Message":"Hello rk-dev!"}
Access the Prometheus client: http://localhost:8080/metrics
Visual monitoring
We've started prometheus monitoring in the local process, and all that's left is how to view the monitoring in a [nice] web page.
There are many tools on the market, but we choose the [simple], [popular], and [free] methods, that is, Prometheus + Grafana.
Architecture diagram
Let's take a look at what the whole process looks like.
In fact, the principle is very simple, it is to [hijack] API requests, and record [time], [error code] and other information. After that, let the Prometheus service actively pull data from the [created service]. Finally, let the Grafana service pull data from Prometheus and display the data table.
quick start
1. Create prometheus.yml
Let's first create the prometheus.yml configuration file so that the prometheus service can pull data from localhost:8080/metrics.
In the configuration below, we do not specify /metrics, because prometheus uses /metrics to pull data by default.
Notice! We set targets to host.docker.internal:8080 instead of localhost:8080, this is because prometheus is in the container and our service is in the local.
This is a convenient way to access the local machine's port from within a container. explain
global:
scrape_interval: 1s # Make scrape interval to 1s for testing.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
- job_name: 'rk-demo'
scrape_interval: 1s
static_configs:
- targets: ['host.docker.internal:8080']
2. Start Prometheus
We use docker to start.
Prometheus uses port 9090 by default.
$ docker run -p 9090:9090 -v /<your path>/rk-demo/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
3. Verify Prometheus
Please follow the above [Verification], start main.go, and send a /rk/v1/healthy request.
Then, let's take a look at the data in the prometheus service.
Visit: localhost:9090, and search for rk_greeter_resCode, we can see that the data is already in prometheus.
Visit: localhost:9090/targets, we can see that prometheus has pulled data every second.
4. Start Grafana
Grafana uses port 3000 by default.
$ docker run -p 3000:3000 --name grafana grafana/grafana
Access: localhost:3000
At this time, grafana will let you log in. The default username and password are as follows.
Username: admin Password: admin
5. Add Prometheus data source in Grafana
Grafana is just a web UI tool, in order to see the data report, we tell Grafana where to look for Prometheus.
Select Prometheus as the data source.
Fill in the Prometheus address, the same as above, because Grafana runs in Docker, so we don't use localhost:9090, but host.docker.internal:9090.
6. Import Dashboard
We can edit the Grafana Dashboard by ourselves, but this is not an easy task. For services started with rk-boot, we provide the default [free] Grafana Dashboard template.
Note that the Dashboard imported here only matches [services created according to the above logic].
Why? Because rk-boot will use rk_<Entry name>_xxx as the metrics name of prometheus by default.
Move to Dashboard import page
Import Dashboard No. 15111, defined at: https://grafana.com/grafana/dashboards/15111
Specify the Prometheus data source, which is the Prometheus we configured above.
start monitoring
Notice! If the number of requests is too small, it will be displayed as 0, please send several more requests.
concept
We can already get monitoring data from Grafana, now let's look at the middleware in rk-boot, what type of monitoring data is added.
The monitoring interceptor will record the following monitoring by default.
Monitoring item | type of data | Details |
---|---|---|
elapsedNano | Summary | RPC time consuming |
resCode | Counter | Counters based on RPC return codes |
errors | Counter | RPC error based counters |
The above three monitoring items have the following labels.
Label | Details |
---|---|
entryName | Mux entry name |
entryType | Mux entry type |
realm | Environment variables: REALM, eg: rk |
region | Environment variables: REGION, eg: beijing |
the | Environment variables: AZ, eg: beijing-1 |
domain | Environment variables: DOMAIN, eg: prod |
instance | local hostname |
appVersion | Get from AppInfoEntry |
appName | Get from AppInfoEntry |
restMethod | Http method. eg: GET |
restPath | Http path. eg: /rk/v1/healthy |
type | Service type. eg: Mux |
resCode | Return code, eg: OK |