kubernetes上部署Fluentd+Elasticsearch+kibana日志收集系统
1 工具介绍
Fluentd:用于收集、处理、传输日志数据。
Elasticsearch:用于实时查询和解析数据。
Kibana:用于数据可视化。
2 工作流程
Fluentd收集日志,将日志过滤处理后输出到Elasticsearch中,Elasticsearch的watcher插件将异常信息通过邮件发送给相关负责人。Kibana配置上ES服务接口,通过Kibana服务访问即可。3 安装
三个工具全部运行在kubernettes上。
fluentd.yaml配置文件
apiVersion: v1
kind: Pod
metadata:
name: fluentd
spec:
containers:
- name: fluentd-es
image: index.tenxcloud.com/google_containers/fluentd-elasticsearch:1.9
imagePullPolicy: Never
env:
- name: FLUENTD_ARGS
value: -qq
volumeMounts:
- name: containers
mountPath: /var/lib/docker/containers
- name: varlog
mountPath: /varlog
- name: podlog
mountPath: /var/lib/kubelet/pods
volumes:
- name: containers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: podlog
hostPath:
path: /var/lib/kubelet/pods
kind: Pod
metadata:
name: fluentd
spec:
containers:
- name: fluentd-es
image: index.tenxcloud.com/google_containers/fluentd-elasticsearch:1.9
imagePullPolicy: Never
env:
- name: FLUENTD_ARGS
value: -qq
volumeMounts:
- name: containers
mountPath: /var/lib/docker/containers
- name: varlog
mountPath: /varlog
- name: podlog
mountPath: /var/lib/kubelet/pods
volumes:
- name: containers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: podlog
hostPath:
path: /var/lib/kubelet/pods
Elasticsearch-rc.yaml配置文件
apiVersion: v1
kind: ReplicationController
metadata:
name: elasticsearch
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
name: elasticsearch-logging
template:
metadata:
name: es
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
containers:
- image: index.tenxcloud.com/google_containers/elasticsearch:1.4
imagePullPolicy: Never
name: elasticsearch-logging
ports:
- name: es-port
containerPort: 9200
- name: es-tran-port
containerPort: 9300
volumeMounts:
- name: es-persistent-storage
mountPath: /data
volumes:
- name: es-persistent-storage
emptyDir: {}
kind: ReplicationController
metadata:
name: elasticsearch
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
name: elasticsearch-logging
template:
metadata:
name: es
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
containers:
- image: index.tenxcloud.com/google_containers/elasticsearch:1.4
imagePullPolicy: Never
name: elasticsearch-logging
ports:
- name: es-port
containerPort: 9200
- name: es-tran-port
containerPort: 9300
volumeMounts:
- name: es-persistent-storage
mountPath: /data
volumes:
- name: es-persistent-storage
emptyDir: {}
Elasticsearch-service.yaml配置文件
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-logging
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
type: NodePort
ports:
# the port that this service should serve on
- port: 9200
targetPort: 9200
nodePort: 30009
# - containerPort: 9200
selector:
name: elasticsearch-logging
metadata:
name: elasticsearch-logging
labels:
name: elasticsearch-logging
kubernetes.io/cluster-service: "true"
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
type: NodePort
ports:
# the port that this service should serve on
- port: 9200
targetPort: 9200
nodePort: 30009
# - containerPort: 9200
selector:
name: elasticsearch-logging
kibana-rc.yaml文件
apiVersion: v1
kind: ReplicationController
metadata:
name: kibana-logging
labels:
name: kibana-logging
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
name: kibana-logging
template:
metadata:
name: kibana
labels:
name: kibana-logging
kubernetes.io/cluster-service: "true"
spec:
containers:
- image: gcr.io/google_containers/kibana:1.3
name: kibana-logging
imagePullPolicy: Never
env:
- name: "ES_SCHEME"
value: "http"
- name: "ES_HOST"
value: "<your-ip>/api/v1/proxy/namespaces/default/services/elasticsearch-logging/"
ports:
- name: kibana-port
containerPort: 80
kind: ReplicationController
metadata:
name: kibana-logging
labels:
name: kibana-logging
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
name: kibana-logging
template:
metadata:
name: kibana
labels:
name: kibana-logging
kubernetes.io/cluster-service: "true"
spec:
containers:
- image: gcr.io/google_containers/kibana:1.3
name: kibana-logging
imagePullPolicy: Never
env:
- name: "ES_SCHEME"
value: "http"
- name: "ES_HOST"
value: "<your-ip>/api/v1/proxy/namespaces/default/services/elasticsearch-logging/"
ports:
- name: kibana-port
containerPort: 80
kibana-service.yaml配置文件
apiVersion: v1
kind: Service
metadata:
name: kibana-logging
labels:
name: kibana-logging
kubernetes.io/cluster-service: "true"
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
# type: LoadBalancer
ports:
# the port that this service should serve on
- port: 5601
targetPort: 80
# - containerPort: 9200
selector:
name: kibana-logging
kind: Service
metadata:
name: kibana-logging
labels:
name: kibana-logging
kubernetes.io/cluster-service: "true"
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
# type: LoadBalancer
ports:
# the port that this service should serve on
- port: 5601
targetPort: 80
# - containerPort: 9200
selector:
name: kibana-logging
http://blog.csdn.net/ptmozhu/article/details/52301174