aodh源码分析
1 介绍
aodh是从ceilometer拆分出来的告警组件，现在主要包括:evaluator服务，notifier服务，listener服务。
evaluator服务主要用于每隔一定时间进行告警的校验，如果校验有告警产生，则与notifier服务通信，进行告警通知
notifier服务接收来自evaluator服务的通信，将触发的告警以日志，http请求等形式进行告警分发
listener服务可接收来自第三方的告警内容并进行存储
2 Aodh的evaluator服务
2.1 服务启动部分在: aodh/evaluator/__init__.py的AlarmEvaluationService类的start函数，具体如下: def start(self):
super(AlarmEvaluationService, self).start()
self.partition_coordinator.start()
self.partition_coordinator.join_group(self.PARTITIONING_GROUP_NAME)
# allow time for coordination if necessary
delay_start = self.partition_coordinator.is_active()
if self.evaluators:
interval = self.conf.evaluation_interval
# 以定时任务形式每隔一定时间调用_evaluate_assigned_alarms方法
self.tg.add_timer(
interval,
self._evaluate_assigned_alarms,
initial_delay=interval if delay_start else None)
if self.partition_coordinator.is_active():
heartbeat_interval = min(self.conf.coordination.heartbeat,
self.conf.evaluation_interval / 4)
self.tg.add_timer(heartbeat_interval,
self.partition_coordinator.heartbeat)
# Add a dummy thread to have wait() working
self.tg.add_timer(604800, lambda: None)
# NOTE(r-mibu): The 'event' type alarms will be evaluated by the
# event-driven alarm evaluator, so this periodical evaluator skips
# those alarms.
all_alarms = self._storage_conn.get_alarms(enabled=True,
exclude=dict(type='event'))
all_alarms = list(all_alarms)
all_alarm_ids = [a.alarm_id for a in all_alarms]
selected = self.partition_coordinator.extract_my_subset(
self.PARTITIONING_GROUP_NAME, all_alarm_ids)
return list(filter(lambda a: a.alarm_id in selected, all_alarms))分析:
以定时任务形式每隔一定时间调用_evaluate_assigned_alarms方法
self.tg.add_timer(
interval,
self._evaluate_assigned_alarms,
initial_delay=interval if delay_start else None)2.2 _evaluate_assigned_alarms方法具体如下:
def _evaluate_assigned_alarms(self):
try:
alarms = self._assigned_alarms()
LOG.info(_('initiating evaluation cycle on %d alarms') %
len(alarms))
for alarm in alarms:
self._evaluate_alarm(alarm)
except Exception:
LOG.exception(_('alarm evaluation cycle failed'))分析:
这个方法是从数据库中获取所有告警，然后遍历每个告警，对每个告警调用
_evaluate_alarm方法进行告警验证
2.3 _evaluate_alarm方法具体如下:
def _evaluate_alarm(self, alarm):
"""Evaluate the alarms assigned to this evaluator."""
if alarm.type not in self.evaluators:
LOG.debug('skipping alarm %s: type unsupported', alarm.alarm_id)
return
LOG.debug('evaluating alarm %s', alarm.alarm_id)
try:
self.evaluators[alarm.type].obj.evaluate(alarm)
except Exception:
LOG.exception(_('Failed to evaluate alarm %s'), alarm.alarm_id)分析:
该方法是对当前待校验告警，获取其告警类型alarm.type，然后找到该告警类型对应的evaluator对象
【目前有: ThresholdEvaluator, GnocchiResourceThresholdEvaluator等】，调用该
evaluator对象的evaluate方法，最终会调用aodh/evaluator/threshold.py的
ThresholdEvaluator类的evaluate方法
2.4 aodh/evaluator/threshold.py中ThresholdEvaluator类的evaluate方法具体如下:
def evaluate(self, alarm):
if not self.within_time_constraint(alarm):
LOG.debug('Attempted to evaluate alarm %s, but it is not '
'within its time constraint.', alarm.alarm_id)
return
state, trending_state, statistics, outside_count = self.evaluate_rule(
alarm.rule, alarm_id=alarm.alarm_id)
self._transition_alarm(alarm, state, trending_state, statistics,
outside_count)分析:
上述方法调用evaluate_rule方法进行告警的验证； 具体就是根据
ceilometer监控数据源或者gnocchi监控数据源查询指定时间范围的监控项的监控数据
是否超过阈值,返回最终的告警状态
调用_transition_alarm方法来修改告警状态，并与notifier服务通信，进行告警触发；
具体参见2.5分析
2.5 aodh/evaluator/threshold.py中ThresholdEvaluator类的_transition_alarm方法具体如下:
def _transition_alarm(self, alarm, state, trending_state, statistics,
outside_count):
unknown = alarm.state == evaluator.UNKNOWN
continuous = alarm.repeat_actions
if trending_state:
if unknown or continuous:
state = trending_state if unknown else alarm.state
reason, reason_data = self._reason(alarm, statistics, state,
outside_count)
self._refresh(alarm, state, reason, reason_data)
return
if state == evaluator.UNKNOWN and not unknown:
LOG.warning(_LW('Expecting %(expected)d datapoints but only get '
'%(actual)d') % {
'expected': alarm.rule['evaluation_periods'],
'actual': len(statistics)})
# Reason is not same as log message because we want to keep
# consistent since thirdparty software may depend on old format.
reason = _('%d datapoints are unknown') % alarm.rule[
'evaluation_periods']
last = None if not statistics else statistics[-1]
reason_data = self._reason_data('unknown',
alarm.rule['evaluation_periods'],
last)
self._refresh(alarm, state, reason, reason_data)
elif state and (alarm.state != state or continuous):
reason, reason_data = self._reason(alarm, statistics, state,
outside_count)
self._refresh(alarm, state, reason, reason_data)分析:
最重要的逻辑在
elif state and (alarm.state != state or continuous):
reason, reason_data = self._reason(alarm, statistics, state,
outside_count)
self._refresh(alarm, state, reason, reason_data)如果已经告警，并且和之前的状态不同或者设置了连续告警，就调用_refresh方法
来进行告警通知，具体见2.6
2.6 在aodh/evaluator/__init__.py的Evaluator类的_refresh方法如下:
def _refresh(self, alarm, state, reason, reason_data, always_record=False):
"""Refresh alarm state."""
try:
previous = alarm.state
alarm.state = state
if previous != state or always_record:
LOG.info(_('alarm %(id)s transitioning to %(state)s because '
'%(reason)s') % {'id': alarm.alarm_id,
'state': state,
'reason': reason})
try:
self._storage_conn.update_alarm(alarm)
except storage.AlarmNotFound:
LOG.warning(_LW("Skip updating this alarm's state, the"
"alarm: %s has been deleted"),
alarm.alarm_id)
else:
self._record_change(alarm)
self.notifier.notify(alarm, previous, reason, reason_data)
elif alarm.repeat_actions:
self.notifier.notify(alarm, previous, reason, reason_data)
except Exception:
# retry will occur naturally on the next evaluation
# cycle (unless alarm state reverts in the meantime)
LOG.exception(_('alarm state update failed'))
分析:
这里通过self._storage_conn.update_alarm(alarm)来更新告警，
self._record_change(alarm)会记录告警历史
最重要的是
self.notifier.notify(alarm, previous, reason, reason_data)
会进行告警通知，原理是:让evaluator服务和notifier服务进行通信。
其中:self.notifier的定义如下
if conf.ipc_protocol == 'rpc':
self.notifier = rpc.RPCAlarmNotifier(self.conf)
else:
self.notifier = queue.AlarmNotifier(self.conf)也就是说这里根据是否配置的是rpc通信或者消息队列通信方式来实例化
不同的Notifier对象， 并调用该对象的notify方法
3 Aodh的notifier服务
源码:aodh/notifier/__init__.py中的AlarmNotifierService类，代码如下class AlarmNotifierService(os_service.Service):
NOTIFIER_EXTENSIONS_NAMESPACE = "aodh.notifier"
def __init__(self, conf):
super(AlarmNotifierService, self).__init__()
transport = messaging.get_transport(conf)
self.notifiers = extension.ExtensionManager(
self.NOTIFIER_EXTENSIONS_NAMESPACE,
invoke_on_load=True,
invoke_args=(conf,))
if conf.ipc_protocol == 'rpc':
self.ipc = 'rpc'
self.rpc_server = messaging.get_rpc_server(
conf, transport, conf.notifier_rpc_topic, self)
else:
self.ipc = 'queue'
target = oslo_messaging.Target(topic=conf.notifier_topic)
self.listener = messaging.get_notification_listener(
transport, [target],
[AlarmEndpoint(self.notifiers)])
def start(self):
super(AlarmNotifierService, self).start()
if self.ipc == 'rpc':
self.rpc_server.start()
else:
self.listener.start()
# Add a dummy thread to have wait() working
self.tg.add_timer(604800, lambda: None)
def stop(self):
if self.ipc == 'rpc':
self.rpc_server.stop()
else:
self.listener.stop()
self.listener.wait()
super(AlarmNotifierService, self).stop()
def notify_alarm(self, context, data):
process_alarm(self.notifiers, data)
分析:
1) __init__中
self.notifiers 部分是不同的通知对象的列表,然后根据self.ipc来区分是使用rpc或者消息队列的
方式进行aodh的evaluator和notifier服务之间的通信方式，根据aodh.conf中如下值
# The protocol used to communicate between evaluator and notifier services.
# (string value)
# Allowed values: queue, rpc
#ipc_protocol = queue
默认aodh采用消息队列的方式进行evaluator和notifier进行通信
2) 如果使用rpc方式进行evaluator和notifier通信，则从aodh/messaging.py中调用get_rpc_server方法
来获取rpc server,具体代码如下
def get_rpc_server(conf, transport, topic, endpoint):
"""Return a configured oslo_messaging rpc server."""
target = oslo_messaging.Target(server=conf.host, topic=topic)
return oslo_messaging.get_rpc_server(transport, target,
[endpoint], executor='threading',
serializer=_SERIALIZER)其中传入的topic值为conf.notifier_rpc_topic，具体在aodh.conf中值如下:
#notifier_rpc_topic = alarm_notifier
其中传入的endpoint是self,即AlarmNotifierService类本身，然后根据:
如果触发告警，那么aodh/evaluator/__init__.py中Evaluator类中的
_refresh方法调用了
self.notifier.notify(alarm, previous, reason, reason_data)
其中:self.notifier的定义如下
if conf.ipc_protocol == 'rpc':
self.notifier = rpc.RPCAlarmNotifier(self.conf)
else:
self.notifier = queue.AlarmNotifier(self.conf)也就是说这里根据是否配置的是rpc通信或者消息队列通信方式来实例化
不同的Notifier对象， 并调用该对象的notify方法，
总结:
以rpc通信方式为例,被调用的aodh/rpc.py中RPCAlarmNotifier类的分析具体见4
以消息队列通信方式为例，被调用的aodh/queue.py中的AlarmNotifier类的分析具体见5
4 evaluator和notifier之间的rpc通信方式
如果采用rpc通信方式， evaluator服务调用了aodh/evaluator/__init__.py中Evaluator类中的_refresh方法，该方法调用了aodh/rpc.py中的RPCAlarmNotifier类的notify方法。源码: aodh/rpc.py中RPCAlarmNotifier类
class RPCAlarmNotifier(object):
def __init__(self, conf):
transport = messaging.get_transport(conf)
self.client = messaging.get_rpc_client(
transport, topic=conf.notifier_rpc_topic,
version="1.0")
def notify(self, alarm, previous, reason, reason_data):
actions = getattr(alarm, models.Alarm.ALARM_ACTIONS_MAP[alarm.state])
if not actions:
LOG.debug('alarm %(alarm_id)s has no action configured '
'for state transition from %(previous)s to '
'state %(state)s, skipping the notification.',
{'alarm_id': alarm.alarm_id,
'previous': previous,
'state': alarm.state})
return
self.client.cast({},
'notify_alarm', data={
'actions': actions,
'alarm_id': alarm.alarm_id,
'alarm_name': alarm.name,
'severity': alarm.severity,
'previous': previous,
'current': alarm.state,
'reason': six.text_type(reason),
'reason_data': reason_data,
'alarm_rule': alarm.rule})分析:
1) __init__方法指定了rpc的topic为conf.notifier_rpc_topic，具体在aodh.conf中值如下
#notifier_rpc_topic = alarm_notifier
2)notify方法指定了调用rpc server中的notify_alarm方法,
而rpc server是aodh/notifier/__init__.py中的AlarmNotifierService类
该类中包含了
def notify_alarm(self, context, data):
process_alarm(self.notifiers, data)总结: 这里关于evaluator和notifier之间rpc通信方式中，步骤1: evaluator服务调用了aodh/evaluator/__init__.py中Evaluator类中的_refresh方法，该方法调用了aodh/rpc.py中RPCAlarmNotifier类的notify方法步骤2:rpc client实际是: aodh/rpc.py中RPCAlarmNotifier类入口是: 该类的notify方法，它指定了调用rpc server中的notify_alarm方法步骤3:rpc server实际是: aodh/notifier/__init__.py中的AlarmNotifierService类，该类定义了notify_alarm方法来被rpc client进行调用5 evaluator和notifier之间的消息队列通信方式
如果采用消息队列通信方式， evaluator服务调用了aodh/evaluator/__init__.py中Evaluator类中的_refresh方法，该方法调用了aodh/queue.py中的AlarmNotifier类的notify方法。aodh/queue.py中的AlarmNotifier类具体如下:
class AlarmNotifier(object):
def __init__(self, conf):
self.notifier = oslo_messaging.Notifier(
messaging.get_transport(conf),
driver='messagingv2',
publisher_id="alarming.evaluator",
topic=conf.notifier_topic)
def notify(self, alarm, previous, reason, reason_data):
actions = getattr(alarm, models.Alarm.ALARM_ACTIONS_MAP[alarm.state])
if not actions:
LOG.debug('alarm %(alarm_id)s has no action configured '
'for state transition from %(previous)s to '
'state %(state)s, skipping the notification.',
{'alarm_id': alarm.alarm_id,
'previous': previous,
'state': alarm.state})
return
payload = {'actions': actions,
'alarm_id': alarm.alarm_id,
'alarm_name': alarm.name,
'severity': alarm.severity,
'previous': previous,
'current': alarm.state,
'reason': six.text_type(reason),
'reason_data': reason_data,
'alarm_rule': alarm.rule}
self.notifier.sample({}, 'alarm.update', payload)分析:
1) 上述__init__方法中初始化了一个oslo_messaging的Notifier对象，
用于进行消息队列通信，其中topic=conf.notifier_topic,在aodh.conf
中其值如下:
#notifier_topic = alarming
2) 上述notify方法调用了
self.notifier.sample({}, 'alarm.update', payload)
来发送数据，该方法具体形式如下:
def sample(ctxt, event_type, payload):
参数：
ctxt： 请求的上下文字典，可以设置为空，字典类型
event_type:发送的事件类型，例如: alarm.update，字符串类型
payload:通知的具体内容，是一个字典，字典中键值对：键表示属性名称，值表示属性对应的值
3）
因为该aodh/queue.py中的AlarmNotifier类是消息队列的客户端，即消息发送方，
消息接收方是aodh/notifier/__init__.py的AlarmNotifierService类中
__init__方法中定义的self.listener
target = oslo_messaging.Target(topic=conf.notifier_topic)
self.listener = messaging.get_notification_listener(
transport, [target],
[AlarmEndpoint(self.notifiers)])上述AlarmEndpoint即为通知告警的真正处理类，该类在aodh/notifier/__init__.py
文件中，具体内容如下:
class AlarmEndpoint(object):
def __init__(self, notifiers):
self.notifiers = notifiers
def sample(self, ctxt, publisher_id, event_type, payload, metadata):
"""Endpoint for alarm notifications"""
process_alarm(self.notifiers, payload)其中sample方法即为被消息队列客户端所调用的真正方法
总结: 这里关于evaluator和notifier之间消息队列通信方式中，
步骤1: evaluator服务调用了aodh/evaluator/__init__.py中Evaluator类中的
_refresh方法，调用了aodh/queue.py中的AlarmNotifier类的notify方法
步骤2:
消息队列 client端实际是: aodh/queue.py中AlarmNotifier类
入口是: 该类的notify方法，它指定了调用rpc server中的
notify_alarm方法
步骤3:
消息队列 server端实际是: aodh/notifier/__init__.py中的AlarmNotifierService类，该类的__init__方法中指定了处理告警的类是
aodh/notifier/__init__.py中的AlarmEndpoint,该AlarmEndpoint
类的sample方法会被调用
最后告警通知的处理可以进行log,或者http请求等进行告警的处理。
6 总结rpc和消息队列通信方式
6.1 rpc通信方式client端部分:
1 初始化client端:
transport = messaging.get_transport(conf)
self.client = messaging.get_rpc_client(
transport, topic=conf.notifier_rpc_topic,
version="1.0")2 通过client端调用服务端方法:
self.client.cast({},
'notify_alarm', data={
'actions': actions,
'alarm_id': alarm.alarm_id,
'alarm_name': alarm.name,
'severity': alarm.severity,
'previous': previous,
'current': alarm.state,
'reason': six.text_type(reason),
'reason_data': reason_data,
'alarm_rule': alarm.rule})server端部分:
1 初始化server端
self.rpc_server = messaging.get_rpc_server(
conf, transport, conf.notifier_rpc_topic, self)2 启动server端:
self.rpc_server.start()3 server端的处理方法:
def notify_alarm(self, context, data):
process_alarm(self.notifiers, data)6.2 消息队列通信方式
client端部分:
1 初始化client端:
self.notifier = oslo_messaging.Notifier(
messaging.get_transport(conf),
driver='messagingv2',
publisher_id="alarming.evaluator",
topic=conf.notifier_topic)2 发送消息:
self.notifier.sample({}, 'alarm.update', payload)server端部分:
1 初始化server端:
target = oslo_messaging.Target(topic=conf.notifier_topic)
self.listener = messaging.get_notification_listener(
transport, [target],
[AlarmEndpoint(self.notifiers)])2 启动server端:
self.listener.start()3 server端处理方法
def sample(self, ctxt, publisher_id, event_type, payload, metadata):
"""Endpoint for alarm notifications"""
process_alarm(self.notifiers, payload)6.3 rpc和消息队列通信方式的相同点
client端初始化部分:
rpc client端需要: transport, topic, version
消息队列 client端需要: transport, topic, driver, publisher_id
rpc server端需要: transport, topic, 处理方法类对象列表
消息队列 server端需要: transport, target列表(每个target用topic初始化), 处理类对象列表
总结:
client端: rpc和消息队列都需要用到transport和topic
server端: rpc和消息队列都需要提供transport和处理类对象列表
7 总结aodh
aodh组件通过不同的数据源进行告警的校验，然后又采用了rabbitmq消息队列的方式进行了告警验证服务和告警通知服务之间的通信，最后将告警的结果以不同的形式进行
分发处理。
参考:
[1] https://github.com/openstack/aodh/tree/mitaka-eol