Enabling Ceilometer Events

Together with Christian Brinker, I spent a good amount of hours troubleshooting our OpenStack Liberty installation. To be more precise, the Telemetry Service called “Ceilometer”. As Christian is working on a billing software for OpenStack he started digging into the Ceilometer API – something I nevery really had a close look at. He found out, that metrics about cpu, disk, network etc was property processed and stored and could be queried, BUT there was absolutely no trace of any events happening in the cloud. Well, seemed odd!

We started out by triggering events such as Cinder volume creations and deletions and checked again: Nothing. After digging through the configuration files for a while, we figured that a good part of them was missing! Take a look at what the Liberty package on Ubuntu 14.04 comes with:

mewald@controller:~$ dpkg -L ceilometer-common | grep etc
/etc
/etc/logrotate.d
/etc/logrotate.d/ceilometer-common
/etc/ceilometer
/etc/ceilometer/ceilometer.conf
/etc/ceilometer/api_paste.ini
/etc/ceilometer/rootwrap.conf
/etc/ceilometer/policy.json
/etc/ceilometer/pipeline.yaml
mewald@controller:~$

We compared what existed here with what the Git repository of Ceilometer comes with and quickly found out that two files were not too unimportant: event_definitions.yaml and event_pipeline.yaml. The Liberty packages just doesnt ship those! Theses config files are responsible for telling Ceilometer which events to process and how to process them, so we added them and restarted the services.

Still, the output of ceilometer event-list showed all empty tables. We spent a good while proving that event notifications actually hit the message queue and we could even see that ceilometer-agent-notification received the messages and took them off of the queue. So the problem had to be right there in the controller node.

After some struggling with the configuration and some research we found out that a very important piece in ceilometer.conf was missing: The default file does not contain the “notification” section, which may specify the following:

[notification]
store_events = True

This tells Ceilometer to make events persistent in the datebase which enabled us to make queries for events through the API.

To safe you this trouble, I uploaded our configuration files for you for copy & paste:

ceilometer.conf:

https://gist.github.com/anonymous/df763e044db9d1554bef

pipeline.yaml:

https://gist.github.com/anonymous/dbab8d0d3391007e0de0

event_definitions.yaml:

https://gist.github.com/anonymous/c7c494f223f5dfb03346

event_pipeline.yaml:

https://gist.github.com/anonymous/960ba3e17538d595d288