Gnocchi is an open supply time series database created in 2014 when OpenStack was in search of a extremely scalable, fault-tolerant time sequence database that didn’t rely on a specialised database (e.g., Hadoop, Cassandra, and many others.).
Gnocchi was initially constructed inside OpenStack, however later moved out of the mission, as a result of it was constructed to be platform-agnostic. Even so, Gnocchi continues to be used as the principle time sequence backend by this cloud platform; for instance, OpenStack Ceilometer leverages Gnocchi’s giant scalability and high-availability properties to make sure its telemetry is at all times up and quick.
The downside that Gnocchi solves is storage and indexing of time sequence knowledge and sources at giant scale. Modern cloud platforms will not be solely large, however in addition they are dynamic and probably multi-tenant. Gnocchi takes all of that under consideration.
Aggregation
Gnocchi takes a novel method to time sequence storage: Rather than storing uncooked knowledge factors, it aggregates them earlier than storing them. This built-in function is totally different from most different time sequence databases, which normally assist this mechanism as an possibility and compute aggregation (common, minimal, and many others.) at question time.
Because Gnocchi computes all of the aggregations at ingestion, getting the info again is extraordinarily quick, because it simply must learn again the pre-computed outcomes.
The manner these knowledge factors are aggregated is configurable on a per-metric foundation, utilizing an archive coverage. An archive coverage defines which aggregations to compute and what number of aggregates to maintain. Gnocchi helps a wild variety of aggregation strategies, corresponding to minimal, most, common, Nth percentile, commonplace deviation, and many others. Those aggregations are computed over a time frame (referred to as granularity) and are saved for an outlined timespan. Aggregates are saved in a compressed format, making certain the info factors take as little area as doable.
For instance, think about you outline an archive coverage to maintain the typical, minimal, and most of a time sequence with five-minute granularity for 30 days. In that case, Gnocchi will compute the typical, minimal, and most of the values over five-minute ranges, retaining as much as eight,640 factors (the variety of five-minute aggregates you may have over 30 days).
Getting began
Before putting in Gnocchi, it’s good to resolve which measures and mixture storage drivers you wish to use. Gnocchi can leverage extremely scalable storage methods, corresponding to Ceph, OpenStack Swift, Redis, and even Amazon S3. If none of these choices can be found to you, you may nonetheless use a typical file system.
You additionally want a database to index the sources and metrics that Gnocchi will deal with—each PostgreSQL and MySQL are supported.
The installation page describes find out how to arrange Gnocchi and write its configuration file. Once this configuration file is written, the gnocchi-upgrade program will arrange the storage engine and the database index so they’re prepared for use.
Gnocchi consists of two central providers: an HTTP server, offering a REST API, and a metric processing daemon. The former is named gnocchi-api; the latter is known as gnocchi-metricd. You must run each providers to make use of Gnocchi. The REST API can be utilized by shoppers to question Gnocchi and write knowledge to it. The metricd service will ingest measures obtained by the API, compute the aggregates, and retailer them in the long-term mixture storage.
Both these providers are stateless and due to this fact horizontally scalable. Contrary to many time sequence databases, there isn’t a restrict on the variety of metricd daemons or API endpoints you can run with Gnocchi. If your load begins to extend, you simply must spawn extra daemons to deal with the circulation of latest requests. The similar applies if you wish to deal with high-availability situations: simply begin extra Gnocchi daemons on unbiased servers.
Using the command-line device
Once Gnocchi is deployed, you may set up the Gnocchi shopper command-line device utilizing pip (the Python package deal installer); simply kind pip set up gnocchiclient. It sends requests to the HTTP REST API and codecs and prints again the replies.
To arrange metrics, Gnocchi supplies the notion of sources. A useful resource can have any variety of metrics connected to it:
$ gnocchi useful resource create server42
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| created_by_project_id | |
| created_by_user_id | admin |
| creator | admin |
| ended_at | None |
| id | 43164383-6dc3-5034-a675-9a39493ca7df |
| metrics | |
| original_resource_id | server42 |
| project_id | None |
| revision_end | None |
| revision_start | 2017-10-20T13:19:46.902754+00:00 |
| started_at | 2017-10-20T13:19:46.902642+00:00 |
| kind | generic |
| user_id | None |
+-----------------------+--------------------------------------+
Gnocchi handles revisions of sources; if any attribute is modified, Gnocchi will file that change and create a brand new entry within the historical past of the useful resource.
Once this useful resource is created, we will create a brand new metric for this useful resource:
$ gnocchi metric create -r server42 cpu
+------------------------------------+------------------------------------------------------------------+
| Field | Value |
+------------------------------------+------------------------------------------------------------------+
| archive_policy/aggregation_methods | std, rely, min, max, sum, imply |
| archive_policy/back_window | zero |
| archive_policy/definition | - factors: 8640, granularity: zero:05:00, timespan: 30 days, zero:00:00 |
| archive_policy/identify | low |
| created_by_project_id | |
| created_by_user_id | admin |
| creator | admin |
| id | ab0b5dab-31e9-4760-a58f-0dc324047f9f |
| identify | cpu |
| useful resource/created_by_project_id | |
| useful resource/created_by_user_id | admin |
| useful resource/creator | admin |
| useful resource/ended_at | None |
| useful resource/id | 43164383-6dc3-5034-a675-9a39493ca7df |
| useful resource/original_resource_id | server42 |
| useful resource/project_id | None |
| useful resource/revision_end | None |
| useful resource/revision_start | 2017-10-20T13:19:46.902754+00:00 |
| useful resource/started_at | 2017-10-20T13:19:46.902642+00:00 |
| useful resource/kind | generic |
| useful resource/user_id | None |
| unit | None |
+------------------------------------+------------------------------------------------------------------+
This creates a metric named cpu and attaches it to the useful resource server42. The archive coverage utilized by default for brand spanking new metrics is low, which is offered by default when putting in Gnocchi. The low archive coverage shops varied aggregation strategies for 30 days over 5 minutes granularity.
This new metric has no measures, so it’s time to ship a number of:
$ gnocchi measures add -m "2017-10-20 12:00@42" -m "2017-10-20 12:03@18" -m "2017-10-20 12:06@56" -r server42 cpu
$ gnocchi measures present -r server42 cpu
+---------------------------+-------------+-------+
| timestamp | granularity | worth |
+---------------------------+-------------+-------+
| 2017-10-20T12:00:00+02:00 | 300.zero | 30.zero |
| 2017-10-20T12:05:00+02:00 | 300.zero | 56.zero |
+---------------------------+-------------+-------+
The three measures despatched are aggregated into two knowledge factors for 12:00 and 12:05. By default, the aggregation methodology proven is imply, however you may request another aggregation methodology:
$ gnocchi measures present --aggregation rely -r server42 cpu
+---------------------------+-------------+-------+
| timestamp | granularity | worth |
+---------------------------+-------------+-------+
| 2017-10-20T12:00:00+02:00 | 300.zero | 2.zero |
| 2017-10-20T12:05:00+02:00 | 300.zero | 1.zero |
+---------------------------+-------------+-------+
The rely aggregation methodology computes the variety of knowledge factors obtained over every interval.
The default useful resource kind is generic, however you may create your personal useful resource kind with any additional variety of attributes. Those attributes could be non-compulsory and may have particular sorts to verify your knowledge is correctly structured and straightforward to eat:
$ gnocchi resource-type create -a ipaddress:string:true network-switch
+----------------------+----------------------------------------------------------+
| Field | Value |
+----------------------+----------------------------------------------------------+
| attributes/ipaddress | max_length=255, min_length=zero, required=True, kind=string |
| identify | network-switch |
| state | lively |
+----------------------+----------------------------------------------------------+
$ gnocchi useful resource create --type network-switch --attribute ipaddress:192.168.2.three sw18-prs.instance.com
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| created_by_project_id | |
| created_by_user_id | admin |
| creator | admin |
| ended_at | None |
| id | 4fbdf14d-5650-58f5-b232-dc3cf91c28d0 |
| ipaddress | 192.168.2.three |
| metrics | |
| original_resource_id | sw18-prs.instance.com |
| project_id | None |
| revision_end | None |
| revision_start | 2017-10-20T15:08:30.935529+00:00 |
| started_at | 2017-10-20T15:08:30.935513+00:00 |
| kind | network-switch |
| user_id | None |
+-----------------------+--------------------------------------+
If the attribute worth ever modifications, its historical past could be retrieved:
$ gnocchi useful resource replace --type network-switch --attribute ipaddress:192.168.2.48 sw18-prs.instance.com
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| created_by_project_id | |
| created_by_user_id | admin |
| creator | admin |
| ended_at | None |
| id | 4fbdf14d-5650-58f5-b232-dc3cf91c28d0 |
| ipaddress | 192.168.2.48 |
| metrics | |
| original_resource_id | sw18-prs.instance.com |
| project_id | None |
| revision_end | None |
| revision_start | 2017-10-20T15:10:11.903018+00:00 |
| started_at | 2017-10-20T15:08:30.935513+00:00 |
| kind | network-switch |
| user_id | None |
+-----------------------+--------------------------------------+$ gnocchi useful resource historical past --format yaml --type network-switch sw18-prs.instance.com
- creator: admin
ended_at: null
id: 4fbdf14d-5650-58f5-b232-dc3cf91c28d0
ipaddress: 192.168.2.three
metrics:
original_resource_id: sw18-prs.instance.com
project_id: null
revision_end: '2017-10-20T15:10:11.903018+00:00'
revision_start: '2017-10-20T15:08:30.935529+00:00'
started_at: '2017-10-20T15:08:30.935513+00:00'
kind: network-switch
user_id: null
- creator: admin
ended_at: null
id: 4fbdf14d-5650-58f5-b232-dc3cf91c28d0
ipaddress: 192.168.2.48
metrics:
original_resource_id: sw18-prs.instance.com
project_id: null
revision_end: null
revision_start: '2017-10-20T15:10:11.903018+00:00'
started_at: '2017-10-20T15:08:30.935513+00:00'
kind: network-switch
user_id: null
The revision fields point out what time the modifications had been made. Both the previous and the brand new attribute values of ipaddress are accessible, making it straightforward to hint how a useful resource has been modified.
Gnocchi affords different options, like search in sources and metrics, superior cross-aggregation of metrics, entry management checklist (ACL) administration, and extra.
Integration with different instruments
Gnocchi can combine with many different instruments. For instance, it’s doable to make use of its collectd plugin to gather metrics with collectd and ship them to Gnocchi.
Gnocchi additionally helps Grafana, the usual device for displaying charts from time sequence databases, via a Grafana plugin.
Gnocchi’s roadmap
Gnocchi is actively developed, and new options are at all times coming. The subsequent model (four.1) ought to present extra computing potentialities for cross-metric aggregation. Also, an ingestion level for Prometheus is being constructed so Gnocchi can be utilized as a long-term and scalable storage possibility for the Prometheus time sequence alerting and monitoring system.
