Prometheus is an more and more well-liked—for good purpose—open supply instrument that gives monitoring and alerting for functions and servers. Prometheus’ nice power is in monitoring server-side metrics, which it shops as time-series data. While Prometheus would not lend itself to software efficiency administration, lively management, or consumer expertise monitoring (though a GitHub extension does make consumer browser metrics obtainable to Prometheus), its prowess as a monitoring system and talent to realize high-scalability by means of a federation of servers make Prometheus a powerful alternative for all kinds of use circumstances.
In this text, we’ll take a better have a look at Prometheus’ structure and performance after which look at an in depth occasion of the instrument in motion.
Prometheus structure and elements
Prometheus consists of the Prometheus server (dealing with service discovery, metrics retrieval and storage, and time-series information evaluation by means of the PromQL question language), an information mannequin for metrics, a graphing GUI, and native assist for Grafana. There can also be an non-obligatory alert supervisor that permits customers to outline alerts through the question language and an non-obligatory push gateway for short-term software monitoring. These elements are located as proven within the following diagram.
Prometheus can mechanically seize commonplace metrics through the use of brokers to execute general-purpose code within the software setting. It can even seize customized metrics by means of instrumentation, putting customized code throughout the supply code of the monitored software. Prometheus formally helps client libraries for Go, Python, Ruby, and Java/Scala and likewise permits customers to jot down their very own libraries. Additionally, many unofficial libraries for different languages can be found.
Developers can even make the most of third-party exporters to mechanically activate instrumentation for a lot of well-liked software program options they is likely to be utilizing. For instance, customers of JVM-based functions like open supply Apache Kafka and Apache Cassandra can simply gather metrics by leveraging the prevailing JMX exporter. In different circumstances, an exporter will not be wanted as a result of the appliance will expose metrics which are already within the Prometheus format. Those on Cassandra may additionally discover Instaclustr’s freely obtainable Cassandra Exporter for Prometheus to be useful, because it integrates Cassandra metrics from a self-managed cluster into Prometheus software monitoring.
Also vital: Developers can leverage an obtainable node exporter to watch kernel metrics and host hardware. Prometheus presents a Java client as properly, with numerous options that may be registered both piecemeal or directly by means of a single DefaultExports.initialize(); command—together with reminiscence swimming pools, rubbish assortment, JMX, classloading, and thread counts.
Prometheus information modeling and metrics
Prometheus offers 4 metric sorts:
- Counter: Counts incrementing values; a restart can return these values to zero
- Gauge: Tracks metrics that may go up and down
- Histogram: Observes information in accordance with specified response sizes or durations and counts the sums of noticed values together with counts in configurable buckets
- Summary: Counts noticed information just like a histogram and presents configurable quantiles which are calculated over a sliding time window
Prometheus time-series information metrics every embody a string identify, which follows a naming conference to incorporate the identify of the monitored information topic, the logical kind, and the models of measure used. Each metric contains streams of 64-bit float worth which are timestamped right down to the millisecond, and a set of key:worth pairs labeling the scale it measures. Prometheus mechanically provides Job and Instance labels to every metric to maintain observe of the configured job identify of the info goal and the <host>:<port> piece of the scraped goal URL, respectively.
Prometheus instance: the Anomalia Machina anomaly detection experiment
Before shifting into the instance, obtain and start utilizing open supply Prometheus by following this getting started information.
To display easy methods to put Prometheus into motion and carry out software monitoring at a excessive scale, let’s check out a latest experimental Anomalia Machina project we accomplished at Instaclustr. This venture—only a check case, not a commercially obtainable answer—leverages Kafka and Cassandra in an software deployed by Kubernetes, which performs anomaly detection on streaming information. (Such detection is essential to make use of circumstances together with IoT functions and digital advert fraud, amongst different areas.) The experimental software depends closely on Prometheus to gather software metrics throughout distributed cases and make them available to view.
This diagram shows the experiment’s structure:
Our objectives in using Prometheus included monitoring the appliance’s extra generic metrics, similar to throughput, in addition to the response instances delivered by the Kafka load generator (the Kafka producer), the Kafka client, and the Cassandra consumer tasked with detecting any anomalies within the information. Prometheus displays the system’s hardware metrics as properly, such because the CPU for every AWS EC2 occasion operating the appliance. The venture additionally counts on Prometheus to watch application-specific metrics similar to the full variety of rows every Cassandra learn returns and, crucially, the variety of anomalies it detects. All of this monitoring is centralized for simplicity.
In observe, this implies forming a check pipeline with producer, client, and detector strategies, in addition to the next three metrics:
- A counter metric, known as prometheusTest_requests_total, increments every time that every pipeline stage executes with out incident, whereas a stage label permits for monitoring the profitable execution of every stage, and a whole label tracks the full pipeline rely.
- Another counter metric, known as prometheusTest_anomalies_total, counts any detected anomalies.
- Finally, a gauge metric known as prometheusTest_duration_seconds tracks the seconds of length for every stage (once more utilizing a stage label and a whole label).
The code behind these measurements increments counter metrics utilizing the inc() methodology and units the time worth of the gauge metric with the setToTime() methodology. This is demonstrated within the following annotated instance code:
import java.io.IOException;
import io.prometheus.consumer.Counter;
import io.prometheus.consumer.Gauge;
import io.prometheus.consumer.exporter.HTTPServer;
import io.prometheus.consumer.hotspot.DefaultExports;
// https://github.com/prometheus/client_java
// Demo of how we plan to make use of Prometheus Java consumer to instrument Anomalia Machina.
// Note that the Anomalia Machina software may have Kafka Producer and Kafka client and remainder of pipeline operating in a number of separate processes/cases.
// So metrics from every may have completely different host/port mixtures.
public class PrometheusWeblog {
static String appName = "prometheusTest";
// counters can solely improve in worth (till course of restart)
// Execution rely. Use a single Counter for all levels of the pipeline, levels are distinguished by labels
static closing Counter pipelineCounter = Counter.construct()
.identify(appName + "_requests_total").assist("Count of executions of pipeline stages")
.labelNames("stage")
.register();
// in principle might additionally use pipelineCounter to rely anomalies discovered utilizing one other label
// however much less potential for confusion having one other counter. Doesn't want a label
static closing Counter anomalyCounter = Counter.construct()
.identify(appName + "_anomalies_total").assist("Count of anomalies detected")
.register();
// A Gauge can go up and down, and is used to measure present worth of some variable.
// pipelineGauge will measure length in seconds of every stage utilizing labels.
static closing Gauge pipelineGauge = Gauge.construct()
.identify(appName + "_duration_seconds").assist("Gauge of stage durations in seconds")
.labelNames("stage")
.register();
public static void essential(String[] args)
// Allow default JVM metrics to be exported
DefaultExports.initialize();
// Metrics are pulled by Prometheus, create an HTTP server because the endpoint
// Note if there are a number of processes operating on the identical server want to vary port quantity.
// And add all IPs and port numbers to the Prometheus configuration file.
HTTPServer server = null;
strive catch (IOException e)
// now run 1000 executions of the whole pipeline with random time delays and growing charge
int max = 1000;
for (int i=zero; i < max; i++)
server.cease();
// the three levels of the pipeline, for every we improve the stage counter and set the Gauge length time
public static void producer()
class Local ;
String identify = Local.class.getEnclosingMethod().getName();
pipelineGauge.labels(identify).setToTime(() -> );
pipelineCounter.labels(identify).inc();
public static void client()
class Local ;
String identify = Local.class.getEnclosingMethod().getName();
pipelineGauge.labels(identify).setToTime(() -> );
pipelineCounter.labels(identify).inc();
// detector returns true if anomaly detected else false
public static boolean detector()
class Local ;
String identify = Local.class.getEnclosingMethod().getName();
pipelineGauge.labels(identify).setToTime(() ->
strive
Thread.sleep(1 + (lengthy)(Math.random()*200));
catch (InterruptedException e)
);
pipelineCounter.labels(identify).inc();
return (Math.random() > zero.95);
}
Prometheus collects metrics by polling (“scraping”) instrumented code (in contrast to another monitoring options that obtain metrics through push strategies). The code instance above creates a required HTTP server on port 1234 in order that Prometheus can scrape metrics as wanted.
The following pattern code addresses Maven dependencies:
<!-- The consumer -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<model>LATEST</model>
</dependency>
<!-- Hotspot JVM metrics-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<model>LATEST</model>
</dependency>
<!-- Exposition HTTPServer-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<model>LATEST</model>
</dependency>
<!-- Pushgateway exposition-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_pushgateway</artifactId>
<model>LATEST</model>
</dependency>
The code instance under tells Prometheus the place it ought to look to scrape metrics. This code can merely be added to the configuration file (default: Prometheus.yml) for primary deployments and exams.
international:
scrape_interval: 15s # By default, scrape targets each 15 seconds.
# scrape_configs has jobs and targets to scrape for every.
scrape_configs:
# job 1 is for testing prometheus instrumentation from a number of software processes.
# The job identify is added as a label job=<job_name> to any timeseries scraped from this config.
- job_name: 'testprometheus'
# Override the worldwide default and scrape targets from this job each 5 seconds.
scrape_interval: 5s
# that is the place to place a number of targets, e.g. for Kafka load turbines and detectors
static_configs:
- targets: ['localhost:1234', 'localhost:1235']
# job 2 offers working system metrics (e.g. CPU, reminiscence and so on).
- job_name: 'node'
# Override the worldwide default and scrape targets from this job each 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']
Note the job named “node” that makes use of port 9100 on this configuration file; this job presents node metrics and requires operating the Prometheus node exporter on the identical server the place the appliance is operating. Polling for metrics ought to be executed with care: doing it too usually can overload functions, too sometimes may end up in lag. Where software metrics cannot be polled, Prometheus additionally presents a push gateway.
Viewing Prometheus metrics and outcomes
Our experiment initially used expressions, and later Grafana, to visualise information and overcome Prometheus’ lack of default dashboards. Using the Prometheus interface (or http://localhost:9090/metrics), choose metrics by identify after which enter them within the expression field for execution. (Note that it is common to expertise error messages at this stage, so do not be discouraged in case you encounter a number of points.) With appropriately functioning expressions, outcomes will probably be obtainable for show in tables or graphs as applicable.
Using the irate or rate perform on a counter metric will produce a helpful charge graph:
Here is the same graph of a gauge metric:
Grafana offers far more sturdy graphing capabilities and built-in Prometheus assist with graphs capable of show a number of metrics:
To allow Grafana, set up it, navigate to http://localhost:3000/, create a Prometheus information supply, and add a Prometheus graph utilizing an expression. A observe right here: An empty graph usually factors to a time vary concern, which may normally be solved through the use of the “Last 5 minutes” setting.
Creating this experimental software provided a wonderful alternative to construct our data of what Prometheus is able to and resulted in a high-scale experimental manufacturing software that may monitor 19 billion real-time information occasions for anomalies every day. By following this information and our instance, hopefully, extra builders can efficiently put Prometheus into observe.