Consuming Metrics from Prometheus

Starting with Orka v2.2.0 it is now possible to collect metrics from your Orka cluster, including host metrics for each node in the cluster as well as metrics for the Orka API server.

Consuming Metrics from the Orka Cluster Prometheus Instance

By default the Prometheus instance running in-cluster is exposed on port 9090 of the Orka load-balancer IP address. Typically, this will be either http://10.221.188.20:9090 or http://10.221.188.100:9090. Consult your IP plan if you aren't sure.

🚧

Outside Access

Make sure that TCP traffic is permitted to port 9090 for the Orka load-balancer IP address through your firewall or ACL!

If you are connected to the Orka cluster from your local machine via a VPN connection, you can run PromQL queries in your browser by navigating to http://<orka-load-balancer>:9090/graph. For example, to get the number of Orka VM deployments over the last 2 hours, try the following query:

floor(increase(orka_server_http_requests_total{route="/resources/vm/deploy"}[2h]))

Integrating Orka Metrics with an Existing Prometheus Instance

If your organization is already using Prometheus and you wish to collect additional data from your Orka cluster, follow the instructions in this section.

Orka Server and Operator Metrics

Metrics for the Orka server and operator are available via the /metrics endpoint of the Orka API address. For most environments this is typically 10.221.188.20 or 10.221.188.100, but check your IP plan if you aren't sure.

Use port 8080 in the endpoint for the operator metrics.

To verify you are able to access the Orka server and operator metrics, try cURLing the endpoints:

curl -I 10.221.188.20/metrics
curl -I 10.221.188.20:8080/metrics

You should receive a response of 200 OK if all is well.

👍

TIP

To view descriptions and additional information on the available Orka server metrics, try curl 10.221.188.20/metrics. For Orka operator metrics, try curl 10.221.188.20:8080/metrics

In order to scrape the Orka server and operator metrics, add jobs to your Prometheus configuration:

scrape_configs:
  # ...
  
  - job_name: 'orka/orka-server'
    scrape_interval: 30s
    scrape_timeout: 30s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['10.221.188.20']
        labels:
          service: 'orka-server'
          
  - job_name: 'orka/orka-operator'
    scrape_interval: 30s
    scrape_timeout: 30s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['10.221.188.20:8080']
        labels:
          service: 'orka-operator'

Prometheus Node Exporters

Every node in the Orka cluster runs a Prometheus Node Exporter on port 9100. This includes both Intel and ARM nodes. To ensure you are able to scrape the node exporter data, try cURLing the metrics endpoint from any node:

curl -I http://10.221.188.31:9100/metrics

If the node exporter is running and reachable from outside the cluster, you should receive a response of 200 OK.

🚧

Outside Access

If your Prometheus instance is hosted outside of the cluster, make sure that TCP traffic is permitted to port 9100 for all hosts on the Orka private network! For more information, see the section on IP plans.

In order to scrape the node exporter data, add a job to your Prometheus configuration. For example:

scrape_configs:
  # ...

  - job_name: 'orka/node-exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['10.221.188.5:9100', '10.221.188.6:9100', '10.221.188.7:9100']
        labels:
          group: 'master'
      - targets: ['10.221.188.31:9100', '10.221.188.32:9100', '10.221.188.33:9100']
        labels:
          group: 'x86'
      - targets: ['10.221.188.34:9100']
        labels:
          group: 'arm'

The above configuration will work well for a small cluster with only a few Orka nodes. However, for a larger cluster, it is recommended to use the Prometheus instance running in-cluster that is configured to use Kubernetes service discovery.

👍

TIP

As an alternative to the static config, you can use file-based service discovery to scrape the targets