Kubernetes ServiceMonitor for external targets
When using the prometheus-operator, it is a very easy to add new targets by just creating new ServiceMonitor resources. But this only works for targets inside the cluster. There are lots of discussions on the internet how to add external resources to be scraped/monitored. The consensus seems to be to create a service that has endpoints pointing to the external resources. This seems to be nice and aligned on how k8s works. Only problem is that there does not seem to be a simple guide on how to populate the instance labels with meaningful names. If you just use endpoints, the instance name will be the IP of the endpoint. The solution is to use relabeling in the serviceMonitor and add the names you want to have to the endpoint. The endpoints the would look like this:
- ip: 192.168.1.1
hostname: oneendpoint
and the hostname would then be attached to the target instance label by this relableing config:
relabelings:
- sourceLabels: [__meta_kubernetes_endpoint_hostname]
targetLabel: instance
Here is a full example with Service,Enpoint and ServiceMonitor
Create a service without endpoints:
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: something
name: something
spec:
clusterIP: None
ports:
- name: metrics
port: 9100
And then manually create endpoints that are going to be scraped by kubernetes:
apiVersion: v1
kind: Endpoints
metadata:
labels:
app.kubernetes.io/name: something
name: something
subsets:
- addresses:
- ip: 192.168.1.1
hostname: oneendpoint
- ip: 192.168.1.2
hostname: anotherone
- ip: 192.168.1.3
hostname: thirdendpoint
ports:
- name: metrics
port: 9100
Now you have a service that points to three endpoints. To tell prometheus to scrape these, you add the serviceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/name: something
name: something
spec:
endpoints:
- interval: 60s
port: metrics
relabelings:
- sourceLabels: [__meta_kubernetes_endpoint_hostname]
targetLabel: instance
selector:
matchLabels:
app.kubernetes.io/name: something
the selector of the serviceMonitor points to the service, and this points to the endpoints. Works. All metrics end up in prometheus.
Result: Three endpoints are “automatically” discovered by prometheus and instance names are provided by the endpoint. Seems to be better than hardcoding instance names in the serviceMonitor.