When using the prometheus-operator, it is a very easy to add new targets by just creating new ServiceMonitor resources. But this only works for targets inside the cluster. There are lots of discussions on the internet how to add external resources to be scraped/monitored. The consensus seems to be to create a service that has endpoints pointing to the external resources. This seems to be nice and aligned on how k8s works. Only problem is that there does not seem to be a simple guide on how to populate the instance labels with meaningful names. If you just use endpoints, the instance name will be the IP of the endpoint. The solution is to use relabeling in the serviceMonitor and add the names you want to have to the endpoint. The endpoints the would look like this:

  - ip: 192.168.1.1
    hostname: oneendpoint

and the hostname would then be attached to the target instance label by this relableing config:

    relabelings:
      - sourceLabels: [__meta_kubernetes_endpoint_hostname]
        targetLabel: instance

Here is a full example with Service,Enpoint and ServiceMonitor

Create a service without endpoints:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: something
  name: something
spec:
  clusterIP: None
  ports:
  - name: metrics
    port: 9100

And then manually create endpoints that are going to be scraped by kubernetes:

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app.kubernetes.io/name: something
  name: something
subsets:
- addresses:
  - ip: 192.168.1.1
    hostname: oneendpoint
  - ip: 192.168.1.2
    hostname: anotherone
  - ip: 192.168.1.3
    hostname: thirdendpoint
  ports:
  - name: metrics
    port: 9100

Now you have a service that points to three endpoints. To tell prometheus to scrape these, you add the serviceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: something
  name: something
spec:
  endpoints:
  - interval: 60s
    port: metrics
    relabelings:
      - sourceLabels: [__meta_kubernetes_endpoint_hostname]
        targetLabel: instance
  selector:
    matchLabels:
      app.kubernetes.io/name: something

the selector of the serviceMonitor points to the service, and this points to the endpoints. Works. All metrics end up in prometheus.

Result: Three endpoints are “automatically” discovered by prometheus and instance names are provided by the endpoint. Seems to be better than hardcoding instance names in the serviceMonitor.