My Workday

Prometheus provides great short-term metrics storage for the Grafana stack. However, it has its scalability limitations. Using remoteWrite to transition to a new architecture while preserving the existing flow of metrics is a great option for some.

When enabled, remote write increases the memory footprint of Prometheus. On average the increase is about 25%, but that number depends on the shape of the data. It is because each remote write destination starts a queue which reads from the write-ahead log (WAL), writes the samples into an in memory queue owned by a shard, which then sends a request to the configured end-point.

       |--> queue (shard 1) ---> remote end-point
WAL ---|--> queue (shard 2) ---> remote end-point
       |--> queue (shard n) ---> remote end-point

Below example is in Helm and demonstrates how to enable remoteWrite.

  remoteWrite:
    - url: https://<target end-point e.g. mimir>/api/v1/push
      basic_auth:
        username: <user>
        password: <password>
      tls_config:
        insecure_skip_verify: false