monitoring

Full monitoring and alerting stack containing:

• Alloy: Log and metric collector
  • Github
  • Website
• Loki: Log aggregation
  • Github
  • Website
• Grafana: Query and visualize metrics
  • Github
  • Website
• Prometheus: Metric collection
  • Github
  • Website
• AlertManager: Sending alerts
  • Github
  • Website
• AlertManager-ntfy: Forward alerts to ntfy
  • Github

Examples

Simple

{
  nps.stacks.monitoring.enable = true;
}

With Grafana OIDC Login

{config, ...}: {
  nps.stacks.monitoring = {
    enable = true;

    grafana = {
      oidc = {
        enable = true;
        clientSecretHash = "$pbkdf2-sha512$...";
        clientSecretFile = config.sops.secrets."grafana/authelia/client_secret".path;
      };
    };
  };
}

With Prometheus Rules + Ntfy Alerting

{config, ...}: {
  nps.stacks.monitoring = {
    monitoring.enable = true;

    prometheus.rules.groups = let
      cpuThresh = 90;
      ramThresh = 85;
    in [
      {
        name = "resource.usage";
        interval = "30s";
        rules = [
          {
            alert = "HighCpuUsage";
            expr = ''100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > ${toString cpuThresh}'';
            for = "20m";
            labels = {
              severity = "warning";
            };
            annotations = {
              summary = "High CPU usage";
              description = "CPU usage is above ${toString cpuThresh}% (current value: {{ $value }}%)";
            };
          }
          {
            alert = "HighMemoryUsage";
            expr = ''(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > ${toString ramThresh}'';
            labels = {
              severity = "warning";
            };
            annotations = {
              summary = "High memory usage";
              description = "Memory usage is above ${toString ramThresh}% (current value: {{ $value }}%)";
            };
          }
        ];
      }
    ];

    alertmanager = {
      enable = true;
      ntfy = {
        enable = true;
        tokenFile = config.sops.secrets."ntfy/monitoring/ntfy_access_token".path;
        settings.ntfy.notification.topic = "monitoring";
      };
    };
  };
}

Stack Options

nps.stacks.monitoring.alertmanager.enable

Whether to enable the Alertmanager.

When setting alertmanager.ntfy.enable, a route and receiver configuration will be setup for ntfy. If using without ntfy, you will have to provide your own route and receiver configurations via the settings option for Alertmanager to startup correctly.

Type

boolean

Default

false

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alertmanager.ntfy.enable

Whether to setup and configure alertmanager-ntfy. This allows alerts to be forwarded to ntfy.

See https://github.com/alexbakker/alertmanager-ntfy

Type

boolean

Default

config.nps.stacks.ntfy.enable

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alertmanager.ntfy.settings

alertmanager-ntfy configuration. Will be provided as the config.yml.

See https://github.com/alexbakker/alertmanager-ntfy/pkgs/container/alertmanager-ntfy#configuration

Type

YAML 1.1 value

Default

{ }

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alertmanager.ntfy.tokenFile

Path to the file containing the token that is used for authentication against the ntfy service. Only required if ntfy is configured to require authentication.

Type

null or absolute path

Default

null

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alertmanager.settings

Alertmanager configuration. Will be provided as the alertmanager.yml.

See https://prometheus.io/docs/alerting/latest/configuration/

Type

YAML 1.1 value

Default

{ }

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alloy.collectByDefault

Whether to collect logs from a containers by default. If set to false, only containers that explicitly set the alloy.enable=true option will be collected.

If set to true, you can opt out for individual containers by setting alloy.enable=false in the container's configuration.

See https://tarow.github.io/nix-podman-stacks/docs/container-options.html#services.podman.containers.<name>.alloy.enable

Type

boolean

Default

false

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alloy.config

Configuration for Alloy. A default configuration will be automatically provided by this monitoring module. The default configuration will ship logs of all containers that set the alloy.enable=true option to Loki. Multiple definitions of this option will be merged together into a single file.

See https://grafana.com/docs/alloy/latest/get-started/configuration-syntax/

Type

strings concatenated with "\n"

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alloy.enable

Whether to enable Alloy.

Type

boolean

Default

true

Example

true

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.alloy.useSocketProxy

Whether to access the Podman socket through the read-only proxy for the monitoring stack. Will be enabled by default if the 'socket-proxy' stack is enabled.

Type

boolean

Default

config.nps.stacks.socket-proxy.enable

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.enable

Enable the monitoring stack. This stack provides monitoring services including Grafana, Loki, Alloy, and Prometheus. Configuration files for each service will be provided automatically to work out of the box.

Type

boolean

Default

false

Example

true

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.dashboards

List of paths to Grafana dashboard JSON files.

Type

list of absolute path

Default

[ ]

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.datasources

Datasource configuration for Grafana. Loki and Prometheus datasources will be automatically configured.

Type

YAML 1.1 value

Default

null

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.enable

Whether to enable Grafana.

Type

boolean

Default

true

Example

true

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.oidc.adminGroup

Users of this group will be assigned the Grafana 'Admin' role.

Type

string

Default

"grafana_admin"

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.oidc.clientSecretFile

The file containing the client secret for the OIDC client that will be registered in Authelia.

For examples on how to generate a client secret, see

https://www.authelia.com/integration/openid-connect/frequently-asked-questions/#client-secret

Type

absolute path

Example

config.sops.secrets."immich/authelia/client_secret".path"

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.oidc.clientSecretHash

The client secret hash. For examples on how to generate a client secret, see https://www.authelia.com/integration/openid-connect/frequently-asked-questions/#client-secret

The value can be passed in multiple ways:

1. As a literal string
2. As an absolute path to a file containing the hash (toFile)
3. As an absolute oath to a file containing the client_secret, in which case the hash will be automatically computed (toHash)
4. As null

If left unset (null), the client secret will be read from the file specified in the clientSecretFile option and hashed automatically before being passed to the Authelia container.

Type

null or string or (submodule)

Default

null

Example

# Literal String:
"$pbkdf2-sha512$310000$cbOAIWbfz3vCVXIPIp6d2A$J0klwULa6TvPRCU1HAfuKua/dMKTl8gbTYJz2N73ejGUu0LUGz/y3kwmJLuKuAYGg3WQOT0q9ZzVHHUvpKpgvQ"

# Client secret hash stored in a file
{ fromFile = config.sops.secrets."immich/client_secret_hash".path; }

# Client secret stored in a file: Hash will be computed dynamically
{ toHash = config.sops.secrets."immich/client_secret".path; }

# Null (default): Hash will be computed automatically based on the clientSecretFile option
# Equivalent to { toHash = cfg.oidc.clientSecretFile; }
null

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.oidc.enable

Whether to enable OIDC login with Authelia. This will register an OIDC client in Authelia and setup the necessary configuration.

For details, see:

• https://www.authelia.com/integration/openid-connect/clients/grafana/
• https://docs.mealie.io/documentation/getting-started/authentication/oidc-v2/
• https://docs.mealie.io/documentation/getting-started/installation/backend-config/#openid-connect-oidc

Type

boolean

Default

false

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.oidc.userGroup

Users of this group will be assigned the Grafana 'Viewer' role.

Type

string

Default

"grafana_user"

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.grafana.settings

Settings for Grafana. Will be written to the 'grafana.ini' file. See https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#configure-grafana

Type

attribute set of section of an INI file (attrs of INI atom (null, bool, int, float or string))

Default

{ }

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.loki.config

Configuration for Loki. A default configuration will be automatically provided by this monitoring module.

See https://grafana.com/docs/loki/latest/configuration/

Type

YAML 1.1 value

Default

{ }

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.loki.enable

Whether to enable Loki.

Type

boolean

Default

true

Example

true

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.podmanExporter.enable

Whether to enable Podman Metrics Exporter.

Type

boolean

Default

true

Example

true

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.podmanExporter.useSocketProxy

Whether to access the Podman socket through the read-only proxy for the monitoring stack. Will be enabled by default if the 'socket-proxy' stack is enabled.

Type

boolean

Default

config.nps.stacks.socket-proxy.enable

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.prometheus.enable

Whether to enable Prometheus.

Type

boolean

Default

true

Example

true

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.prometheus.rules

Alerting rule configuration for Prometheus. If provided, the rules will added to the rule_file setting.

See https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

Type

YAML 1.1 value

Default

{ }

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.prometheus.settings

Configuration for Prometheus. A default configuration will be automatically provided by this monitoring module.

See https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Type

YAML 1.1 value

Default

{ }

Declaration

<nps/modules/monitoring>

Container Extension

The monitoring stack adds the alloy container option to the existing services.podman.containers.<name> options. When enabled for a container, Alloy will scrape the container's logs and ship them to Loki.

By default log collection is disabled unless nps.stacks.monitoring.alloy.collectByDefault is set to true. When collecting by default, individual containers can opt out by setting alloy.enable = false.

Example:

{
  nps.stacks.monitoring.enable = true;

  # Collect logs from this container
  nps.stacks.streaming.containers.jellyfin.alloy.enable = true;
}

services.podman.containers.<name>.alloy.enable

Alloy Log Scraping. If enabled, Alloy will scrape logs from the container and ship them to Loki.;

Type

boolean

Default

config.nps.stacks.monitoring.alloy.collectByDefault

Declaration

<nps/modules/monitoring/extension.nix>

Container Aliases

nps.stacks.monitoring.containers.alertmanager

Alias of {option}services.podman.containers.alertmanager.

Type

submodule

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.containers.alertmanager-ntfy

Alias of {option}services.podman.containers.alertmanager-ntfy.

Type

submodule

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.containers.alloy

Alias of {option}services.podman.containers.alloy.

Type

submodule

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.containers.grafana

Alias of {option}services.podman.containers.grafana.

Type

submodule

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.containers.loki

Alias of {option}services.podman.containers.loki.

Type

submodule

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.containers.podman-exporter

Alias of {option}services.podman.containers.podman-exporter.

Type

submodule

Declaration

<nps/modules/monitoring>

nps.stacks.monitoring.containers.prometheus

Alias of {option}services.podman.containers.prometheus.

Type

submodule

Declaration

<nps/modules/monitoring>