Skip to content

Grafana Alloy doesn't create systemd service file after installation via UI #3359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Aryston opened this issue Apr 16, 2025 · 0 comments
Open

Grafana Alloy doesn't create systemd service file after installation via UI #3359

Aryston opened this issue Apr 16, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@Aryston
Copy link

Aryston commented Apr 16, 2025

What's wrong?

After successfully installing Grafana Alloy via the "Connections" -> "Add new connection" -> "Linux Server" method within the Grafana UI, a systemd service file for grafana-alloy is not created. This results in Alloy not starting automatically after a system reboot. Consequently, users have to manually start Alloy after each restart.

Steps to reproduce

  1. Navigate to "Connections" in the Grafana UI.
  2. Click on "Add new connection".
  3. Select "Linux Server".
  4. Follow the prompts to install Grafana Alloy on a Fedora 41 or Fedora 42 system.
  5. Once the installation is reported as successful in the UI, reboot the Linux server.
  6. After the reboot, check for the existence of a grafana-alloy.service file (e.g., in /etc/systemd/system/).
  7. Observe that the service file is missing.
  8. Attempt to check the status of the grafana-alloy service (e.g., using systemctl status grafana-alloy). Observe that the service cannot be found.

System information

Linux PTX-PC-178 6.14.2-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 21:50:55 UTC 2025 x86_64 GNU/Linux

Software version

alloy, version v1.8.1 (branch: HEAD, revision: dc3b14b)

Configuration

$ egrep -v username\|password\|id config.alloy 
remotecfg {
	url            = "https://fleet-management-prod-023.grafana.net"
	poll_frequency = "60s"

	basic_auth {
	}
}

prometheus.remote_write "metrics_service" {
	endpoint {
		url = "https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push"

		basic_auth {
		}
	}
}

loki.write "grafana_cloud_loki" {
	endpoint {
		url = "https://logs-prod-035.grafana.net/loki/api/v1/push"

		basic_auth {
		}
	}
}

discovery.relabel "integrations_node_exporter" {
  targets = prometheus.exporter.unix.integrations_node_exporter.targets

  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }

  rule {
    target_label = "job"
    replacement = "integrations/node_exporter"
  }
}

prometheus.exporter.unix "integrations_node_exporter" {
  disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]

  filesystem {
    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
    mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
    mount_timeout        = "5s"
  }

  netclass {
    ignored_devices = "^(veth.*|cali.*|[a-f0-9]{15})$"
  }

  netdev {
    device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
  }
}

prometheus.scrape "integrations_node_exporter" {
  targets    = discovery.relabel.integrations_node_exporter.output
  forward_to = [prometheus.relabel.integrations_node_exporter.receiver]
}

prometheus.relabel "integrations_node_exporter" {
  forward_to = [prometheus.remote_write.metrics_service.receiver]

  rule {
    source_labels = ["__name__"]
    regex         = "node_scrape_collector_.+"
    action        = "drop"
  }
}

loki.relabel "integrations_node_exporter" {
  forward_to = [loki.write.grafana_cloud_loki.receiver]
  rule {
    target_label = "job"
    replacement  = "integrations/node_exporter"
  }
  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }
}

journal_module "integrations_node_exporter" {
  forward_to = [loki.relabel.integrations_node_exporter.receiver]
}

//JOURNAL
declare "journal_module" {
  argument "forward_to" {
      optional = false
  }

  loki.source.journal "default"  {
      max_age       = "12h0m0s"
      forward_to    = [loki.process.default.receiver]
      relabel_rules = loki.relabel.default.rules
  }

  loki.relabel "default" {
      rule {
          source_labels = ["__journal__systemd_unit"]
          target_label  = "unit"
      }
      rule {
      }
      rule {
          source_labels = ["__journal__transport"]
          target_label  = "transport"
      }
      rule {
          source_labels = ["__journal_priority_keyword"]
          target_label  = "level"
      }
      forward_to    = []
  }
  loki.process "default" {
      forward_to    = argument.forward_to.value
  }
}

Logs

$ ./alloy-linux-amd64 run ./config.alloy
ts=2025-04-16T14:01:05.331690541Z level=info "boringcrypto enabled"=false
ts=2025-04-16T14:01:05.323244033Z level=info source=/go/pkg/mod/github.com/!kim!machine!gun/[email protected]/memlimit/memlimit.go:170 msg="memory is not limited, skipping" package=github.com/KimMachineGun/automemlimit/memlimit
ts=2025-04-16T14:01:05.331745944Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster
ts=2025-04-16T14:01:05.331757731Z level=info msg="running usage stats reporter"
ts=2025-04-16T14:01:05.331762167Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b
ts=2025-04-16T14:01:05.33177206Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=otel duration=1.317µs
ts=2025-04-16T14:01:05.331782464Z level=info msg="replaying WAL, this may take a while" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=wal dir=data-alloy/prometheus.remote_write.metrics_service/wal
ts=2025-04-16T14:01:05.331792629Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=wal segment=0 maxSegment=4
ts=2025-04-16T14:01:05.331798998Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=wal segment=1 maxSegment=4
ts=2025-04-16T14:01:05.331804474Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=wal segment=2 maxSegment=4
ts=2025-04-16T14:01:05.331812208Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=wal segment=3 maxSegment=4
ts=2025-04-16T14:01:05.331817427Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=wal segment=4 maxSegment=4
ts=2025-04-16T14:01:05.331822662Z level=info msg="Starting WAL watcher" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=rw remote_name=f90047 url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push queue=f90047
ts=2025-04-16T14:01:05.331835179Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=rw remote_name=f90047 url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push
ts=2025-04-16T14:01:05.33188655Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=prometheus.remote_write.metrics_service duration=6.727755ms
ts=2025-04-16T14:01:05.331897001Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=rw remote_name=f90047 url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push queue=f90047
ts=2025-04-16T14:01:05.331903941Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=prometheus.relabel.integrations_node_exporter duration=919.941µs
ts=2025-04-16T14:01:05.331910673Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=tracing duration=13.205µs
ts=2025-04-16T14:01:05.331922441Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=logging duration=241.804µs
ts=2025-04-16T14:01:05.331957747Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=labelstore duration=18.773µs
ts=2025-04-16T14:01:05.332471016Z level=info msg="Parsed flag --collector.diskstats.device-exclude" component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=diskstats flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2025-04-16T14:01:05.333471046Z level=info msg="Parsed flag --collector.netdev.device-exclude" component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=netdev flag=^(veth.*|cali.*|[a-f0-9]{15})$
ts=2025-04-16T14:01:05.333557708Z level=info msg="Parsed flag --collector.filesystem.mount-points-exclude" component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)
ts=2025-04-16T14:01:05.333578368Z level=info msg="Parsed flag --collector.filesystem.fs-types-exclude" component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2025-04-16T14:01:05.333772868Z level=info msg="Enabled node_exporter collectors" component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter
ts=2025-04-16T14:01:05.333803652Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=arp
ts=2025-04-16T14:01:05.333822177Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=bcache
ts=2025-04-16T14:01:05.333832982Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=bonding
ts=2025-04-16T14:01:05.333842759Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=conntrack
ts=2025-04-16T14:01:05.333852615Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=cpu
ts=2025-04-16T14:01:05.333867573Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=cpufreq
ts=2025-04-16T14:01:05.333877207Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=diskstats
ts=2025-04-16T14:01:05.333893423Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=dmi
ts=2025-04-16T14:01:05.333904258Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=edac
ts=2025-04-16T14:01:05.333916465Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=entropy
ts=2025-04-16T14:01:05.333926004Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=fibrechannel
ts=2025-04-16T14:01:05.333935496Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=filefd
ts=2025-04-16T14:01:05.333950608Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=filesystem
ts=2025-04-16T14:01:05.333960527Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=hwmon
ts=2025-04-16T14:01:05.333970101Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=loadavg
ts=2025-04-16T14:01:05.333979621Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=mdadm
ts=2025-04-16T14:01:05.333995467Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=meminfo
ts=2025-04-16T14:01:05.334005215Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=netclass
ts=2025-04-16T14:01:05.334017412Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=netdev
ts=2025-04-16T14:01:05.334027011Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=netstat
ts=2025-04-16T14:01:05.334039095Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=nfs
ts=2025-04-16T14:01:05.334048536Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=nfsd
ts=2025-04-16T14:01:05.334057986Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=nvme
ts=2025-04-16T14:01:05.334069821Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=os
ts=2025-04-16T14:01:05.334079365Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=powersupplyclass
ts=2025-04-16T14:01:05.334088843Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=pressure
ts=2025-04-16T14:01:05.334107303Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=processes
ts=2025-04-16T14:01:05.334119729Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=rapl
ts=2025-04-16T14:01:05.334129538Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=schedstat
ts=2025-04-16T14:01:05.334139223Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=selinux
ts=2025-04-16T14:01:05.334148806Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=sockstat
ts=2025-04-16T14:01:05.334160646Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=softnet
ts=2025-04-16T14:01:05.334170298Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=stat
ts=2025-04-16T14:01:05.334179837Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=tapestats
ts=2025-04-16T14:01:05.334195464Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=textfile
ts=2025-04-16T14:01:05.334205246Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=thermal_zone
ts=2025-04-16T14:01:05.334215132Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=time
ts=2025-04-16T14:01:05.334224672Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=timex
ts=2025-04-16T14:01:05.334236758Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=udp_queues
ts=2025-04-16T14:01:05.334246306Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=uname
ts=2025-04-16T14:01:05.33425589Z level=info component_path=/ component_id=prometheus.exporter.unix.integrations_node_exporter collector=vmstat
ts=2025-04-16T14:01:05.334293575Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=prometheus.exporter.unix.integrations_node_exporter duration=2.321864ms
ts=2025-04-16T14:01:05.334438498Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=discovery.relabel.integrations_node_exporter duration=114.814µs
ts=2025-04-16T14:01:05.337903676Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=prometheus.scrape.integrations_node_exporter duration=3.421908ms
ts=2025-04-16T14:01:05.337941025Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=declare.journal_module duration=5.632µs
ts=2025-04-16T14:01:05.337964025Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=livedebugging duration=12.634µs
ts=2025-04-16T14:01:05.33839407Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=loki.write.grafana_cloud_loki duration=418.485µs
ts=2025-04-16T14:01:05.338564685Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=loki.relabel.integrations_node_exporter duration=153.961µs
ts=2025-04-16T14:01:05.338689494Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=b16598ed0f87f668982bafcbf6da5d22
ts=2025-04-16T14:01:05.338786388Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=b16598ed0f87f668982bafcbf6da5d22 node_id=loki.relabel.default duration=83.174µs
ts=2025-04-16T14:01:05.338807826Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=b16598ed0f87f668982bafcbf6da5d22 node_id=argument.forward_to duration=7.745µs
ts=2025-04-16T14:01:05.338927322Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=b16598ed0f87f668982bafcbf6da5d22 node_id=loki.process.default duration=109.179µs
ts=2025-04-16T14:01:05.340425863Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=b16598ed0f87f668982bafcbf6da5d22 node_id=loki.source.journal.default duration=1.484472ms
ts=2025-04-16T14:01:05.340468757Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=b16598ed0f87f668982bafcbf6da5d22 duration=1.818226ms
ts=2025-04-16T14:01:05.340484423Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=journal_module.integrations_node_exporter duration=1.90409ms
ts=2025-04-16T14:01:05.384732909Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=remotecfg duration=44.23338ms
ts=2025-04-16T14:01:05.384800721Z level=info msg="applying non-TLS config to HTTP server" service=http
ts=2025-04-16T14:01:05.384814431Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=http duration=34.378µs
ts=2025-04-16T14:01:05.384838059Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=ui duration=7.288µs
ts=2025-04-16T14:01:05.384857439Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b node_id=cluster duration=6.416µs
ts=2025-04-16T14:01:05.384870409Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=df86a8ff1078fd362a83add6ad4ab35b duration=61.010558ms
ts=2025-04-16T14:01:05.384987672Z level=info msg="scheduling loaded components and services"
ts=2025-04-16T14:01:05.385301335Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=3c2964b23cc56e3e251546c4f76216fd
ts=2025-04-16T14:01:05.385478875Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=3c2964b23cc56e3e251546c4f76216fd node_id=loki.relabel.default duration=148.449µs
ts=2025-04-16T14:01:05.385510276Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=3c2964b23cc56e3e251546c4f76216fd node_id=argument.forward_to duration=8.729µs
ts=2025-04-16T14:01:05.385543271Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=3c2964b23cc56e3e251546c4f76216fd node_id=loki.process.default duration=17.704µs
ts=2025-04-16T14:01:05.385587132Z level=info msg="finished node evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=3c2964b23cc56e3e251546c4f76216fd node_id=loki.source.journal.default duration=29.179µs
ts=2025-04-16T14:01:05.385568491Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s
ts=2025-04-16T14:01:05.385604298Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=journal_module.integrations_node_exporter trace_id=3c2964b23cc56e3e251546c4f76216fd duration=417.943µs
ts=2025-04-16T14:01:05.385688667Z level=info msg="scheduling loaded components and services"
ts=2025-04-16T14:01:05.385688045Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=PTX-PC-178
ts=2025-04-16T14:01:05.386116558Z level=info msg="now listening for http traffic" service=http addr=127.0.0.1:12345
ts=2025-04-16T14:01:05.395930328Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id=remotecfg trace_id=058ce3e5a98c5f7c68e983470fad05df
ts=2025-04-16T14:01:05.396251254Z level=info msg="starting complete graph evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad
ts=2025-04-16T14:01:05.396355334Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad node_id=prometheus.exporter.self.integrations_alloy_health duration=70.578µs
ts=2025-04-16T14:01:05.396621347Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad node_id=discovery.relabel.integrations_alloy_health duration=234.854µs
ts=2025-04-16T14:01:05.404560185Z level=info msg="replaying WAL, this may take a while" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=wal dir=data-alloy/remotecfg/self_monitoring_metrics.default/prometheus.remote_write.default/wal
ts=2025-04-16T14:01:05.406707917Z level=info msg="WAL segment loaded" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=wal segment=0 maxSegment=2
ts=2025-04-16T14:01:05.40902296Z level=info msg="WAL segment loaded" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=wal segment=1 maxSegment=2
ts=2025-04-16T14:01:05.409319589Z level=info msg="WAL segment loaded" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=wal segment=2 maxSegment=2
ts=2025-04-16T14:01:05.411276472Z level=info msg="Starting WAL watcher" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push queue=90589e
ts=2025-04-16T14:01:05.411322741Z level=info msg="Starting scraped metadata watcher" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push
ts=2025-04-16T14:01:05.411366131Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad node_id=prometheus.remote_write.default duration=14.711304ms
ts=2025-04-16T14:01:05.411519935Z level=info msg="Replaying WAL" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push queue=90589e
ts=2025-04-16T14:01:05.413077116Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad node_id=prometheus.relabel.integrations_alloy_health duration=1.661137ms
ts=2025-04-16T14:01:05.416724548Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad node_id=prometheus.scrape.integrations_alloy_health duration=3.589137ms
ts=2025-04-16T14:01:05.416790525Z level=info msg="finished complete graph evaluation" controller_path=/remotecfg controller_id=self_monitoring_metrics.default trace_id=7a56a53f033804a1cd5ea2bad580fcad duration=20.681723ms
ts=2025-04-16T14:01:05.416848578Z level=info msg="finished node evaluation" controller_path=/ controller_id=remotecfg trace_id=058ce3e5a98c5f7c68e983470fad05df node_id=self_monitoring_metrics.default duration=20.866051ms
ts=2025-04-16T14:01:05.417137978Z level=info msg="starting complete graph evaluation" controller_path=/remotecfg controller_id=self_monitoring_logs_linux.default trace_id=ec70ee59719e83993e59c2be980cc090
ts=2025-04-16T14:01:05.418327983Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_logs_linux.default trace_id=ec70ee59719e83993e59c2be980cc090 node_id=loki.write.grafana_cloud_loki duration=1.152336ms
ts=2025-04-16T14:01:05.420095947Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_logs_linux.default trace_id=ec70ee59719e83993e59c2be980cc090 node_id=loki.source.journal.alloy_logs_tag duration=1.715665ms
ts=2025-04-16T14:01:05.42087556Z level=info msg="finished node evaluation" controller_path=/remotecfg controller_id=self_monitoring_logs_linux.default trace_id=ec70ee59719e83993e59c2be980cc090 node_id=loki.source.journal.alloy_logs_unit duration=730.31µs
ts=2025-04-16T14:01:05.420918393Z level=info msg="finished complete graph evaluation" controller_path=/remotecfg controller_id=self_monitoring_logs_linux.default trace_id=ec70ee59719e83993e59c2be980cc090 duration=3.906107ms
ts=2025-04-16T14:01:05.420946705Z level=info msg="finished node evaluation" controller_path=/ controller_id=remotecfg trace_id=058ce3e5a98c5f7c68e983470fad05df node_id=self_monitoring_logs_linux.default duration=4.065282ms
ts=2025-04-16T14:01:05.420979068Z level=info msg="finished node evaluation" controller_path=/ controller_id=remotecfg trace_id=058ce3e5a98c5f7c68e983470fad05df node_id=declare.self_monitoring_metrics duration=5.13µs
ts=2025-04-16T14:01:05.421004775Z level=info msg="finished node evaluation" controller_path=/ controller_id=remotecfg trace_id=058ce3e5a98c5f7c68e983470fad05df node_id=declare.self_monitoring_logs_linux duration=3.124µs
ts=2025-04-16T14:01:05.421025482Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id=remotecfg trace_id=058ce3e5a98c5f7c68e983470fad05df duration=25.19883ms
ts=2025-04-16T14:01:05.437943113Z level=info msg="scheduling loaded components and services"
ts=2025-04-16T14:01:05.438085235Z level=info msg="scheduling loaded components and services"
ts=2025-04-16T14:01:05.438179525Z level=info msg="scheduling loaded components and services"
ts=2025-04-16T14:01:06.541158065Z level=error msg="final error sending batch" component_path=/remotecfg/self_monitoring_logs_linux.default component_id=loki.write.grafana_cloud_loki component=client host=logs-prod-035.grafana.net status=401 tenant="" error="server returned HTTP status 401 Unauthorized (401): {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:01:25.162752858Z level=info msg="Done replaying WAL" component_path=/ component_id=prometheus.remote_write.metrics_service subcomponent=rw remote_name=f90047 url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push duration=19.83196059s
ts=2025-04-16T14:01:35.376404891Z level=info msg="Done replaying WAL" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push duration=29.965013744s
ts=2025-04-16T14:01:35.439114331Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:02:35.45371671Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:03:35.470622456Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:04:35.486555464Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:05:35.500952995Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:06:35.513728575Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:07:35.527724799Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:08:35.541727004Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:09:35.55608605Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:10:35.570508284Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:11:35.584345237Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:12:35.599576272Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"

ts=2025-04-16T14:13:35.614383345Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:14:35.625871467Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:15:35.64082069Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
ts=2025-04-16T14:16:35.655540948Z level=error msg="non-recoverable error" component_path=/remotecfg/self_monitoring_metrics.default component_id=prometheus.remote_write.default subcomponent=rw remote_name=90589e url=https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push failedSampleCount=216 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: {\"status\":\"error\",\"error\":\"authentication error: invalid token\"}"
@Aryston Aryston added the bug Something isn't working label Apr 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Aryston