Q&A - Unraid > Vector > VictoriaLogs: [Files checkpointed.] is being suppressed to avoid flooding.

[tknx]

Question

Noticed logs weren't getting updated in VictoriaLogs and went to check Vector... Looks like it is concerned about flooding?

Setup is

Unraid system running Vector as a docker sending to a Raspberry Pi with a VictoriaLogs docker.

It worked initially so I think the pipeline is essentially correct but ran into something or other. Of course if the config can be better, I will take the advice.

Vector Config

[sources.syslog]
type = "file"
include = ["/var/log/syslog"]
ignore_older = 86400  # optional, ignores very old logs
read_from = "beginning"  # or "end"

[transforms.to_syslog]
type = "remap"
inputs = ["syslog"]
source = '''
.message = string!(.message)
'''

[sinks.vlogs]
type = "elasticsearch"
inputs = ["syslog"]
endpoints = ["http://10.0.0.6:9428/insert/elasticsearch/"]
api_version = "v8"
compression = "gzip"

[sinks.vlogs.healthcheck]
enabled = false

[sinks.vlogs.query]
_msg_field = "message"
_time_field = "timestamp"
_stream_fields = ["host", "container_name"]

Vector Logs

2025-07-17T01:15:35.899122Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:15:36.900404Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2025-07-17T01:15:39.595310Z DEBUG sink{component_kind="sink" component_id=vlogs component_type=elasticsearch}: vector::utilization: utilization=0.000000019052839811759045
2025-07-17T01:15:45.707370Z DEBUG source{component_kind="source" component_id=syslog component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.9160848e-5, "other": 1.4576959e-5, "reading": 1.2884675e-7, "sending": 2.804369e-6, "sleeping": 0.99995327}
2025-07-17T01:15:45.910960Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2025-07-17T01:15:45.910968Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:15:46.912275Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2025-07-17T01:15:54.594986Z DEBUG sink{component_kind="sink" component_id=vlogs component_type=elasticsearch}: vector::utilization: utilization=0.000000020024270555202636
2025-07-17T01:15:55.921666Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2025-07-17T01:15:55.921674Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:15:55.952830Z DEBUG source{component_kind="source" component_id=syslog component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.2901388e-5, "other": 1.2044814e-5, "reading": 1.17613695e-7, "sending": 2.3310938e-6, "sleeping": 0.99996257}
2025-07-17T01:15:56.922484Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2025-07-17T01:16:04.595755Z DEBUG sink{component_kind="sink" component_id=vlogs component_type=elasticsearch}: vector::utilization: utilization=0.000000018737466331654798
2025-07-17T01:16:05.932342Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2025-07-17T01:16:05.932350Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:16:06.199745Z DEBUG source{component_kind="source" component_id=syslog component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.4792713e-5, "other": 1.2993026e-5, "reading": 2.0357412e-7, "sending": 2.6063537e-6, "sleeping": 0.9999594}
2025-07-17T01:16:06.932834Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2025-07-17T01:16:15.943941Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2025-07-17T01:16:15.943950Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:16:16.446526Z DEBUG source{component_kind="source" component_id=syslog component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.4501524e-5, "other": 1.2510309e-5, "reading": 1.6727255e-7, "sending": 2.3567472e-6, "sleeping": 0.99996054}
2025-07-17T01:16:16.945417Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2025-07-17T01:16:19.595674Z DEBUG sink{component_kind="sink" component_id=vlogs component_type=elasticsearch}: vector::utilization: utilization=0.000000021862836538908218
2025-07-17T01:16:25.954919Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2025-07-17T01:16:25.954929Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:16:26.690983Z DEBUG source{component_kind="source" component_id=syslog component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.4411524e-5, "other": 1.3692912e-5, "reading": 1.2904582e-7, "sending": 2.4671958e-6, "sleeping": 0.9999593}
2025-07-17T01:16:26.956406Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2025-07-17T01:16:34.595329Z DEBUG sink{component_kind="sink" component_id=vlogs component_type=elasticsearch}: vector::utilization: utilization=0.00000002257610999645052
2025-07-17T01:16:35.964997Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2025-07-17T01:16:35.965006Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2025-07-17T01:16:36.937371Z DEBUG source{component_kind="source" component_id=syslog component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 2.5339059e-5, "other": 1.1623548e-5, "reading": 1.4307527e-7, "sending": 2.291449e-6, "sleeping": 0.99996054}
2025-07-17T01:16:36.966674Z DEBUG vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.

[pront]

Hi @tknx, suppression logs are very normal. They are there to reduce noise. However, this:

event_throughput=0.000/sec bytes_throughput=0.000/sec

This indicates Vector is not seeing any new Syslog data.

We can also see very low utilization:

utilization=0.00000002257610999645052

Based on the above I think the root cause is somewhere upstream, outside of Vector.