fix: close idle OTLP connections before the collector does #184

Merged

argoyle merged 1 commits from fix-otlp-idle-connections into main

2026-09-19 13:51:38 +00:00

Author	SHA1	Message	Date
argoyleandClaude Opus 5	f1551a5f47	fix: close idle OTLP connections before the collector does otelsetup / test (push) Skipped otelsetup / vulnerabilities (push) Skipped pre-commit / pre-commit (push) Skipped otelsetup / vulnerabilities (pull_request) Successful in 1m0s Details otelsetup / test (pull_request) Successful in 1m21s Details pre-commit / pre-commit (pull_request) Successful in 3m36s Details Metrics are pushed every 60s and Alloy's OTLP receiver closes connections idle for 1m, while the exporters keep them for 90s. A push could reuse a connection the collector was closing and fail with EOF or connection reset; the data was dropped (no retry for a POST or for transport errors). In prod that was ~3 failed metric pushes per pod per hour, plus occasional trace batches. Both exporters now use an HTTP client that closes idle connections after 30s. A custom client makes the exporters ignore the OTLP timeout and certificate env vars; none are set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rfQ5HJ7zwfuYQcWXd3Mm3	2026-09-19 15:43:14 +02:00

Author

SHA1

Message

Date

argoyleandClaude Opus 5

f1551a5f47

fix: close idle OTLP connections before the collector does

otelsetup / test (push) Skipped

otelsetup / vulnerabilities (push) Skipped

pre-commit / pre-commit (push) Skipped

otelsetup / vulnerabilities (pull_request) Successful in 1m0s

Details

otelsetup / test (pull_request) Successful in 1m21s

Details

pre-commit / pre-commit (pull_request) Successful in 3m36s

Details

Metrics are pushed every 60s and Alloy's OTLP receiver closes connections idle for 1m, while the exporters keep them for 90s. A push could reuse a connection the collector was closing and fail with EOF or connection reset; the data was dropped (no retry for a POST or for transport errors). In prod that was ~3 failed metric pushes per pod per hour, plus occasional trace batches.

Both exporters now use an HTTP client that closes idle connections after 30s. A custom client makes the exporters ignore the OTLP timeout and certificate env vars; none are set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rfQ5HJ7zwfuYQcWXd3Mm3

2026-09-19 15:43:14 +02:00