Add WebSocket failover counter metric and URL change logging #661
Conversation
… URL change logging
NPM Publishing labels 🏷️🛑 This PR needs labels to indicate how to increase the current package version in the automated workflows. Please add one of the following labels: |
cawthorne
changed the title
| logger.info('Websocket URL has changed, closing connection to reconnect...') | ||
| censorLogs(() => | ||
| logger.debug( | ||
| `Websocket url has changed from ${this.currentUrl} to ${urlFromConfig}, closing connection...`, |
There was a problem hiding this comment.
Does this no longer close the connection?
There was a problem hiding this comment.
It does, but that string got moved to the info level log, so it is always visible with info level logs.
| }), | ||
| wsConnectionFailoverCount: new client.Gauge({ | ||
| name: 'ws_connection_failover_count', | ||
| help: 'The number of consecutive connection issues (unresponsive/no data, abnormal closures), used to trigger URL failover. Resets to 0 when data flows successfully.', |
There was a problem hiding this comment.
I don't see where this is reset to 0.
There was a problem hiding this comment.
It doesn't. The underlying variable it is meant to expose streamHandlerInvocationsWithNoConnection also never resets. It just increments forever, and Tiingo uses modulo arithmetic on it, it is used in this PR:
https://github.com/smartcontractkit/external-adapters-js/pull/4543/files (even before my changes).
Open to resetting it if there is a good reason.
There was a problem hiding this comment.
I don't know if it should reset, but the description says "Resets to 0 when data flows successfully."
2 participants
Summary
Adds observability for WebSocket failover mechanism to help diagnose connection issues.
Problem
During a Tiingo incident (2026-01-13 03:19-03:32 UTC), we could not determine if failover triggered:
streamHandlerInvocationsWithNoConnectioncounter not exposed as metricCENSOR_SENSITIVE_LOGS=trueThis made it impossible to answer:
Changes
1. New Prometheus Metric
ws_connection_failover_countgauge metricstreamHandlerInvocationsWithNoConnectionvalue in real-timetransport_namefor per-transport tracking