Observability

ib-interface uses OpenTelemetry to export structured log data to SigNoz or any OTLP-compatible backend.

Architecture

Python's standard logging module is bridged to OpenTelemetry via LoggingHandler. All log records emitted under the ib_insync logger are captured, enriched with OTel resource attributes, and exported.

Setup

1. Start SigNoz

Run SigNoz locally with Docker Compose:

git clone -b main https://github.com/SigNoz/signoz.git && cd signoz/deploy
docker compose -f docker/clickhouse-setup/docker-compose.yaml up -d

SigNoz UI will be available at http://localhost:3301.

2. Enable telemetry in ib-interface

from ib_interface.telemetry import setup_telemetry, shutdown_telemetry

setup_telemetry(
    service_name="ib-interface",
    otlp_endpoint="localhost:4317",
)

Call shutdown_telemetry() on exit to flush pending log records.

If otlp_endpoint is omitted, logs are written to the console in OTel format (useful for local development).

3. Verify

After connecting to TWS/Gateway, check the SigNoz Logs tab. Filter by service.name = ib-interface to see log entries.

Instrumented Events

Event	Logger	Level	Key Attributes
Connection established	ib_insync.ib	INFO	host, port, clientId
Connection lost	ib_insync.ib	WARNING	reason
Error callback	ib_insync.wrapper	ERROR/WARNING	reqId, errorCode, errorString, errorTime
Order placed	ib_insync.ib	INFO	orderId, action, orderType, totalQuantity
Order status change	ib_insync.wrapper	INFO	orderId, status, filled, remaining
Order fill	ib_insync.wrapper	INFO	orderId, avgFillPrice
Protocol type detected	ib_insync.decoder	DEBUG	protocol (text/protobuf), msgId

Dashboard Configuration

Create a SigNoz dashboard with the following panels:

Connection Health

• Query: Filter logs where body contains "connect" or "disconnect"
• Visualization: Timeline showing connection/disconnection events
• Purpose: Monitor connection stability over time

Error Rate

• Query: Filter logs with severity_number >= 13 (WARNING and above)
• Visualization: Bar chart grouped by 5-minute intervals
• Aggregation: Count of error/warning log records
• Purpose: Detect error spikes that may indicate TWS/Gateway issues

Order Latency

• Query: Filter logs where body contains "order" or "fill"
• Visualization: Timeline of order placement to fill events
• Group by: orderId attribute
• Purpose: Track order execution performance

Protocol Distribution

• Query: Filter logs from ib_insync.decoder where body contains "protocol"
• Visualization: Pie chart of text vs protobuf message counts
• Purpose: Verify protobuf adoption and monitor protocol mix

Alerting Rules

Configure alerts in SigNoz under Settings > Alert Rules.

Connection Drop Alert

Field	Value
Name	IB Connection Lost
Query	service.name = ib-interface AND severity_text = WARNING AND body CONTAINS "disconnect"
Condition	Count > 0 in 1-minute window
Severity	Critical
Channel	Slack / PagerDuty / Email
Description	TWS/Gateway connection was lost. Check network and TWS status.

Error Spike Alert

Field	Value
Name	IB Error Spike
Query	service.name = ib-interface AND severity_number >= 17 (ERROR+)
Condition	Count > 10 in 5-minute window
Severity	Warning
Channel	Slack / Email
Description	Elevated error rate detected. Review error codes in SigNoz logs.

No Activity Alert

Field	Value
Name	IB No Log Activity
Query	service.name = ib-interface
Condition	Count = 0 in 15-minute window
Severity	Warning
Channel	Slack / Email
Description	No log records received. Service may be down or telemetry misconfigured.

Environment Variables

SigNoz and OTel can also be configured via environment variables:

Variable	Default	Description
OTEL_SERVICE_NAME	ib-interface	Service name in SigNoz
OTEL_EXPORTER_OTLP_ENDPOINT	(none)	OTLP collector endpoint
OTEL_EXPORTER_OTLP_INSECURE	true	Use insecure gRPC for local dev

These are standard OTel environment variables and are recognized by the OpenTelemetry SDK. The setup_telemetry() function parameters take precedence when provided.