Telemetry

Gather relevant insights about your application and modules with custom metrics and telemetry.

Overview

The telemetry package provides observability tooling for Cosmos SDK applications using OpenTelemetry. It offers a unified initialization point for traces, metrics, and logs via the OpenTelemetry declarative configuration API.

This package:

• Initializes OpenTelemetry SDK using YAML configuration files
• Provides backward compatibility with Cosmos SDK’s legacy go-metrics wrapper API
• Includes built-in instrumentation for host, runtime, and disk I/O metrics

Quick Start

1. Start a Local Telemetry Backend

docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

2. Create Configuration File

Create an otel.yaml file:

file_format: "1.0-rc.3"
resource:
  attributes:
    - name: service.name
      value: my-cosmos-app

tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

meter_provider:
  readers:
    - pull:
        exporter:
          prometheus/development:
            host: 0.0.0.0
            port: 9464

logger_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

extensions:
  instruments:
    host: {}
    runtime: {}
    diskio: {}
  propagators:
    - tracecontext

3. Initialize Telemetry

Option A: Environment Variable (Recommended)

Set OTEL_EXPERIMENTAL_CONFIG_FILE to your config path. This initializes the SDK before any meters/tracers are created, avoiding atomic load overhead.

export OTEL_EXPERIMENTAL_CONFIG_FILE=/path/to/otel.yaml

Option B: Node Config Directory

An empty otel.yaml will now be generated in ~/.<node_home>/config/. Place the desired configuration in otel.yaml.

Option C: Programmatic Initialization

The SDK will first attempt to initialize via env var, then using the config in the node’s home directory. You may optionally initialize telemetry yourself using the telemetry.InitializeOpenTelemetry function:

err := telemetry.InitializeOpenTelemetry("/path/to/otel.yaml")
if err != nil {
    log.Fatal(err)
}
defer telemetry.Shutdown(context.Background())

Configuration

OpenTelemetry Configuration

The package uses the OpenTelemetry declarative configuration spec. Key sections:

Section	Purpose
resource	Service identity and attributes
tracer_provider	Trace export configuration
meter_provider	Metrics export configuration
logger_provider	Log export configuration

For examples containing available options, see the OpenTelemetry configuration examples.

Extensions

The extensions section of the otel.yaml configuration file provides additional features not yet supported by the standard otelconf:

extensions:
  # Optional file-based exporters
  trace_file: "/path/to/traces.json"
  metrics_file: "/path/to/metrics.json"
  metrics_file_interval: "10s"
  logs_file: "/path/to/logs.json"

  # Custom instrumentation additions
  instruments:
    host: {}
    runtime: {}
    diskio:
      disable_virtual_device_filter: true # removes the automatic filtering of virtual disks. Operating systems such as Linux typically add virtual disks, which can add duplication to disk io data. These disks usually take the form of loopback, RAID, partitions, etc.

  # Trace context propagation
  propagators:
    - tracecontext
    - baggage
    - b3
    - jaeger

Custom Instruments

Host Instrumentation (host)

Reports host-level metrics using go.opentelemetry.io/contrib/instrumentation/host:

• CPU usage
• Memory usage
• Network I/O

extensions:
  instruments:
    host: {}

Runtime Instrumentation (runtime)

Reports Go runtime metrics using go.opentelemetry.io/contrib/instrumentation/runtime:

• Goroutine count
• GC statistics
• Memory allocations

extensions:
  instruments:
    runtime: {}

Disk I/O Instrumentation (diskio)

Reports disk I/O metrics using gopsutil:

Metric	Description
system.disk.io	Bytes read/written
system.disk.operations	Read/write operation counts
system.disk.io_time	Time spent on I/O operations
system.disk.operation_time	Time per read/write operation
system.disk.merged	Merged read/write operations

extensions:
  instruments:
    diskio: {}
    # Or with options:
    diskio:
      disable_virtual_device_filter: true  # Include loopback, RAID, partitions on Linux

By default, virtual devices (loopback, RAID, partitions) are filtered out on Linux to avoid double-counting I/O.

Propagators

Configure trace context propagation for distributed tracing:

Propagator	Description
tracecontext	W3C Trace Context (default)
baggage	W3C Baggage
b3	Zipkin B3 single header
b3multi	Zipkin B3 multi-header
jaeger	Jaeger propagation

Developer Usage

Using Meters and Tracers

After initialization, use standard OpenTelemetry APIs:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric"
)

var (
    tracer = otel.Tracer("my-package")
    meter  = otel.Meter("my-package")

    myCounter metric.Int64Counter
)

func init() {
   var err error
   myCounter, err = meter.Int64Counter("my.counter")
   if err != nil {
	   panic(err)
   }
}

func MyFunction(ctx context.Context) error {
    ctx, span := tracer.Start(ctx, "MyFunction")
    defer span.End()

    myCounter.Add(ctx, 1)

    // ... your code
    return nil
}

Shutdown

Always call Shutdown() when the application exits:

func (a *App) Close() {
    telemetry.Shutdown(ctx)
}

Legacy API (Deprecated)

The package provides backward-compatible wrappers for github.com/hashicorp/go-metrics. These are deprecated and users should migrate to OpenTelemetry APIs directly.

OpenTelemetry Bridge

Cosmos SDK v0.54.0+ provides a bridge to send existing go-metrics to the meter provider defined in your OpenTelemetry config. To bridge your metrics, set the metrics-sink in app.toml to “otel”.

###############################################################################
###                         Telemetry Configuration                         ###
###############################################################################
[telemetry]

# other fields...

metrics-sink = "otel"

Legacy Configuration

cfg := telemetry.Config{
    ServiceName:            "my-service",
    Enabled:                true,
    EnableHostname:         true,
    EnableHostnameLabel:    true,
    EnableServiceLabel:     true,
    PrometheusRetentionTime: 60,  // seconds
    GlobalLabels:           [][]string{{"chain_id", "cosmoshub-1"}},
    MetricsSink:            "otel",  // "mem", "statsd", "dogstatsd", "otel"
    StatsdAddr:             "localhost:8125",
}

m, err := telemetry.New(cfg)

Legacy Metrics Functions

All are deprecated; prefer OpenTelemetry:

// Counters
telemetry.IncrCounter(1.0, "tx", "count")
telemetry.IncrCounterWithLabels([]string{"tx", "count"}, 1.0, labels)

// Gauges
telemetry.SetGauge(42.0, "mempool", "size")
telemetry.SetGaugeWithLabels([]string{"mempool", "size"}, 42.0, labels)

// Timing
start := telemetry.Now()
// ... operation
telemetry.MeasureSince(start, "tx", "process_time")

// Module-specific helpers
telemetry.ModuleMeasureSince("bank", start, "send", "time")
telemetry.ModuleSetGauge("bank", 100.0, "balance", "total")

Metrics Sink Types

Sink	Description
mem	In-memory sink with SIGUSR1 dump support (default)
statsd	StatsD protocol
dogstatsd	Datadog DogStatsD
otel	OpenTelemetry (bridges to configured MeterProvider)

Best Practices

1. Use environment variable initialization for production to avoid atomic load overhead
2. Always call Shutdown() to ensure metrics/traces are flushed
3. Thread context.Context properly for correct span correlation

Viewing Telemetry Data

With Grafana LGTM running:

1. Open http://localhost:3000
2. Use the Drilldown views to explore:
   • Traces: Distributed trace visualization
   • Metrics: Query and dashboard metrics
   • Logs: Structured log search

Related Documentation

• OpenTelemetry Go SDK
• OpenTelemetry Configuration Spec
• otelconf Go Package

Cosmos SDK Metrics

The following metrics are emitted from the Cosmos SDK.

Metric	Description	Unit	Type
tx_count	Total number of txs processed via FinalizeBlock	tx	counter
tx_successful	Total number of successful txs processed via FinalizeBlock	tx	counter
tx_failed	Total number of failed txs processed via FinalizeBlock	tx	counter
tx_gas_used	The total amount of gas used by a tx	gas	gauge
tx_gas_wanted	The total amount of gas requested by a tx	gas	gauge
tx_msg_send	The total amount of tokens sent in a MsgSend (per denom)	token	gauge
tx_msg_withdraw_reward	The total amount of tokens withdrawn in a MsgWithdrawDelegatorReward (per denom)	token	gauge
tx_msg_withdraw_commission	The total amount of tokens withdrawn in a MsgWithdrawValidatorCommission (per denom)	token	gauge
tx_msg_delegate	The total amount of tokens delegated in a MsgDelegate	token	gauge
tx_msg_begin_unbonding	The total amount of tokens undelegated in a MsgUndelegate	token	gauge
tx_msg_begin_begin_redelegate	The total amount of tokens redelegated in a MsgBeginRedelegate	token	gauge
tx_msg_ibc_transfer	The total amount of tokens transferred via IBC in a MsgTransfer (source or sink chain)	token	gauge
ibc_transfer_packet_receive	The total amount of tokens received in a FungibleTokenPacketData (source or sink chain)	token	gauge
new_account	Total number of new accounts created	account	counter
gov_proposal	Total number of governance proposals	proposal	counter
gov_vote	Total number of governance votes for a proposal	vote	counter
gov_deposit	Total number of governance deposits for a proposal	deposit	counter
staking_delegate	Total number of delegations	delegation	counter
staking_undelegate	Total number of undelegations	undelegation	counter
staking_redelegate	Total number of redelegations	redelegation	counter
ibc_transfer_send	Total number of IBC transfers sent from a chain (source or sink)	transfer	counter
ibc_transfer_receive	Total number of IBC transfers received to a chain (source or sink)	transfer	counter
ibc_client_create	Total number of clients created	create	counter
ibc_client_update	Total number of client updates	update	counter
ibc_client_upgrade	Total number of client upgrades	upgrade	counter
ibc_client_misbehaviour	Total number of client misbehaviors	misbehaviour	counter
ibc_connection_open-init	Total number of connection OpenInit handshakes	handshake	counter
ibc_connection_open-try	Total number of connection OpenTry handshakes	handshake	counter
ibc_connection_open-ack	Total number of connection OpenAck handshakes	handshake	counter
ibc_connection_open-confirm	Total number of connection OpenConfirm handshakes	handshake	counter
ibc_channel_open-init	Total number of channel OpenInit handshakes	handshake	counter
ibc_channel_open-try	Total number of channel OpenTry handshakes	handshake	counter
ibc_channel_open-ack	Total number of channel OpenAck handshakes	handshake	counter
ibc_channel_open-confirm	Total number of channel OpenConfirm handshakes	handshake	counter
ibc_channel_close-init	Total number of channel CloseInit handshakes	handshake	counter
ibc_channel_close-confirm	Total number of channel CloseConfirm handshakes	handshake	counter
tx_msg_ibc_recv_packet	Total number of IBC packets received	packet	counter
tx_msg_ibc_acknowledge_packet	Total number of IBC packets acknowledged	acknowledgement	counter
ibc_timeout_packet	Total number of IBC timeout packets	timeout	counter
store_iavl_get	Duration of an IAVL Store#Get call	ms	summary
store_iavl_set	Duration of an IAVL Store#Set call	ms	summary
store_iavl_has	Duration of an IAVL Store#Has call	ms	summary
store_iavl_delete	Duration of an IAVL Store#Delete call	ms	summary
store_iavl_commit	Duration of an IAVL Store#Commit call	ms	summary
store_iavl_query	Duration of an IAVL Store#Query call	ms	summary