Using OpenTelemetry instrumentation

Install tomodachi using the opentelemetry extras to enable instrumentation for OpenTelemetry. In addition, install with the opentelemetry-exporter-prometheus extras to use Prometheus exporter metrics.

pip install tomodachi[opentelemetry]
pip install tomodachi[opentelemetry,opentelemetry-exporter-prometheus]

When added as a Poetry dependency the opentelemetry extras can be enabled by adding tomodachi = {extras = ["opentelemetry"]} to the pyproject.toml file, and when added to a requiements.txt file the opentelemetry extras can be enabled by adding tomodachi[opentelemetry] to the file.

Auto instrumentation: `tomodachi --opentelemetry-instrument`

Passing the --opentelemetry-instrument argument to tomodachi run will automatically instrument the service with the appropriate exporters and configuration according to the set set OTEL_* environment variables..

If tomodachi is installed in the environment, using tomodachi --opentelemetry-instrument service.py is mostly equivalent to running opentelemetry-instrument tomodachi run service.py and will load distros, configurators and instrumentors automatically in the same way as the opentelemetry-instrument CLI would do.

OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
    tomodachi --opentelemetry-instrument run service/app.py

Auto instrumention by setting env variable value: `TOMODACHI_OPENTELEMETRY_INSTRUMENT=1`

The environment variable TOMODACHI_OPENTELEMETRY_INSTRUMENT if set will also enable auto instrumentation in the same way.

OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 \
    tomodachi run service/app.py

Auto instrumentation using the `opentelemetry-instrument` CLI

Auto instrumentation using the opentelemetry-instrument CLI can be achieved by starting services using opentelemetry-instrument [otel-options] tomodachi run [options] <service.py ...>.

# either define the OTEL_* environment variables to specify instrumentation specification
OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
    opentelemetry-instrument tomodachi run service/app.py

# or use the arguments passed to the opentelemetry-instrument command
opentelemetry-instrument \
    --logs_exporter console \
    --traces_exporter console \
    --metrics_exporter console \
    --service_name example-service \
    tomodachi run service/app.py

Manual instrumentation

Auto instrumentation using either tomodachi --opentelemetry-instrument, setting the TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 env value or using the opentelemetry-instrument CLI are the recommended ways of instrumenting services, as they will automatically instrument the service (and libs with instrumentors installed) with the appropriate exporters and configuration.

However, instrumentation can also be enabled by importing the TomodachiInstrumentor instrumentation class and calling its' instrument function before defining the service class.

import tomodachi  
from tomodachi.opentelemetry import TomodachiInstrumentor

TomodachiInstrumentor().instrument()

class Service(tomodachi.Service):  
    name = "example-service"

    @tomodachi.http(GET, r"/example")
    async def example(self, request):
        return 200, "hello world"

Starting such a service with the appropriate OTEL_* environment variables would properly instrument traces, logs and metrics for the service without the need to use the opentelemetry-instrument CLI.

OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
    tomodachi run service/app.py

Service name dynamically set if missing `OTEL_SERVICE_NAME` value

If the OTEL_SERVICE_NAME environment variable value (or --service_name argument to opentelemetry-instrument) is not set, the resource' service.name will instead be set to the name attribute of the service class.

In case the service class uses the default generic names ("service" or "app"), the resource' service.name will instead be set to the default as specified in https://github.com/open-telemetry/semantic-conventions/tree/main/docs/resource#service.

In the rare case where there's multiple tomodachi services started within the same Python process, it should be noted that OTEL traces, metrics and logging will primarily use the OTEL_SERVICE_NAME, and if it's missing then use the name from the first instrumented service class. The same goes for the service.instance.id resource attribute, which will be set to the first instrumented service class' uuid value (which in most cases is automatically assigned on service start). Multi-service execution won't accurately distinguish the service name of tracers, meters and loggers. The recommended solution if this is an issue, is to split the services into separate processes instead.

Exclude lists to exclude certain URLs from traces and metrics

To exclude certain URLs from traces and metrics, set the environment variable OTEL_PYTHON_TOMODACHI_EXCLUDED_URLS (or OTEL_PYTHON_EXCLUDED_URLS to cover all instrumentations) to a string of comma delimited regexes that match the URLs.

Regexes from the OTEL_PYTHON_AIOHTTP_EXCLUDED_URLS environment variable will also be excluded.

For example,

export OTEL_PYTHON_TOMODACHI_EXCLUDED_URLS="client/.*/info,healthcheck"

will exclude requests such as https://site/client/123/info and https://site/xyz/healthcheck.

You can also pass comma delimited regexes directly to the instrument method:

TomodachiInstrumentor().instrument(excluded_urls="client/.*/info,healthcheck")

Prometheus meter provider (experimental)

The tomodachi.opentelemetry module also provides a Prometheus meter provider that can be used to export metrics to Prometheus. Run opentelemetry-instrument with the --meter_provider tomodachi_prometheus argument (or set OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus environment value) to enable the Prometheus meter provider.

Environment variables to configure Prometheus meter provider

OTEL_PYTHON_TOMODACHI_PROMETHEUS_ADDRESS specifies the host address the Prometheus export server should listen on. (default: "localhost")
OTEL_PYTHON_TOMODACHI_PROMETHEUS_PORT specifies the port the Prometheus export server should listen on. (default: 9464)
OTEL_PYTHON_TOMODACHI_PROMETHEUS_INCLUDE_SCOPE_INFO specifies whether to include scope information as otel_scope_info value. (default: true)
OTEL_PYTHON_TOMODACHI_PROMETHEUS_INCLUDE_TARGET_INFO specifies whether to include resource attributes as target_info value. (default: true)
OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED specifies whether exemplars (experimental) should be collected and used in Prometheus export. (default: false)
OTEL_PYTHON_TOMODACHI_PROMETHEUS_NAMESPACE_PREFIX specifies the namespace prefix for Prometheus metrics. A final underscore is automatically added if prefix is used. (default: "")

Dependency requirement for Prometheus meter provider

The tomodachi_prometheus meter provider requires that the opentelemetry-exporter-prometheus and prometheus_client packages are installed.

Use tomodachi extras opentelemetry-exporter-prometheus to automatically include a compatible version of the exporter.

The tomodachi_prometheus meter provider requires that the opentelemetry-exporter-prometheus package is installed. Use tomodachi extras opentelemetry-exporter-prometheus to automatically include a compatible version of the exporter.

OpenMetrics output from Prometheus with exemplars enabled

With exemplars enabled (env variable: OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED=true), make sure to call the Prometheus client with the accept header application/openmetrics-text to ensure exemplars are included in the response.

curl http://localhost:9464/metrics -H "Accept: application/openmetrics-text"

💡 Note that if the accept header application/openmetrics-text is missing from the request, exemplars will be excluded from the response.

Example: starting a service with instrumentation

This example will start and instrument a service with OTLP exported traces sent to the endpoint otelcol:4317 and metrics that can be scraped by Prometheus from port 9464.

All metrics except for target_info and otel_scope_inf will be prefixed with "tomodachi_".

Additionally exemplars will be added to the Prometheus collected metrics that includes sample exemplars with trace_id and span_id labels.

TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=otelcol:4317 \
OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED=true \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_ADDRESS=0.0.0.0 \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_PORT=9464 \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_NAMESPACE_PREFIX=tomodachi \
	tomodachi run service/app.py

Auto instrumentation: tomodachi --opentelemetry-instrument

Auto instrumention by setting env variable value: TOMODACHI_OPENTELEMETRY_INSTRUMENT=1

Auto instrumentation using the opentelemetry-instrument CLI