|Docs

Instrument an App with OpenTelemetry

opentelemetryobservabilitytracingmonitoring

A trace follows one request from Railway's edge, through your service, and into the services it calls. Railway's built-in tracing starts the trace at the edge and receives OpenTelemetry spans from your services, so there is nothing to deploy and no backend to run. This guide enables tracing for a service in an environment, adds spans from inside it with automatic instrumentation or an OpenTelemetry SDK, forces a trace for one request, and finds the trace behind a request a user reports.

Note: Tracing is a preview feature under active development.

Prerequisites

  • A Railway project with a service you want to trace.
  • A public domain on that service. Railway's edge starts traces for requests it routes, so a service without a public domain only contributes the spans it exports itself.

1. Enable tracing

Tracing is configured per service and per environment from the Tracing setup panel on the Traces page.

  1. Navigate to the Traces tab in your project's top navigation. The page follows the environment selected in the dashboard, so pick the one you're instrumenting first.
  2. Click Tracing setup. The panel opens on its own while the environment has no traces yet; its subtitle names the environment.
  3. Turn on the Traced switch in the row of the service you're instrumenting. Leave Manual instrumentation selected, since the SDK you add below exports the spans.

Every request to the service in that environment is now traced. The switch is per environment: when the app later goes to production, turn it on there too. See Enable tracing for the details.

2. Look at the first traces

The edge starts tracing requests to the service's domains right away, before any change to your code.

  1. Send a few requests to the service's public domain.
  2. Back on the Traces page, each request appears as a row with the time it started, the root span's name, the service, the duration, and the span count. Toggle live updates on to watch new traces arrive.
  3. Click a trace. The waterfall shows an edge span with the method, path, status, and cache result, and a proxy span for the hop into your service's region.

Each traced response also carries an x-railway-trace-id header:

curl -sI https://your-app.up.railway.app/ | grep -i x-railway-trace-id

Paste that ID into the Trace ID field at the top of the Traces page to open the trace directly.

At this point a trace ends at the proxy. The next step adds what happens inside your service.

3. Add spans from inside your service

Two options: automatic instrumentation traces the process from the outside with no code changes, and an OpenTelemetry SDK gives complete traces with your own spans. Pick one per service; running both produces duplicate spans for every request.

Option A: Automatic instrumentation

Automatic instrumentation attaches eBPF probes to your service's processes on the host and exports a server span for each incoming HTTP or gRPC request, a client span for each outgoing plaintext call, and client spans for database and cache calls on protocols it decodes. It supports Node.js, Go, Python, Ruby, and Java processes on a best-effort basis.

  1. In the Tracing setup panel, find the service's row and pick Automatic instrumentation, then confirm.
  2. Send a few more requests. Railway instruments the running processes within a minute, with no redeploy.
  3. Open a new trace. The service's spans appear under the edge and proxy spans.

Automatic instrumentation can't see custom work inside a function, attach business attributes, or follow a request into an outbound TLS call. When you want those, move to an SDK and switch the service back to Manual instrumentation once the SDK's spans arrive.

Option B: An OpenTelemetry SDK

When tracing is on for a service, Railway adds the standard OTEL_* variables on the next deploy: the exporter endpoint, protocol, and headers for Railway's receiver, OTEL_SERVICE_NAME set to the service's name, and OTEL_SERVICE_VERSION set to the commit SHA. They show up in the service's Variables tab with the other variables Railway provides, and every OpenTelemetry SDK reads them. Leave the exporter endpoint alone: an OTEL_EXPORTER_OTLP_ENDPOINT you set yourself takes precedence, and the spans then go wherever it points instead of to the Traces page. See Provided variables for the full list.

Railway's receiver accepts traces only, so also set these two variables on the service to keep the SDK from exporting metrics and logs to it:

OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none

For a Node.js app, install the auto-instrumentation bundle and load it before your code with the start command:

npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
node --require @opentelemetry/auto-instrumentations-node/register server.js

The bundle instruments the http module, Express, Fastify, Koa, NestJS, pg, ioredis, mongodb, and more, and the register module configures the SDK from the variables alone.

For a Python app, install the distribution and prefix the start command with opentelemetry-instrument:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument gunicorn app:app --bind 0.0.0.0:$PORT

Redeploy the service and send a few requests. In the Tracing setup panel, the App indicator for the service turns green when its first span arrives, and new traces show the request's server span and the library calls under it.

The per-language pages cover ESM and Next.js on Node.js, framework notes for Python, and the setup for Deno, Go, Java, Ruby, .NET, Rust, and PHP.

Add a custom span

Auto-instrumentation gives you one span per request and per library call. Wrap the work you care about in your own span with the OpenTelemetry API. In Node.js:

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("checkout");

async function calculateTotal(cart) {
  return tracer.startActiveSpan("calculate-total", async (span) => {
    try {
      span.setAttribute("cart.items", cart.items.length);
      return await priceItems(cart);
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

The span nests under the request span that's active when the function runs. Attributes you set are searchable on the Traces page, for example with @cart.items:>10. The per-language pages show the same span in each language.

4. Follow a request across services

A request that your service makes to another service over the private network carries the trace along. An SDK adds the W3C traceparent header to outgoing requests, and automatic instrumentation injects it into plaintext HTTP and gRPC calls. When the callee is traced as well, its spans join the same trace, and the waterfall shows the caller's client span with the callee's server span nested under it.

Enable tracing on every service that takes part in the request. A service without a public domain has no edge spans, but its own spans still appear in traces that other services propagate to it.

To find traces in which one service called another, filter on the caller's client spans:

@service:api AND @kind:client

5. Trace a specific request

The edge traces every request to a traced service and passes its decision along in the traceparent header, so an SDK with its default parent-based sampler records exactly what the edge did. There is no sample rate to tune.

To follow one request end to end, or to start a trace from an instrumented client outside Railway, send a traceparent header of your own. The edge follows its sampled flag: set means traced, clear means not.

curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  https://your-app.up.railway.app/

Generate a new random trace ID (the second field) for each request you force. If a busy service exports more than the span limits allow, disable noisy instrumentations first, or configure a traceidratio sampler in the SDK; a parent-based one follows the edge and wouldn't reduce anything.

6. Find the trace behind a problem

When a user reports a slow or failed request, ask for the x-railway-trace-id header from the response, or return it from your frontend's error handling, and paste it into the Trace ID field. Without an ID, search. The filter on the Traces page uses the same syntax as logs and matches spans anywhere in the trace:

@status:error
@component:edge AND @duration:>1000
@http.route:/checkout AND @http.response.status_code:500

Any attribute your instrumentation emits is a filter key, and the input suggests the keys present in the selected time range. See Search traces for the built-in fields.

If spans are missing

  • No traces at all. Check that tracing is on for the service in Tracing setup for the environment you're looking at, and that the service has a public domain. Send a request and look for the x-railway-trace-id response header.
  • Edge spans only. Railway adds the OTEL_* variables on the first deploy after you enable tracing, so redeploy. Check that the SDK loads before the app starts serving and that the service doesn't set its own exporter endpoint.
  • The app's spans form their own traces. The SDK isn't reading traceparent. Most SDKs enable the W3C Trace Context propagator by default; the Go SDK needs it set explicitly. Allow the header through any proxy or framework in front of your handlers.

More cases are in Troubleshooting.

Next steps