--- title: Tracing description: Trace requests from Railway's edge through your services with built-in OpenTelemetry tracing, and explore them on the Traces page. --- Tracing follows a single request from Railway's edge, through your service, and into the services it calls. Railway starts the trace at the edge, receives OpenTelemetry spans from your services through a built-in collector, and shows the result on the **Traces** page of your project. **Note:** This is a preview feature under active development. Railway records the edge's part of every request to a traced service without changes to your code. To see what happens inside your service, instrument it with an OpenTelemetry SDK or turn on [automatic instrumentation](/observability/tracing/automatic-instrumentation). ## How it works A trace is a tree of spans. Each span records one unit of work with a start time, a duration, a status, and attributes such as the HTTP route or a database statement. Every hop that takes part in a request adds spans to the same trace, identified by a shared trace ID. On Railway, a traced request passes through three components: 1. **Edge.** When a request for a traced service arrives at Railway's edge, the edge records a server span with the method, path, response status, cache result, and upstream, and forwards a W3C `traceparent` header to your service. 2. **Proxy.** The ingress proxy in your service's region routes the request to one of its replicas and adds a span for that hop. 3. **Service.** If your app runs an OpenTelemetry SDK, the SDK reads the `traceparent` header, continues the trace, and exports its spans to Railway's collector. Requests your service makes to other services carry the same header, so their spans join the trace as well. Railway runs an OTLP receiver on every host. Services reach it at the address in `OTEL_EXPORTER_OTLP_ENDPOINT`, and the receiver attributes each span to the service that sent it. Railway stamps the project, environment, service, deployment, and replica onto every span as resource attributes, so you can filter by them on the Traces page. The receiver accepts traces only, over OTLP/HTTP and OTLP/gRPC. It doesn't accept metrics or logs. Railway collects [metrics](/observability/metrics) and [logs](/observability/logs) for every service without instrumentation. ## Enable tracing Tracing is set per service and per environment. Each service has two switches in every environment: **Traced**, which makes the edge trace requests to the service and adds the OpenTelemetry [variables](#provided-variables) on its next deploy, and **Automatic instrumentation**, which instruments the service's processes without code changes and only takes effect while the service is traced. Every request to a traced service is traced. There is no project-wide setting and no sample rate. 1. Navigate to the **Traces** tab in your project's top navigation. The Traces page follows the environment selected in the dashboard. 2. Click **Tracing setup** to open the panel. It opens on its own while the environment has no traces yet, and its subtitle names the environment the settings apply to. 3. Turn on the **Traced** switch in the row of each service you want traced. 4. Pick **Automatic instrumentation** or **Manual instrumentation** in the same row to say how the service exports its own spans. See [Instrument your service](#instrument-your-service). The panel lists every service in the environment except databases, and shows for each when the edge and the app last exported a span, so you can watch the first spans arrive. A service without a public domain is marked as such: the edge never sees its requests, so only the spans it exports itself appear. The switches belong to the environment you are looking at. Tracing a service in `production` doesn't trace it in `staging`; switch environment and turn it on there as well. The service's **Settings → Tracing** section links to the same panel. ### What happens when you enable tracing - The edge starts tracing requests to the service's domains in that environment right away. - On the next deploy in that environment, Railway adds the OpenTelemetry [variables](#provided-variables) to the service. An app that runs an OpenTelemetry SDK exports spans from that deploy on. - A service without a public domain never receives requests from the edge, so it has no edge spans. Its own spans still appear in traces that other services propagate to it over the [private network](/networking/private-networking). ### Infrastructure as Code In [Infrastructure as Code](/infrastructure-as-code/reference#tracing) the same two switches are the service's `tracing` block, and `railway config pull` renders `tracing: { enabled: true, autoInstrumentation: true }` for a traced service. Setting it in `service()` requires CLI 5.63.0 or newer and an SDK that accepts the field; see the reference for the version floors. ## Force or suppress a trace The edge writes its decision into the `traceparent` header it forwards, and an SDK with the default parent-based sampler follows it. If a client sends its own `traceparent` header, the edge follows that header's sampled flag instead: a request whose flag is set is traced, and one whose flag is clear is not. This lets an instrumented client start a trace that continues into Railway, lets a caller keep a request out of tracing, and lets you check a single request while debugging: ```bash curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \ https://your-app.up.railway.app/ ``` Generate a new random trace ID (the second field) for each request you force. Reusing an ID merges the requests into one trace. ## Provided variables When tracing is enabled for a service in an environment, Railway adds these variables on its next deploy there. They appear in the service's **Variables** tab with the other variables Railway provides. | Variable | Value | |---|---| | `OTEL_EXPORTER_OTLP_ENDPOINT` | The address of Railway's OTLP receiver on the host running your service | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | | `OTEL_EXPORTER_OTLP_HEADERS` | A header the receiver requires on every export | | `OTEL_SERVICE_NAME` | The name of the service in Railway | | `OTEL_SERVICE_VERSION` | The commit SHA of the deployment, or the deployment ID for image and CLI deployments | Every OpenTelemetry SDK reads these variables, so an SDK configured without an explicit endpoint exports to Railway. Don't hardcode the endpoint or the header in your code, and don't set a different `OTEL_SERVICE_NAME` unless you want spans attributed under another name. A variable you set yourself takes precedence. If you set `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` on the service, for example to keep exporting to your own collector, Railway adds none of the tracing variables, and your app's spans don't reach the Traces page. The edge still traces requests to the service. Railway sets no sampler variables; the edge traces every request, and the SDK's default parent-based sampler follows it. Many SDKs export metrics and logs to the same endpoint by default. Railway's receiver doesn't accept them, so set these variables on the service to keep the SDK from trying: ```plaintext OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none ``` The per-language pages note which SDKs need this. ## View traces Navigate to the **Traces** tab in your project's top navigation. Like the log explorer, the Traces page is scoped to the selected environment. The page lists the environment's traces for the selected time range, with the newest at the bottom. Each row shows when the trace started, the root span's name, the service that handled the request, the duration, and the number of spans and errors. A histogram above the list shows how traces are distributed over the time range. Toggle live updates on to see new traces as they arrive. Click a trace to open it. The trace view shows every span in the trace as a waterfall, nested under its parent, with a timing bar and a color per service. Click a span to see its attributes, events, links, and status. The trace ID is in the panel header with a copy button, and the open trace is part of the page URL, so you can share a link to it. ### Open a trace by ID Enter a trace ID in the **Trace ID** field at the top of the Traces page and press Enter. The field accepts the 32-character hexadecimal ID or a whole `traceparent` header, so a value copied from a client's logs or a response header pastes as is. Traces belong to the environment their spans were exported from. Opening a trace ID from another environment shows no spans. ### Find the trace ID of a request Railway's edge adds an `x-railway-trace-id` response header to every response it traced. Read it from a client, from the browser's developer tools, or with curl: ```bash curl -sI https://your-app.up.railway.app/ | grep -i x-railway-trace-id ``` The header is absent when the request wasn't traced. Ask a user who reports a problem to include it, or return it from your frontend's error handling, and open the trace directly. Inside your service, the OpenTelemetry API exposes the current trace ID. Add it to your [structured logs](/observability/logs#structured-logs) as an attribute to jump from a log line to its trace. ## Search traces The filter on the Traces page uses the same syntax as [logs](/observability/logs#filter-syntax): keywords, `@field:value` pairs, the operators `AND`, `OR`, and `-` (negation), and parentheses for grouping. A filter matches spans, and the result lists every trace that contains a matching span. Free text matches span names. These fields are built in: | Field | Matches | |---|---| | `@trace:` | Trace ID, 32 hexadecimal characters | | `@span:` | Span ID, 16 hexadecimal characters | | `@name:` | Span name. End the value with `*` for a prefix match | | `@serviceName:` | The `service.name` the exporter sent | | `@service:` | Railway service, by name or ID | | `@deployment:` | Deployment ID | | `@replica:` | Replica ID | | `@component:` | Which hop exported the span: `edge`, `proxy`, or `service` | | `@kind:` | Span kind: `server`, `client`, `internal`, `producer`, or `consumer` | | `@status:` | Span status: `ok`, `error`, or `unset` | | `@duration:` | Span duration in milliseconds, with `>`, `<`, or a range such as `100-250` | Any other `@key` looks the key up in the span's attributes and resource attributes. The filter input suggests the attribute keys present on the spans in the selected time range, so you can filter on what your instrumentation emits without knowing the names by heart. ### Examples Find traces that contain an error. ```text @status:error ``` Find requests the edge served in more than a second. ```text @component:edge AND @duration:>1000 ``` Find traces in which the `api` service made an outgoing call. ```text @service:api AND @kind:client ``` Find server errors on one route. ```text @http.route:/checkout AND @http.response.status_code:500 ``` Find traces that include a database query starting with `SELECT`. ```text @name:SELECT* ``` Find traces from one deployment that took longer than half a second. ```text @deployment: AND @duration:>500 ``` ## Command line The [`railway trace`](/cli/trace) command changes the same settings as the Tracing setup panel and lists the same traces. Link a project first, or pass `--project` and `--environment`. ```bash railway trace status --all # Every service in the environment, last spans railway trace enable --service api # Trace a service in the linked environment railway trace enable --auto-instrument # Tracing plus automatic instrumentation railway trace enable --all --environment staging # Every service in staging railway trace disable # Stop tracing the linked service ``` Tracing is set per service and environment, so every subcommand works in the linked environment or the one you pass with `--environment`. ```bash railway trace list --since 30m --errors # Recent traces with an error span railway trace list --all --filter '@http.route:/checkout @duration:>500' railway trace get 4bf92f3577b34da6a3ce929d0e0e4736 # One trace as a span tree ``` `list` takes the same [filter syntax](#search-traces) as the Traces page. Pass `--json` to get one trace or span per line for scripts and agents. See the [command reference](/cli/trace) for every option. ## Retention and limits Traces are retained for the same period as logs on your plan. See [log retention](/observability/logs#log-retention). Railway applies these limits to incoming spans: - Each replica can export 1,000 spans per 10 seconds. Spans over the limit are rejected, and the SDK sees the rejection as an OTLP partial success. - A span can carry 128 attributes, 128 events, and 128 links, each with 128 attributes. Attribute values are cut at 16 KiB. Excess is dropped and counted on the span rather than rejected. - A search returns up to 500 traces. Narrow the filter or the time range to see the rest. ## Instrument your service The edge traces a request on its own, but a trace only shows what happens inside your service once the service exports spans. Two options are available: - [Automatic instrumentation](/observability/tracing/automatic-instrumentation) traces Node.js, Go, Python, Ruby, and Java processes without code changes, on a best-effort basis. - An OpenTelemetry SDK gives complete traces with custom spans. See the guide for your language: - [Node.js](/observability/tracing/nodejs) - [Deno](/observability/tracing/deno) - [Functions](/observability/tracing/functions) - [Python](/observability/tracing/python) - [Go](/observability/tracing/go) - [Java](/observability/tracing/java) - [Ruby](/observability/tracing/ruby) - [.NET](/observability/tracing/dotnet) - [Rust](/observability/tracing/rust) - [PHP](/observability/tracing/php) ## Troubleshooting **No traces appear.** Check that tracing is on for the service in the environment you're looking at, in the **Tracing setup** panel or with `railway trace status`, and that the service has a public domain. Tracing is set per environment, so a service traced in `production` shows no traces in `staging` until you turn it on there too. Send a request and look for the `x-railway-trace-id` response header, as shown in [Find the trace ID of a request](#find-the-trace-id-of-a-request). **Edge spans appear, but the app's spans don't.** Railway adds the `OTEL_*` variables on the first deploy after you enable tracing, so redeploy the service. Check that the app loads the SDK before it starts serving requests, and that the service doesn't set its own `OTEL_EXPORTER_OTLP_ENDPOINT`. The **App** indicator in the Tracing setup panel turns green as soon as the first span from the app arrives. **The app's spans show up as separate traces.** The SDK isn't reading the `traceparent` header. Configure the W3C Trace Context propagator, which most SDKs enable by default but the Go SDK requires you to set explicitly. If a proxy or framework in front of your handlers strips incoming headers, allow `traceparent` through. **The SDK logs export errors for metrics or logs.** Set `OTEL_METRICS_EXPORTER=none` and `OTEL_LOGS_EXPORTER=none`. The receiver accepts traces only. **Spans are missing from a busy service.** The service is exporting more than 1,000 spans per 10 seconds per replica. Reduce the span volume by disabling noisy instrumentations, or configure a sampler in the SDK: `OTEL_TRACES_SAMPLER=traceidratio` with `OTEL_TRACES_SAMPLER_ARG` set to a fraction keeps that share of requests. A parent-based sampler follows the edge, which traces every request, so it wouldn't reduce anything. ## See also - [railway trace](/cli/trace) - enable tracing and read traces from the CLI - [Logs](/observability/logs) - HTTP logs record every request, traced or not - [Instrument an App with OpenTelemetry](/guides/instrument-app-opentelemetry) - a step-by-step walkthrough from enabling tracing to custom spans - [Connect a Third-Party Observability Tool](/guides/third-party-observability) - ship traces to hosted backends for longer retention