Incident Management

Introduction

Railway understands the importance of effective incident management procedures. Everything we do is to minimize downtime of your workloads and applications and maximize communication when any part of the Railway platform goes down.

In the interest of transparency, we publish as much of Railway's procedure to keep Railway's customers in the know on how we handle and learn from incidents.

Monitoring + reporting

Railway has a robust monitoring system in place to proactively detect and address any potential incidents. We continuously monitor Railway's infrastructure, including servers, networks, and applications, to ensure their smooth operation. By monitoring key metrics and performance indicators, we can identify any anomalies or potential issues before they escalate into full-blown incidents.

However, it's important to note that while we strive to stay ahead of incidents, there may be situations where unforeseen issues arise. In such cases, we rely on qualitative customer feedback to help us identify and address any issues promptly. We encourage Railway's customers to report any problems they encounter through the Railway Help Station or Slack.

Status page + uptime

Railway's uptime and incident retrospective can be accessed on the Railway status page at https://status.railway.com/. On this page, you can view the historical uptime of Railway's systems and services. Additionally, you can find detailed information about past incidents, including retrospectives that provide insights into how incidents were handled and what measures were taken to prevent similar issues in the future.

The status page is built and hosted in-house. If you'd like to consume its data programmatically, it's backed by two simple JSON endpoints:

• https://api.railwaystatus.com/status — current status of Railway's systems and services. Its response schema is published at https://api.railwaystatus.com/status/schema.json.
• https://railway.com/.railway/pops — Railway's CDN points of presence (POPs) and their status. These are edge POPs, not deployment regions.

These endpoints are provided as-is, without any support, and their response format may change without notice.

For Enterprise customers, we offer SLOs and guarantees of service that may not be represented on the uptime dashboard.

Uptime methodology

The uptime on the status page is the share of the last 90 days that a component was up. Major outages count as 100% downtime, while partial outages are counted at 30% downtime. This section explains how we calculate uptime.

Definitions

Uptime

The share of the last 90 days, plus today, that a component was up.

downtime = major outage hours + (0.3 × partial outage hours)
uptime   = 1 - downtime / total hours in the last 90 days, plus today

Downtime

The share of the last 90 days, plus today, that a component was down. An hour of major outage counts as an hour of downtime, and an hour of partial outage counts as 18 minutes (30% of 60 = 18).

Incident

An unplanned event that harms a customer's running workload or stops them from doing something on Railway. An incident affects one or more components.

Components

A distinct part of Railway that has its own uptime, such as Builds or Networking - Public.

Major outage

The impact level for a component that is unavailable. Every minute counts in full as downtime.

Partial outage

The impact level for a component that is unavailable for some customers, some regions, or some of its functions, while the rest still works. It shows on the day bar, and each minute counts as 30% of a minute of downtime.

Degraded performance

The impact level for a component that works but is slower, or returns more errors, than normal. It shows on the day bar but doesn't count as downtime.

Scheduled maintenance

Planned work we post in advance, with a start time and an estimated end time. It never counts as downtime, as long as we give advance notice.

Calculations

Component uptime

weighted downtime = major outage time + (0.3 × partial outage time)
component uptime  = 1 - weighted downtime / (91 × 24 hours)

The window is the last 90 days plus today, in UTC.

Incident duration

An incident counts from when we post it to when we resolve it. Time before we post it doesn't count. Time spent monitoring a fix does.

Impact weighting

We weight each impact level by how much of a component is down. A major outage counts 100%, a partial outage 30%, and degraded performance 0%. We call this 100/30/0.

A component that's fully down should cost the most. A component that's down for some customers should still cost something, or a partial outage that lasts for days would leave uptime at 100%. Degraded performance means the component still works, so it doesn't count.

The 30% is a convention adopted by, among others, GitHub and Atlassian. Although it is flawed (a partial outage that hits 5% of customers and one that hits 50% count the same), we use it because it's simple, it's applied the same way every time, and it's a common way to weight partial outages so that they actually count against uptime, which they should.

Two rules keep the weighting honest:

• Each period counts at the level in effect at the time. If an incident is a major outage for 2 hours and then a partial outage for 10 hours, it counts as 2 + (0.3 × 10) = 5 hours of downtime.
• When incidents overlap, the highest level wins. A major outage and a partial outage on the same component during the same hour count as 1 hour of downtime, not 1.3.

Group rollup

A group's uptime counts the time when any of its components was down. We combine the downtime of every component in the group (and every region of a regional component), so time when two components are down at once counts once, not twice. When components overlap at different impact levels, the highest level wins, using the weights above.

For example, if Builds is in a major outage for 10 hours in September and the Dashboard is in a major outage for 10 hours in October, the group counts 20 hours of downtime. If both outages happened during the same 10 hours, the group counts 10.

A group's uptime is never higher than its lowest component, and it never double-counts overlapping incidents.

Rounding

We round every figure to 2 decimal places. Anything at 99.99% or above shows as 100.00%. As illustrated below, it is worth noting that it's possible to "hide" up to 13 mins of downtime in a 91 day lookback period and have that round up from 99.99% to 100%.

Day bar color

Each day takes the color of the worst impact that touched it.

Uptime reference

Philosophy

1. Our first job is keeping your workloads up

Workload or deployment uptime is our number one priority and the number one metric we care about on the status page. As a cloud provider we take seriously the mandate to keep your apps running independent of the platform itself.

2. We call every incident, no matter how small

We post every incident on the status page, however small and it is our policy to write thorough incident reports for every major outage. For full transparency, those reports are listed below.

3. Upstream failures are our failures

We own everything up and down the stack, including the upstream providers we choose, such as cloud and transit providers. When they fail, it counts against our uptime. Third-party services you connect to Railway, such as GitHub, Docker Hub, and npm, are shown separately and don't count against it.

4. Degradation procedure and precedence

In the event that we determine it's necessary to restrict certain components to the Railway userbase, we do so in reverse order of plan service level:

• Trial users
• Hobby users
• Pro users
• Enterprise users