---
title: "Reliability"
description: "Resources for building reliable systems including on-call best practices, incident management, and customer communication strategies."
url: "https://www.rubick.com/wiki/process/reliability/"
---

[Edit this page on github](https://github.com/jadeforrest/blog/edit/master/src/content/wiki/process/reliability.md)

[Wiki](https://www.rubick.com/wiki/) / [Processes & Methodology](https://www.rubick.com/wiki/process/) / Reliability

# 🔧Reliability

Resources for building reliable systems including on-call best practices, incident management, and customer communication strategies.

## Retrospective Prime Directive

> **“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”**

[https://retrospectivewiki.org/index.php?title=The\\\_Prime\\\_Directive](https://retrospectivewiki.org/index.php?title=The%5C_Prime%5C_Directive)

## Rolling out on call

Some great resources:

-   [https://wiki.dbbs.co/view/suggested-reading-for-on-call](https://wiki.dbbs.co/view/suggested-reading-for-on-call)
    -   [https://thenewstack.io/advice-management-teams-enrolling-changes-on-call-systems/](https://thenewstack.io/advice-management-teams-enrolling-changes-on-call-systems/)
    -   [https://codywilbourn.com/2018/03/22/sustainable-on-call/](https://codywilbourn.com/2018/03/22/sustainable-on-call/)
    -   [https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-incident-management-program/](https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-incident-management-program/)
-   [https://increment.com/on-call/](https://increment.com/on-call/)
-   [https://www.honeycomb.io/blog/oncall-and-sustainable-software-development](https://www.honeycomb.io/blog/oncall-and-sustainable-software-development)
-   [https://response.pagerduty.com](https://response.pagerduty.com)

## Gamedays

## Risk matrices

## Examples of good incident communication to customers

-   [https://github.com/danluu/post-mortems](https://github.com/danluu/post-mortems) (a collection of post-mortems)
-   [https://blog.cloudflare.com/cloudflare-incident-on-october-30-2023](https://blog.cloudflare.com/cloudflare-incident-on-october-30-2023)
-   [https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage](https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage)

## Setting up rotations on Slack

-   [Using Slack Workflows and Google Sheets for free rotations](https://orlevy.medium.com/how-to-set-up-a-round-robin-workflow-in-slack-using-google-sheets-508dbcb3bc1e)