Introduction

Every team wants to get a product out there, to start getting customers. During the beginning of a business’s lifecycle, it is critical to mature the MVP into a product. Being focused on the target may get you to the finish line, but your organization maybe in for a big wake up call.

You’ve developed how your software works? Great. Here’s whats more important: How Does Your Software Fail? In this article, we will work through an example of what I call identifying Failure Containers in a system. Failure Containers allow us to understand the “blast radius” of a construct when (not if) things go wrong.

Glossary

Observability - The ability to understand the Metrics of a System. This directly correlates to detection and remediation time during incidents. Good observability also informs proper scaling.

Scalability - The concept of adding capacity to the System. Scalability correlates to lower barrier to add capacity, and lowered costs for infrastructure.

Stability - The system being resistant to failure, and the construct does not affect the stability of other constructs. This correlates to customer experience.

Efficiency - The value of compute/storage per resource’s cost (temporal or monetary). This correlates to customer experience, and infrastructure costs.

Security - Resistance to misuse from actors inside or outside a sphere of confidence. Poor security can lead to incidents, and lawsuits (yikes!).

Exponential Backoff Retry - Retries wait for a time interval that scales exponentially with the number of retries.

Horizontal Scalability - The ability to add capacity to a system by adding more instances, instead of increasing the hardware. This is considered the most cost effective way to scale a system.

Establishing Good Failure Containers

Let’s take the example of a very simple chat messaging system, that serves multiple countries.

Here an App will send a message to a backend server, which save that message to a DB and will be send to an API. That API will send the message to another customer’s phone.

To identify and tighten our Failure Containers, we will move through the levels of the application one by one where we inspect the observability, scalability stability, efficiency and security of:

  1. Code

  2. Component

  3. Deployment

Code

Let’s take the following code example that processes the message, saves the data, and sends the message to an API.

const database = new DB();
const sendApi = new API();

const processMessage(message: Message) => {
    database.save(message);
    sendApi.send(message);
    database.update(message, 'SENT')
}

If we take a moment to enumerate out the failures we can gain a clearer view of how to strengthen this code:

  1. Input Message could be malformed,.

  2. Dependencies could be down.

  3. We can contribute an outage to the dependencies.

Please note that one failure here causes the rest of the process to fail with ambiguity — the Failure Container here is poor. From focusing on our criteria of what is a good Failure Container, we can increase the quality of this code:

const exponentialBackoffRetry = new ExponentialBackoffRetry()
const database = new DB(exponentialBackoffRetry);
const sendApi = new API(exponentialBackoffRetry);
const messageValidator = new Validator();

const processMessage(message: Message) => {
    // case: message could be malformed, or null
    if (!message || !messageValidator.isValid(message) ) {
        return { status: 400 }
    }

    try {
    database.save(message);
    sendApi.send(message);
    } catch (e) {
        if (e instanceof DBError) {
            // Our DB is down, lets notify the app to retry.
            logger.error(`An Error ocurred when we attempted to write message ${message.id} to Database ${database.table}`)
            return { status: 500 }
        }

        if (e instanceof ApiError) {
            // Send Api failed, we must clear the DB entry as well
            logger.error(`An Error ocurred when we attempted to send message ${message.id} to the Send Api`)
            database.remove(message);
            return { status: 500 }
        }
    }

    try {
        database.update(message, 'SENT')
    } catch (e) {
        // DB is down, we should try again in a few seconds
        if ( e instanceof DBError) {
            logger.error(`An Error ocurred when we attempted to Update Message ${message.id} to be status 'SENT' in ${database.table}`)
            retry( () => database.update(message, 'SENT'), EXPONENTIAL_BACKOFF);
        }
    }
}

We now have a good Failure Container of this Code:

Observability - We are explicit about our responses, and we log what message failed, and a good failure message.

Stability - The enqueuement logic does not affect the processing logic. One request is never affected by another. We also use exponential retries to ensure that a misbehaving component does not overwhelm another.

Scalability - We can add more boxes to handle the incoming load.

Efficiency - We fail as fast as possible if there is something wrong.

Security - We don’t expose any sensitive information to the customer.

Component

We have a great start to hardening our system - code that is well behaved. However please consider the component that is running this code. Our example product is so successful that the world is using it, and now we are getting thousands of messages per second. Our component will fail from the amount of traffic.

As of right now, the response is given after the message is sent. That could be a long wait time for the customer — a poor user experience. Our customer won’t want to wait so long, and our customer doesn’t need any result besides that our system ingested our message. We can change our interaction with the customer’s phone to be asynchronous. This pattern is especially useful if you are worried that latency during the processing of a message will negatively affect a customer’s experience.

I have had great luck with the following pattern:

We can allow the customer to enqueue messages through a lightweight backend server (Enquement Server). That server only validates that the message is properly formed, and then sends the message to a queue that is consumed by the Processing Server. The Enqueument Server then responds to the phone.

Note: We will assume that we can retry infinitely with no problems. In reality you may need to have a maximum retry for your message.

Take a look at our enqueuement example below:


const messageValidator = new Validator();
const exponentialBackoffRetry = new ExponentialBackoffRetry()
const queueService = new QueueService(exponentialBackoffRetry);

const enqueueMessage(message: Message) => {
    if (!message || !messageValidator.isValid(message) ) {
        return { status: 400 }
    }
    try {
    queueService.add(message);
    return {status: 200};
    } catch (e) {
        // The queue failed to be added to, return a code that prompts a retry
        logger.error(`An Error ocurred when we attempted Enqueue ${message.id} to Queue ${queueService.name}.`)
        return {
            status: 5000
        }
    }
}

We’ve thinned out our server here and applied our Failure Container concepts:

Observability - We can observe how things failed.

Stability - The enqueue logic does not affect the processing logic. One request is never affected by another. Again we interact with the dependency with an exponential back off retry,

Scalability - We can add more boxes to handle the incoming load.

Efficiency - We Fail fast. Every piece of code written is critical to delivering value to the customer.

Security - We don’t expose any sensitive information to the customer.

Now lets take a look at how we can change the processing:

const exponentialBackoffRetry = new ExponentialBackoffRetry()
...

const processBatch = () => {
    // Get a batch of messages
    const messages = queueService.getMessages(BATCH_SIZE);

    // The Database saveBatch will return failures and successes
    const { dbFailedMessages, dbSuccessMessages} = database.saveBatch(messages);
    if (dbFailedMessages.length > 0) {
        logger.error(`We were unable to save the following messages ${dbFailedMessages.map( m => m.id)} to Database ${database.table}`)
    }

    // The sendApi sendBatch will return failures and successes
    const {sendFailMessages, sendSuccessMessage} = sendApi.sendBatch(dbSuccessMessages);
    if (sendFailMessages.length > 0) {
        logger.error(`We were unable to send the following messages ${sendFailMessages.map( m => m.id)}`)
        logger.error(`An Error ocurred when we attempted to send message ${message.id} to the Send Api`)
    };

    // Mark which messages were successful.
    queueService.markAsProcessed(sendSuccessMessage);
}

Observability - We can see which operations had failures.

Stability - One record in the batch wont affect others. We only send successfully saved messages. We also are using exponential retry.

Scalability - We can add more boxes to the queue service, database, or this compute handle the incoming load.

Efficiency - We batch all of our messages. We mark successes in the queue, so that failures can be retried.

Security - We don’t expose any sensitive information to the customer.

Deployment

This level takes your application to a whole level, and can seem overwhelming, but lets break it down together (this is also my favorite part of the job).

Remember how our application is serving the world? That means that messages from Europe and United States are all being processed in the same hardware.

There are a couple problems with this:

  1. Contacting the network across the world could be slow.

  2. Our hardware WILL fail to serve all of these customers.

  3. Isolating issues will be difficult.

  4. One size fits all may not be applicable for the geographies we are serving.

Lets create deployments per geography:

Here you see now depending on the traffic routing, we will hit different geographies. These geographies may have different scalability, and security requirements.

Observability - You can filter logs and metrics per region to proactively see scaling issues, and react to bugs/security issues.

Stability - One geography Cannot affect another.

Scalability/Efficiency - We can scale each geography independently, tailoring hardware to the needs of the geography.

Security - Different Geographies may require different levels of security, which we can deploy easily. Additionally, the hardware is completely separate, so a malicious actor who may gain access to one geography cannot affect another.

Conclusion

As you gain customers, don’t forget to consider how things can go wrong. We cannot anticipate when failures can happen, only that they can happen. Failure Containers allow us to be ready to quickly detect, remediate, and event prevent cascading failures in a system. Even if you can’t establish robust Failure Containers because you're focused on growth, understanding them is prepares your organization identify scaling cliffs and mitigate risks as you move forward.

Keep reading

© 2026 Bytebelt Newsletter.
beehiivPowered by beehiiv