A Production Incident is an Opportunity for You

Act fast and communicate broadly

Mar 19, 2025

A Production incident is always something painful. All of us pass through that know it.

Also, we know that our systems benefit from those incidents because an incident in Production is one of the best ways to improve our systems. After a proper postmortem, interesting action items came along ensuring that, that issue, does not happen again.

That’s great, but:

☝🏼 Is there a way you can get some benefit as well?

In today’s issue, I’ll explain how you can turn a Production incident into an opportunity to boost your career.

DISCLAIMER 1: With this, I’m not encouraging you to make Production incidents for thriving. Please, don’t 😊.

DISCLAIMER 2: I’m a true believer in teamwork. In this newsletter, I give you real-life experiences, not nice sentences to cheer you up. In the real life of companies that build software, your ability to lead org-wide initiatives will help you to thrive in your career.

Now we are ready, let’s start with an example use case and the opportunity that can give you, so you can understand better the benefits that the proactivity to solve problems will give you.

💥 The Production Incident

I will make up a Production incident here. If you read me for a while, you know that I’m into Kubernetes technology, so it will be about applications deployed in Kubernetes.

Let’s imagine:

There was a bug, global in your platform, in the way the applications are deployed in Kubernetes.
Such a bug made your Kubernetes deployments have 0 Pods running.
And because of that, all the users of your platform are not able to use your product.
But, your observability dashboard did not raise any alerts, and there are no red lights, all green.

If you see your monitors you might think that your applications are having no issues because the monitors are green… but your applications are not there!

👉🏼 Clearly, there is a gap in our observability; We have an opportunity to improve.

🎯 The Opportunity

☝🏼 It’s all about solving problems.

From the example above, the opportunity is to create a monitor that is capable of detecting that an application does not have Pods running.

Improving monitoring across the entire engineering organization is a great way to make an impact, as all teams will benefit from it.

👉🏼 If you find a gap or something to improve, jump into it right away, don’t wait until the situation gets cold.

Do not wait too much time to make the improvement. If the gap was detected today, better you have the solution tomorrow than the day after tomorrow. Don’t get busted by the analysis paralysis.

Create a Pull Request with your brand-new monitor that fixes the gap.
Share it with that senior engineer you team up with from time to time and get feedback soon.
Test it well, and ship it.

👉🏼 Find early adopters for your improvement, so it will be seen as something ready to use.

You know the expression “A picture is worth a thousand words”? It applies here as well. Nobody will use your improvement, in this example a new monitor, unless they see it working and shiny.

👉🏼 Do not keep it for yourself, announce it loud!

I made the mistake of not announcing my work a couple of times. I can tell you, it’s an unpleasant mistake. Once you have your early adopters happy, ensure you communicate broadly, to tech leads, engineering managers, etc, all over your engineering organization.

Prepare a well-structured email to ensure you communicate properly the value you brought with the improvement you made. ChatGPT could help you with that. In case you have doubts, ask your engineering manager for feedback before sending it. One of the follow-up readings, that I give you at the end of this email, is about frameworks to use for communication.

👉🏼 Keep this in mind: You solve problems for yourself, your peers, and your company.

But, “what are the real benefits I will get by doing this?” you may ask. Let’s see them.

🌸 The Benefits

The benefit for the company is clear: The system and product will become better.

In the example we are following today, observability is improved, allowing issues to be detected before they affect users.

☝🏼 What about the benefits for you to thrive as a Software Engineer or Tech Lead? It's not a huge list, but the important ones.

Leadership and ownership. Taking the initiative to solve a problem that 99% of the engineering teams have, without nobody telling you to do so, shows that you really care and own the system and the product.
A trustworthy person. The fact that you deliver something that brings value and solves a problem, and you share it with everyone, will begin your recognition of reliability; You will be seen as someone who can be trusted to solve problems.
Visibility on management. Solving problems that affect broader boundaries brings more light to you. Management will see you as a problem solver and someone committed to the company’s goals and that’s important when the time of promotions comes.

Only because those benefits, make it worth to work as a team player and focus on solving problems that affect your peers.

👉🏼 These side effects will help you thrive in your career, from Software Engineer to Tech Lead and beyond.

✨ Takeaways

Let’s wrap up for today with some tips you can put in your bag to thrive in your career as a Software Engineer or Tech Lead.

Find something that is missing, and make the improvement. They are everywhere, but you have to pay attention. Some meetings that involve multiple roles could help you to fish something.
Act fast, implement the improvement, and ship it. Do not wait until the incident is forgotten.
Get feedback and support from senior peers in your team and surrounding teams to ensure you deliver something that provides real value.
If you solve a problem for your company, it will resonate beneficially for you.

Hope you enjoyed this issue as much as I did writing it.

Is there anything of interest you would like me to dig out more? Reply to this email or drop me a new message.

We are ✨1105 Optimist Engineers✨!! 🚀

Thanks for your support of my work, really appreciate it!

You rock folks! 🖖🏼

If you enjoyed this article, then click the 💜. It helps!
If you know someone else will benefit from this, ♻️ share this post.