Inside, you’ll find:

• A full, 100-plus-page guide to creating and implementing an effective on-call program.
• Analysis of commonly used on-call models, their effectiveness in the real world, and pitfalls to avoid when setting up your on-call teams.
• An exclusive foreword from Andrew Clay Shafer, Vice President of Transformation at Red Hat and former co-founder of Puppet

To celebrate the new project, we sat down with author and on-call expert Serhat Can for a quick question and answer session.

Q&A with author Serhat Can

Serhat Can is a versatile engineer who has built and operated products as part of Atlassian’s Opsgenie team since 2015. As an engineer and DevOps evangelist, his main interest is helping teams build better on-call and incident response practices.

Atlassian: Do you remember your first on-call shift?

Serhat: Fortunately, I was in good hands. I was about a month into working with my team and started participating in responding to customer issues alongside our customer success team.

I still think that’s a great way to get used to the idea of on-call, and you get the added benefit of learning more about your product. Then, after a few months, I was responding to prod alerts during business hours. It was scary, but I was lucky to have senior engineers right by my side who could help me if I got stuck.

A: Was there anything that surprised you as you dug deeper into this material?

S: The more I wrote, the more I became sure that on-call requires ongoing attention to detail and continuous improvement. There is no silver bullet. The book covers a lot of options and good practices on-call teams may employ while curating their on-call programs. And they are definitely helpful. But the broader picture is that on-call is influenced a lot by your company culture.

If your company cares about you as a human, they’re more likely to care about making your on-call experience better. If your company cares a lot about ownership and has practices in place to make you feel like you are part of something important, you’ll in turn care much more about doing a good job on-call. So, on-call is definitely influenced by these kinds of cultural aspects.

A: It feels like a big theme of your book is that these “technology systems” are only as resilient as the “people systems” that build and operate them. Do you think the rest of the world is realizing that?

S: I must say that most big tech companies realize that these technically skilled people are expensive to hire and keep. Investing in people isn’t just the right thing to do – it makes good business sense. Most smart companies realize that better tech is only possible with better people. Resilience is only possible with people architecting resilient systems and people taking care of these systems. On-call is a way we can clearly see the people affect and improve iteratively.

A: What do you think teams struggle with the most in adopting on-call practices?

Related Article

Flip your thinking to find the right incident management KPIs

By Serhat Can In Opsgenie

S: I spent the last few years speaking at many DevOps conferences and chatting with brilliant people. Almost always, their first question is: how do I convince my management to make alerts and on-call better?

This is partly because some companies are still figuring out how critical IT is to their existence. And part of it is because of the business pressure. I think we can make great improvements to the latter with this book. The former – leadership buy-in – I think is easier now with a lot of people talking about DevOps. But we still need to talk business language and bring some data in. The book also covers that, but I must remind readers that change is not easy. As engineers, we should also try our best to understand our business and offer practical solutions to these challenges.

A: What do you think the future holds for on-call? Will more teams be going on-call? Will we see different types of professions embracing on-call?

S: To be honest, no one likes to be on call. Many professions have been practising on-call since way before those of us in IT. But as our dependency on online services has increased, our need for on-call has risen drastically. If business requires you to be on call, one of the best things you can do as an organization is to get help from as many people as possible. If someone is on call every other week, that becomes life impacting, while being on call every six weeks is much more tolerable. On top of this, if developers are on call, we see them create more quality code and pay more attention to observability.

Another key aspect is increased ownership once developers feel the customer impact of their work. Given these and other benefits, I see many organizations adopting on-call throughout their tech organization. And beyond engineering, we already see many customer support, marketing, and sales organizations making use of ideas from on-call scheduling and escalations to better serve their customers. I definitely see this need becoming more and more clear for many organizations.

The post Download our new on-call book appeared first on Work Life by Atlassian.

Our mission at Atlassian is to unleash the potential of every team. One thing we know great teams have in common? They use playbooks to manage the many processes formulated to keep their organizations running smoothly.

This article covers 5 critical steps to creating an effective incident response playbook. We’ll be using our own Atlassian Incident Management Handbook as a template to develop an incident response plan.

Why agile teams need an incident playbook

An incident response playbook empowers teams with standard procedures and steps for responding and resolving incidents in real time. Playbooks can also include peacetime training and exercises, which will prepare the team for the next incident.

At Atlassian, our incident teams are constantly training, refining, testing, and improving our incident management process. We developed our incident response playbook to:

• Guide autonomous decision-making people and teams in incidents and postmortems.
• Build a consistent culture between teams of how we identify, manage, and learn from incidents.
• Align teams as to what attitude they should be bringing to each part of incident identification, resolution, and reflection.

What’s in an incident response playbook?

Playbooks are a key component of DevOps and IT Ops incident management, as well as cybersecurity. They set the organization’s policies and practices for responding to unplanned outages, help teams bring order to chaos and make sure everyone’s responding to incidents and security threats consistently. 

Building an Incident Response Playbook

In creating our own Atlassian Incident Management Handbook, we’ve identified 5 best practices when it comes to managing an incident. These steps can be translated to a variety of DevOps and IT Ops teams and help guide the process of building an effective incident response playbook.

1. Define incidents for your organization

What to include: A specific definition of what constitutes an incident

Why: You can’t effectively resolve an incident if you don’t know when it’s happening. Different teams define incidents in different ways. If something goes wrong, every second matters, and you don’t need colleagues fighting over semantics.

Example:

The definition of an incident as it appears in the Atlassian Incident Management Handbook:

This definition not only what constitutes an incident, but also defines when an incident is resolved and remediated.

2. Establish predesignated roles

What to include: Incident roles and responsibilities

Why: A proper incident response playbook designates clear roles and responsibilities. Individuals on the incident response team are familiar with each role and know what they’re responsible for during an incident.

Example:

The roles we use at Atlassian are in place to ensure all necessary steps are covered, no duplicate work occurs, and communication runs smoothly and effectively.

• Incident Manager, has overall responsibility and authority for the incident. Empowered to take any action necessary to resolve the incident, which includes paging additional responders in the organization and keeping those involved in an incident focused on restoring service as quickly as possible.
• Tech Lead, a senior technical responder. Responsible for developing theories about what’s broken and why, deciding on changes, and running the technical team. Works closely with the incident manager.
• Communications Manager, a person familiar with public communications, possibly from the customer support team or public relations. Responsible for writing and sending internal and external communications.

3. Enforce a consistent process

What to include: Process steps and workflows

Why: No two incidents are exactly alike. But that doesn’t mean your responders can’t introduce a consistent workflow for responding to incidents.

Outline key steps and phases and make sure team members are clear on what’s expected during each phase – and what comes next. For example, Atlassian outlines the incident response flow over seven steps through three phases in order to drive the incident from detection to resolution.

Example:

As a new incident is detected, the incident manager begins initiating internal communication and response organization. This the team can begin working on fixing the cause of the incident and reaching a resolution. Strong organization in this stage facilitates action, which is powered by frequent communication. Adhering to a consistent process leads to a faster resolution, including a postmortem exercise we will cover below.

4. Enable rapid response

What to include: Templates and checklists

Why: Incident playbooks need to be simple enough for teams to follow in times of stress. Our own process includes a major incident manager “cheat sheet,” which outlines key steps like assessment, escalation, and delegation in a one-page format.

Following a predetermined incident response process doesn’t mean there’s no room to improvise. You have to be flexible and know when to adapt to a changing situation. Incidents, by definition, are scenarios where things don’t go according to plan, but that doesn’t mean you can’t plan for them. The teams who train and practice a set of plays are typically the ones who succeed.

Use this:

Try running an Incident Response Values play to improve team cohesiveness and work out any potential misunderstandings prior to an incident. Use our resource, the Atlassian Team Playbook, to better understand your team’s process in order to build a dynamic playbook.

5. Facilitate comprehensive postmortems

What to include: Outline of the postmortem process and issue fields

Why: A postmortem seeks to maximize the value of an incident by understanding all contributing causes, documenting the incident for future reference and pattern discovery, and enacting effective preventative actions to reduce the likelihood or impact of recurrence.

If you think of an incident as an unscheduled investment in the reliability of your system, then the postmortem is how you maximize the return of that investment.

Try this:

For postmortems to be effective, the process has to make it easy for teams to identify causes and fix them. The exact methods you use depend on your team culture; at Atlassian, we’ve found a combination of methods that work for our postmortem teams:

• Face-to-face meetings help drive appropriate analysis and align the team on what needs fixing.
• Postmortem approvals by delivery and operations team managers incentivize teams to do them thoroughly.
• Designate priority actions with assigned Service Level Objectives (SLO) with reminders and reports to ensure they are completed.

A step-by-step outline of the Atlassian incident response postmortem can be found on page 46 of our Incident Management Handbook.

Ultimately, an incident response playbook should be used to drive teams to work together effectively to resolve incidents as fast as possible. When an incident occurs, no one has time to debate best practices and point fingers. Thorough, well designed playbooks empower teams to do their best work. At Atlassian, our guide to all of these plays is detailed in our Incident Management Handbook.

The post How to create an incident response playbook appeared first on Work Life by Atlassian.

Related Article

5 tips for ramping up on remote work in a hurry

By Sarah Goff-Dupont In Teamwork

At the same time, working remote can make your incident team stronger than ever. Working remote can be a forcing function to realize some best practices that all incident teams can benefit from, remote or not. At Atlassian we’ve been practicing remote-first incident management for years, as we have teams distributed around the globe. An alert about a power failure affecting a cloud server in Oregon might have us waking up engineers in California and New York while looping in support teams in Sydney and updating stakeholders in Texas. We’re able to spin up organized, multi-continent incident response teams in a matter of minutes.

We didn’t figure this out overnight. We got here through years of experience and practice, and aim to constantly improve. Here are some tips for remote incident management teams. Many of them are detailed in our own Incident Management Handbook, which we’ve made available to the public and free.

Over-communicate

Communication is more important than ever when incident response teams are remote. When an incident is underway at Atlassian, one of the incident manager’s first responsibilities is to establish communication channels: specifically, a dedicated Slack channel for that incident, a Zoom room for live discussion, and a Confluence page for capturing notes.

If these canonical channels aren’t established right away, things can go south quickly. Different Slack conversations happen in multiple places, and engineers are off having private one-on-one conversations that other team members don’t know about. Important information is buried, lost, or siloed. Duplicate work happens. It’s not good.

Related Article

9 immediate ways to improve communication in the workplace

By Jamey Austin In Teamwork

And good communication goes beyond the immediate response team. Stakeholder and end-user communication is especially important with distributed teams. When managers don’t see your team physically huddled in a war room and typing feverishly, they get nervous. If you don’t proactively send updates to the right people, you’ll be flooded with interruptions. By being transparent and sharing real-time updates with customers and stakeholders in the places they’re likely to find them (web pages, support portal, social media) you can deflect tickets and interruptions. This gives your incident team more space to work on the incident. We use Statuspage to communicate these updates with internal and external stakeholders during incidents.

Practice openness and transparency

During an incident, your most precious resource is time. You want to mitigate the incident by creating the most productivity in the smallest amount of time possible. Incident management is an exercise in efficiency.

Nothing grinds that efficiency to a halt like duplicate work and siloed communication. Two engineers repeating each other’s work, or multiple teams having different versions of the same conversation, is a dangerous waste of time. And it’s especially easy for this to happen when teams are remote.

That’s why openness and transparency among incident responders is so important. Yes, there might be sensitive information you can’t share with customers or the rest of your organization. But among the response team, aim to default to transparency.

Create a source of truth for incident information

Speaking of transparency, it’s good to create a source of truth for the incident, some record which captures what the incident is, what’s going on with it, and how to find more information. This gives the response team – no matter where they are – a place to get up to speed on the incident and see its status. Tools like Jira Service Desk and Opsgenie are among the most popular options for this.

For folks outside the immediate response team – stakeholders, customers, and colleagues – we use Statuspage as the source of truth for incidents and incident status.

Document, record, capture

It’s not unusual in an Atlassian office to see two engineers calmly working side-by-side on the same incident but in complete silence. You could walk right past them and never even know they’re working on something together.

This is on purpose. We know that capturing and documenting information about the incident during the response is key. Face-to-face conversations aren’t off-limits, but we know they have a way of generating ephemeral information. And ephemeral information isn’t great for incident response.

We want to be creating information other team members can see, learn from, and build on. We want information we can analyze and study after the fact. All that detail is data you can study in an incident postmortem. Postmortems help teams peer into the inner workings of their services and discover areas for improvement. They’re also an effective tool for building trust between leadership and the rest of your organization.

That’s why everything from team communication to notes, theories, and activities are documented and recorded. Seeing something funny in logs? Take a screenshot and put it in Slack. Discover a task you should tackle later? Drop it in Confluence or Jira. Capturing all this data creates a rich record of the incident that may be helpful in ways you don’t even realize yet.

Get your entire team operating from the same playbook

Like we say in our incident handbook, a good incident process should be simple enough for people to follow under stress, but broad enough to work for the variety of incident types you will encounter.

We think our handbook is broad enough to work for different kinds of incidents and different kinds of teams, and you’re welcome to use it yourself. Feel free to download it, copy it, adapt it, share it, and make these practices work for your teams too.

The post 5 tips for incident management when you’re suddenly remote appeared first on Work Life by Atlassian.

We’ve refreshed a lot of workplace norms in recent years. Somewhere along the way, the idea of bringing your “full self” to work appeared.

Advocates claim it’s elevating careers and building happier, more successful teams. But for others, the concept sounds a little out there. (There are some who even make distinctions between “full self” and “whole self”!)

In other words, updating norms is hard.

Steven Aldrich, chief product officer at GoDaddy, explains his understanding like this:

“For me, it’s being comfortable with what I’m good at and not good at, it’s talking about things that I’m passionate about, and it’s being clear that I’m going to make decisions at work and outside of work with the same set of values in a very consistent way.”

Personality at work – you don’t have a choice

The truth is, we all have unique personalities. We have different rhythms and proclivities. Some of us are morning people, some night owls. Some of us really need quiet and isolation to focus, others like the buzz of a workplace and thrive around chatter and interruptions. Some push buttons, some aim to please. But whatever personality we have, it’ll eventually show up at work.

From the Playbook

Help your teammates understand how best to work with you.

Run the Play

Sometimes, the complexity of individual personalities means misunderstandings occur. It’s natural, but often difficult to navigate at work. Atlassian’s unofficial sixth value is: “Seek first to understand.” It really helps when a teammate’s behavior is confusing or inconsistent. Seeking first to understand better before jumping to (often negative) conclusions is a powerful practice to establish. It’s also helpful to create opportunities for people to explain more about who they are, and how they like to work. We created a play in our Altassian Team Playbook called My User Manual to help teams work better together by essentially creating a “personal user manual” to understand how to work best with each teammate.

Because your teammate might be afraid of coming off the wrong way. Maybe you sometimes worry about how you might seem – moody or quiet – and others might think you’re a jerk. When in reality you’re not a jerk, you’re just focused on getting something done. Or maybe you had a tricky morning at home at it’s on your mind.

People are different, and work differently. Everyone brings a different personality to the table.

What Atlassians do: run a personality test

Another way to help personalities blend and not clash is through personality testing.

Walk by the desks of the Jira Service Desk marketing team and you’ll see something interesting hanging on the wall: a poster with the team’s first names on one column and their corresponding personality type in the other.

There’s the ENTP (the “debater”), INTJ (“the architect”), INFP (“the mediator”).

The team used the free personality assessment available at 16Personalities. The assessment takes just a few minutes to do online by asking how well you agree with statements like “people can rarely upset you.” (You can learn more about the methodology behind the assessment here.)

Each personality type includes a detailed explanation, including sections on how different types interact in the workplace. For example, “The running theme for logicians is their desire for solitude, need for intellectual stimulation, and the satisfaction of the final piece of a puzzle clicking into place.”

The assessments gave everyone on the Jira Service Desk team a better baseline for understanding their colleagues, even though many of them had worked together for years. Suddenly past behaviors and quirks all sort of clicked.

“Some of it was spot on,” says Molly Bronstein, a product marketing manager on the Jira Service Desk team. “You learn how to work with people, you learn what makes people tick.”

How to run your own personality test with your team

1. Start small

The Jira Service Desk team didn’t get dozens of people in a room and surprise them with a personality test. It started small, with four people at an offsite taking the test on a whim and sharing their results. Over time, more and more people on the team saw the results and wanted to join the fun.

2. Keep it optional

Not everyone is going to be comfortable taking a personality assessment and sharing their results. Make sure people know it’s not mandatory. On her team, Molly said setting these expectations made it more engaging for the people who did decide to take the test. They felt more committed to understanding their results and their team’s types knowing it was something they opted into, instead of getting pressured into.

3. Broadcast it

The problem with a lot of team building activities is the outputs often seem to vanish into thin air. For the Jira Service Desk team, the most important part (which is a lot easier and nicer if you remember to keep it optional) was showing off the personality types in a big, visible way. That’s why they wrote their types on a poster board and hung it by their desks. This was a good reminder for everyone of how different teammates may interact. It was also great marketing for the project, because other people on the team would see it and want to take the assessment so they could be on the board. Instead of being a one-time activity, it became an ongoing artifact for the team.

One final tip

Remember, these are just exercises. Even though a personality assessment like this might generate some “Oh that’s totally you!” moments, no single assessment is going to fully describe a person.

Related Article

Working with introverts (written by an actual introvert)

By Season Hughes In Teamwork

“This is not an ‘end all, be all,'” Molly says. “Humans are multidimensional. This doesn’t define you.”

That could be what’s missing in this “full self” conversation. How you’re defined, what your “full self” means to you, is ultimately your decision. Maybe it means sharing your personality type with the help of an assessment. Or more courage to wear your heart on your sleeve at work, because that’s who you really are. It’s up to you. But sharing more of your personality in the workplace – who you really are – inspires others to do the same. This creates workplace environments with more camaraderie, more trust and belonging. And guess what? Better outcomes across the board.

The post Personality in the workplace: are you bringing it? appeared first on Work Life by Atlassian.

Especially ones that show up on my calendar last minute. Ones with no agenda or structure. The meetings that start with 11 minutes of chatter and go well past their time slot. Conversations that dip and weave through different topics, finding their way home after looping together six previously-unrelated subjects. Words dancing around the table like Ali and Foreman, improvising into the ether like Coltrane looking for a 13th note.

Sounds fun at a party. But I’m probably a pain in the ass at work.

I’m an extrovert. The thought of an all-day workshop with my team fills me with energy, but staring down eight hours of uninterrupted quiet time? Cue up the anxiety. I’d rather spend a day at a conference talking to customers than quietly combing through NPS data by myself. I’ll take a pick-up basketball game over a solitary jog on the beach.

Extroverts are an easy punching bag these days. We’re the blabbermouths who keep the smart people from getting work done. Deep work takes deep concentration, and who needs Blake from marketing stopping by to chat?

Being an extrovert can be tough. It’s not as easy as it might look. We have insecurities too. A lot of times I feel like an outsider at work. But over the years I’ve come up with a few ways to take advantage of my work style.

My colleague, Season, wrote an excellent piece about working with introverts. Typical extrovert, I decided to butt in with my thoughts.

Tip 1: Schedule social moments, build habits around people

Giving up people time cold turkey isn’t an option. Rather than deprive myself of the energy burst I get from a few minutes of chatter, I find it helps to take ownership of it and schedule it into my day. I try to set up a block of unstructured social time further out into my day. The benefits are two-fold. I get the energy burst from the interaction, and I also get an extra rush in the hours leading up to it. On days where I play basketball at lunch, I find I’m more productive in the morning because I’m looking forward to the time with friends. After the game, I get a boost of energy from the socialization and exercise, and I find I have more focus the rest of the day. It’s a win-win.

This works with improvised one-on-one visits, too. Around 11 a.m. on some days I’ll find a friend on Stride to hit up: “Quick walk and grab coffee this afternoon?” As tempting as it is to ask someone to hang out every time I feel tired, pushing these moments out a few hours forces a little discipline and gives me the added benefit of looking forward to the trip.

For my introvert colleagues: Extroverts like me will have an easier time respecting your quiet time if you help us shake off our social restlessness. Say “yes” to quick coffee walks if you can. If you can’t make it, let us know from the start and suggest another time. It’s a drag when the break I’ve been looking forward to gets brushed aside with a last-minute “Hey, I’m pretty busy” message.

Tip 2: Control your environment, control your destiny

An extrovert working at an office with an open floor plan is like going to a doughnut factory on a diet. So many people to talk to! So many teams and activities around! Does that guy want to have coffee with me? She’s new, I wonder if she wants to play ping pong?

Can I sit in on that meeting?

Perhaps surprisingly, I also work well in focus mode. Noise-canceling headphones and distraction-free app settings help me from getting sucked into a conversation I might not be able to resist. For anyone, putting thought and effort into your physical environment at work is important. I think extroverts are extra sensitive to these distractions, and can easily get sucked into them all day if we don’t set up systems. Don’t just rely on discipline, develop effective systems.

For my introvert colleagues: Nothing complicated here. Respect the same signals and cues to keep distractions limited. Just because I like to talk doesn’t mean I can, or should, right now. But if you want to grab coffee later …

Tip 3: Seek sounding boards and sparring partners

A lot of extroverts use talking as a method of discovery and filtering. I can struggle with a problem or concept in my head for a while, then make it click by talking it through with someone. A silent partner as a sounding board does the trick, but an active sparring partner is even better.

For my introvert colleagues: Conversation is a sport for people like me. Try to keep that in mind if we accidentally say something pointless, off-topic, or possibly abrupt. That’s our way of saying “Here’s all the information on the table, let’s sort it, and rearrange it, and play with it to see where the good stuff is.”

Despite our differences, I agree with Season. Introverts and extroverts can work really well together. If we’re open about our differences and open about strategies that work, at work, we can do amazing things together. Things we can’t do alone.

An effective organization is stronger because of these difference, not in spite of them. The complete team is greater than the sum of its parts. At least I think so.

Stop by my desk, let’s chat about it some more.

For more ways to work better with introverts and extroverts, check out the Atlassian Team Playbook: our free, no-BS guide to unleashing more of your potential.

Browse the Team Playbook

The post Working with extroverts (written by an actual extrovert) appeared first on Work Life by Atlassian.