Hacking Behavior

Read this post if you want to learn how to become a force multiplier through federating and radiating knowledge in a structured way without forcing yourself or others to become experts in the opposite side of the DevOps equation, respectively. That’s really the key — it is not practical to require every software developer to become an infrastructure expert, and vice versa. Fortunately, a little knowledge can go a long way! This talk ties together some basic organization change management techniques with DevOps, specifically around making operations and infrastructure topics more digestible for developers.

What even is Organizational Change Management?!

Tl;dr there are some quick tricks you can use to help changes be more digestible for people. Big changes — say, across organizations that are 10K+ in force — are the result of a lot of little changes (e.g. in individual behavior). A lot of these tricks have to do with making information accessible and less scary to people so that the path to a change is not a big deal. Change isn’t scary so much as loss is, so you need to connect with people in a way that helps them realize they have things to gain from participating in change, and that participating in this change is easy.

I think we as a species don’t give ourselves enough credit for our capacity to change. How many times have you heard the old saying, “you can teach an old dog new tricks”? Sure, everyone has their habits that are tough to bust. But what about all that massive change we volunteer for? Parenthood (99% chance of sleep deprivation, 100% chance of diapers), buying a house, buying a car, going to college…these are all big changes that happen all the time. Why?

Change Management Equation! And some other tricks.

Simple, enough right? So how do these variable work? Remember that organizational change is the result of changes in individual behavior. This is why it’s so important to keep things simple in terms of the Vision and First Steps — it needs to be digestible so that individuals can connect with it. Chances are that you won’t be rolling out any changes in your professional life that are quite as monumental as parenthood — and yet, it is useful to know that massive, everlasting changes are possible when the variables are tuned thusly.

With those variables under our belt, let’s switch to some tactics. These are taken from the book Switch: How to Change Things When Change is Hard, by Chip and Dan Heath. I love this book for its simple language and fantastic examples.

The first tactic is looking for — or creating — Bright Spots. These are examples of things that are going well, or at least going better than average in your current environment. Something like a high performing team that, for example, has better up time that its peers, is a great use case. Pilot programs can also be used to create Bright Spots to demonstrate success and learn a few thing before the grand roll out.

To tie back to our equation for a second, I tend to associate this tactic with Dissatisfaction with the current state and a compelling Vision of the future — it’s proof that the vision is possible AND that you don’t have to settle forthe current state.

Next are Scripting the Critical Moves and Shrinking the Change. These tactics work together to make the first steps to participating in the change (F) achievable. Another way of putting it is that scripting the critical moves shrinks the change. Think about 1–3 small but important things that a person can do to participate in a change, and voila — you’ve mastered these tactics.

What’s a quick example of scripting a critical move to shrink a change, say, in shifting development teams to be more in tune with how their services behave? Something like suggesting a 5 minute team review of logs every morning could go a long way to getting developers to think about recurring patterns in behavior in their services, and encourage them to think proactively about designing for things like observability before coding a new feature. I saw this work really well when I worked in fintech on a trading and accounting system for hedge funds. The US teams would come in in the morning and see what happened overnight in the APAC markets to get an idea of possible trading behavior for the day ahead — and to get ahead of anything nasty that might be brewing. Likewise, teams that did this tended to build and test their new features really effectively, simply because they understood how the systems they owned tended to be wailed on.

Choreographies

Are we in dance class? Sort of!

What I propose below is an example of some things you could try. A recipe book for resolving a version of this scenario in your organization when I know nothing about your stack or your culture would be pretty hilarious, wouldn’t it? You have to come up with your own dance moves from the nuggets that I’m giving you! Your organization has its own rhythms that I could never deign to comprehend.

Regardless of these infinite complexities, your job is to make things as digestible as possible for people so that change isn’t a big deal, and yet yields great results. Address any perceptions of ‘loss’ upfront. Look for some bright spots to see what’s working, and figure out some small, critical moves to script to shrink that change that will lead us to replicating more of that bright spot.

Scenario

So — What’s a common reason you get paged? Sally’s service is broken in production. Oh no!

How do we use Bright Spots to think through the change we need to shrink by scripting critical moves so that we can avoid this from happening as often as it may?

To be clear, these suggestions are not for in the moment remediation. You have to deal with that. But after that’s dealt with, what we want to get to is prevention.

What’s a bright spot we can look for? A high performing team is a great place to start.There are so many things about high performing teams to dig into, but I would recommend looking at how they communicate both within and without their team, and how they get knowledge that allows them to deploy with confidence. How do they get ahead of production incidents through their architectural decisions around observability and testability?

And that leads us to thinking about what the changes are that we need to shrink for Sally. Clearly, we need to help her and her team mates know more about production and its constraints, and influence them to think about these things before writing code. Knowledge and proactive habits are going to do a couple things — it will increase her confidence to deploy, which is great because we want to empower people. Empowerment will likely increase the ownership she takes over her service, too.

So what moves can we script for Sally to increase her knowledge, or what’s a move we can make to give Sally more knowledge, confidence, and ownership over uptime in a way that’s digestible? Something like a 3 minute video outlining what the 2 most common failure modes of services in your stack are is a great place to start, with explanations of where to look for them in your tooling, and symptoms that tend to crop up when these nasties start to rear their head. What about a quick video tutorial for setting up that new observability tool your company invested it? Whatever the medium is (a quick bulleted list of Top of Mind Tidbits is also great), keep it to 5 minutes or less to consume.

You could play a clip or chat through these Top of Mind Tidbits at something like an office hour — or half hour — that takes place every other week. Office hours are great places to talk about other valuable information and source other pain points or information deficits to address.

You could also federate this knowledge through something like a developer <> ops liaison pilot program. Maybe there’s a developer who seems keen on ops whose manager is cool with her being a guinea pig to carry forward some education on your team’s behalf through something like a 2 minute ops recap at daily stand ups. Maybe you borrow her for one hour per week to pass on this knowledge, and you try that out for a month. You can supplement this program with bench marking and analysis of that team’s stats to see if this structured knowledge federation made a difference. Don’t forget to sweeten the deal by chatting with HR to see if you can get ops knowledge into the career ladder for developers too.

There are so many options to align with the realities you work with day to day! And none of this addresses another simple thing you can try, which is learning about the planning cycles and methodologies that engineering uses (e.g. Agile) so that you can have early architecture discussions around observability and testability before code gets written for a new service.

Dont Forget to Be Nice to Yourself and Get Credit for your Work!

I never cease to be amazed at the extent to which software and the operations that support them (which also involves a bunch of software) are all about people. People are the best puzzles you will ever solve. As such, marrying together technical systems with human behavioral systems means that you are dealing with a combination of systems, each with thousands of moving parts. Maybe millions. Probably millions, actually.

Anyway, this stuff is hard. Don’t be hard on yourself. You are going to make mistakes. Learn from them, and you will inevitably have another chance to tweak your approach and try again. Also, make sure to keep your learning curves small by starting small. A biweekly office hour is a great place to start. A 1:1 with the leadership of a high performing team to learn about how they work is another great example. Just start sharing information and asking questions (and then share what you learn!), see what you learn, and let that inform your next steps. Humility is key to this process.

Make sure you get social with these nifty new tactics you’re trying. If you’re an individual contributor, chat to your manager and team mates about sharing ownership of an office hour. Managers, get your team involved in crafting the agenda for an office hour and leading it, or ask them what the top 3 things would be that they would want to teach developers about ops for a liaison program. Whatever the case, you will have more success by building a system of knowledge federation that can be maintained by everyone around you. If you create enough value in this system, everyone will want to pitch in because they love having the transparency it provides.

Becoming a force multiplier by federating information in a structured and predictable way for everyone’s benefit should have a few wonderful outcomes. As you learn and have success with this mission, put it on your resume! Use these successes to justify your upcoming promotion or merit increase! Use them to prompt that conversation to begin with! And if you’re not getting traction with creating all this value, use it to land your next job where this skill is valued!

Anyways, jump in, be humble, have fun, learn, and try again :)

Related posts:

National credentialing organizations play a vital role in ensuring that healthcare professionals have the necessary knowledge and skills to provide safe and effective care to patients. These…

What are you going to do with the time you've been given? Many people will define it as per their choice and their thinking. The answer to this question matters because you can't get wasted time…

A quick introduction to redis, its a Cache!. Redis is an open source, a key-value store. It is often referred to as a data structure server, since the keys can contain strings, hashes, lists, sets…