The socio-technical aspects of microservices: teams that depend on you

This article is part of the series “the socio-technical aspects of microservices”.

When you switch to a microservice architecture, you should expect a change in your team’s effectiveness. When done well, you might unlock the full potential of your team. When done badly, however, all progress might grind to a halt.

If it feels like your microservice architecture is slowing you down and you’d like to do something about it, then this article is for you!

Today we’re going to talk about the importance of maintaining good relationships with the teams that depend on your service.

First, we’ll look into five advantages of good relationships. These are derived from my personal observations — over the past couple of years I’ve worked with several teams and a handful of systems based on microservice architectures.

Second, I’ll mention a few techniques that you can apply to build the kind of relationships that will make your microservice architecture more successful.

1. Good relationships lead to better solutions

A microservice architecture makes it possible for teams to focus on small areas of the business. This focus helps teams to go deep and to build real expertise in that particular area.

This is a good thing, but there’s also a downside to it: the more a team focuses on a small area, the more it will lose track of the bigger picture.

Here’s an example of what I mean: One of the teams I worked with had to get their service to scale up to additional load. Unfortunately, the service was running into rate limits from some of its dependencies. We believed that the problem was our fault and that we should find a solution on our end. But despite our best efforts, we couldn’t find a good solution. All the options we came up with had some undesired tradeoffs.

Things looked pretty bleak for a while, but suddenly our fortune changed! We discovered a straightforward solution to the problem.

What had made the difference was the way we approached the problem: instead of thinking of the problem as something caused by our service, we considered the problem holistically. Together with the other teams, we realized that the majority of the load on our service was unnecessary. We changed a few lines of code in one of our service’s consumers and soon the problem was solved.

There’s a tendency for teams to focus on their services too much. This is problematic, because [what really matters is not the performance of the individual services, but the interactions between the services.

How much value is there in having a perfectly crafted microservice, if it doesn’t work at all with any other service?

We can overcome this unhealthy obsession with individual services, if we bring members from different teams together on a regular basis. This allows teams to break down knowledge silos and to consider how the system works end to end.

2. Good relationships protect you from nasty surprises

The more popular your service becomes, the more load it typically has to sustain. Services that are in high demand face difficult tradeoffs. As a result, those services are finely tuned to particular access patterns.

A service that needs to handle tens of thousands of reads per second and with barely any writes will be designed entirely different from a service that is write heavy.

As long as your consumers stick to your service’s regular access patterns, all is well. A sudden change in those access patterns, however, could lead to a nasty surprise.

Once, my team’s service became unavailable for quite some time, just because one of the consumers rolled out a few changes. These changes used our service differently from how it was intended to be used. As a result, our service was performing many more expensive operations. It just couldn’t cope with the load and so it became unavailable.

The changes made by the other team were rolled out all at once. Since we hadn’t been consulted about these changes, we were entirely unprepared.

Of course we learned our lesson from this incident and started to put in rate limits for our APIs. But rate limits can’t solve every problem.

This is why having good relationships with your service’s consumers is key. They allow you to know about upcoming changes a lot earlier and to work towards safe migration strategies together. And that means less nasty surprises.

3. Good relationships help to resolve incidents faster

Even if you do your best to design your microservices to be fault-tolerant, things will still go wrong occasionally.

When you have an incident at hand, your first goal is to find the root of the problem quickly.

Coming up with a good diagnosis is no trivial task. It requires a solid understanding of the whole system, not just the individual microservices.

The most important question to answer is “how do these services collaborate to accomplish their goals?”. Once you’ve formed an understanding of the end-to-end flows through your system, you often need to drill down into the details of a service.

Things would be easy if a single person could fit all that knowledge into their head. But a system made of microservices is too complex for that.

So when an incident occurs, the first step you should take is to bring together all the experts. This tends to be a lot easier when you already know who the experts are and can bring them in directly.

When your team has good relationships with all its consumers, you’ll know exactly whom to bring in to resolve the incident and who else could be affected by it.

Build solid relationships during the good times and you’ll be able to rely on them in bad times. This can make a big difference in how quickly you’ll be able to resolve incidents.

4. Good relationships ease API deprecation

I once worked on a team that maintained a service with over 70 different API endpoints. Although the majority of those endpoints were marked as deprecated, all of them were still in use by our consumers.

A service with so many API endpoints is quite difficult to maintain. Every time you want to change something, you have to carefully consider who might be affected by the change.

To reduce the cost of maintenance, we decided to remove the deprecated endpoints. But first, we had to migrate our consumers away from the deprecated endpoints.

This could have been an easy task if we had known who our consumers were and how they were using our service. Unfortunately, we didn’t. As a result, we needed to do a lot of research to come up with answers for our questions.

If our team had maintained good relationships with all of our consumers, it would have been straightforward to come up with a plan to remove the deprecated API endpoints and to therefore drive our maintenance cost down.

5. Good relationships can drive your roadmap

Pretty much every team that I worked with had a lot more work to do than it could possibly complete.

To some extend, microservices make this problem worse. There are many aspects that you need to consider separately for your service: it needs to be kept secure, compatible with the platform that it is running on, and scale to increasing amounts of load, just to name a few.

There’s only one effective technique to apply in these situations: You must prioritize the work ruthlessly and say no to everything that didn’t make it to the top of your list.

But what should you prioritize? This is a difficult question, but it gets easier to answer when you bring in your service’s consumers. Your consumers can tell you what is most important. Does your service need new capabilities or does it need to be more resilient? Is the performance okay or does it have to be improved?

I found that listening to your consumers’ feedback makes all the difference.

In my experience, services that were developed together with the consumers solved actual problems and, as a result, were widely adopted. On the other hand, services that were developed following some grand plan often struggled to find any adoption at all.

I’m convinced now, how do I get started?

If you don’t yet have good relationships with your service’s consumers, you should probably start building them. There are different approaches to do so and you should pick the one that fits your particular situation.

If your service is relatively new and has no more than a few consumers, a great approach is to focus on face-to-face time. You’ll probably benefit from weekly or fortnightly meetings. In my experience, you don’t even need an agenda for these meetings to be useful. Quite the opposite: unstructured time makes it easier to build personal relationship and provide a forum to share early ideas.

When your service becomes really popular, you’ll no longer be able to build the same kind of relationships. You simply won’t find enough time to talk to everyone on a regular basis. At this point in time you should focus your efforts on documentation. High quality docs will help your consumers to be successful, even without your explicit help. It’s important to note that maintaining high quality documentation is a lot of effort. But if your service is popular enough, the investment will definitely be worth it.

Which approach you pick depends on your particular situation. Personally, I prefer to build relationships with my service’s consumers using face-to-face time for as long as possible.

What’s next?

In this article we’ve talked about the relationships with consumers that depend on your service. In the next article, we’ll look at the other side of that relationship. We’ll discuss how you can benefit from building good relationships with the teams that your service depends on.