As an engineer, I thought my role was the hardest—handling complexity, debugging, and shipping features—until I became a manager and faced the same question from others.

“What does the day-to-day of a manager look like?”

This question arises because much of a manager’s work is invisible.

Unlike an individual contributor, whose progress is often measured by the code they write or the features they deliver, a manager’s work shows up in dozens of conversations, decisions, and interventions that are rarely obvious from the outside.

A manager might begin the day planning to review a design document and end it by resolving a conflict between teams, coaching an engineer through a difficult career decision, responding to a production incident, aligning stakeholders on a roadmap change, and making several high-impact decisions.

A manager’s day-to-day work can sometimes be unpredictable, filled with constant context switching and shifting priorities, making every day fundamentally different from life as an individual contributor.

These tasks don’t show up on sprint boards but often have more impact than any single feature a manager could deliver.

This invisible nature of management creates an interesting challenge. Unlike engineering work, where completed tasks provide a clear sense of progress, management often happens through conversations, decisions, coordination, and influence.

Without a clear mental model of your day-to-day activities, it’s easy to feel like you’re spending all day reacting to problems rather than accomplishing meaningful work. This is where a lack of fulfillment often comes from for people taking a leap into management from an IC role, because you can’t easily point to that shiny new feature you shipped.

In a previous post, I wrote about the four areas of responsibility of engineering managers. This post focuses on something more practical: the day-to-day activities that allow managers to guide effectively.

Although these activities may look different depending on whether you’re a frontline manager, director, or executive, they all follow the same underlying cycle:

• 1. Gather information
• 1. Share information
• 1. Make decisions
• 1. Influence decisions
• 1. Lead execution
• 1. Plan

Most of a manager’s day is spent repeatedly moving through this cycle. The better you become at each stage, the more effective you’ll be as a leader.

Information gathering

Every decision, proposed path, and action depends on having an accurate understanding of what is happening within the team, the organization, and the product.

When you are an IC, you have a complete picture of your work. Once you become responsible for people, you can’t use the old methods anymore. You’ll need to be much more intentional about how you gather information.

Information will come from every direction—engineers, managers, customers, executives, dashboards, incidents, metrics, and product feedback. The challenge is dealing with and processing the plethora of information.

No manager can process everything. You’ll need to build capability to filter out the noise and synthesize information into a coherent picture of reality.

Without an accurate grasp of the current state, making great decisions is hard. You might solve the wrong problems, prioritize the wrong work, and miss emerging risks. Thus, a significant portion of your time will be spent gathering context, validating assumptions, and building a deeper understanding of what is happening around you and beneath the surface.

Much of engineering leadership can be viewed as a cycle: gather information, develop understanding, make decisions, and influence outcomes. The quality of each step depends heavily on the quality of the information collected at the beginning.

There are three primary areas where engineering leaders gather information: people, process, and product. I hope to get into this in future posts.

Information sharing

If you think of information gathering as collecting context, then information sharing is about synthesizing that information and ensuring its free flow to those who need it to make well-informed decisions or execute effectively towards the intended outcome.

As a manager, you’ll have access to a broader set of information than the people you lead. You’re expected to maintain not only your team’s context, but also an understanding of the broader organizational context, and to bridge the gap between the two.

Each meeting you attend will present an opportunity for you to gather but also share information. You may gather information from one-on-ones with your reports, company all-hands, conversations with fellow engineering managers, product reviews, metrics reviews, discussions with stakeholders, and regular conversations with your manager and skip manager.

Over time, you’ll accumulate pieces of information that may be valuable to an individual, another team, or your entire organization. Synthesizing this information means deciding what is worth sharing, with whom, when, and at the appropriate level of detail.

Some of the most challenging moments for information sharing I have come across came during organizational change: layoffs, reorganizations, project cancellations, or the departure of a key team member. In these situations, simply communicating the facts isn’t enough. People are looking for clarity, empathy, and confidence. You’ll need to communicate honestly while providing enough context to reduce uncertainty and help the team process the change. The way you deliver difficult information can have as much impact as the information itself.

Decision Making

A large part of any leader’s role is making decisions. Leaders make countless decisions, and you’ll have your fair share as well. And your effectiveness will, in part, depend on the quality of the decisions you make and your ability to execute them. You’ll make decisions about priorities, people, technical direction, investments, risks, and trade-offs every day.

This is also one of the hardest parts of an engineering manager’s role. Decisions are rarely straightforward. More often, you’ll face situations where the available information is incomplete, the options are ambiguous, and the trade-offs are unclear. Waiting for certainty is often a decision in itself—one that can carry its own costs. Some decisions will cost you sleep, agony, and pain, especially when they cost people their jobs. Yet, they still need to be made.

Other times, you will have a conviction about the decision to make but must first align before making it.

Decision-making is a skill that develops through repetition, reflection, and experience. The more decisions you make—and the more you examine their outcomes—the better your judgment becomes. Over time, you build intuition for what matters, what can be ignored, and when to take calculated risks.

Good decision-making starts with understanding reality. This is why information gathering is such a large part of your job. Without that foundation, even well-intentioned decisions can miss the mark. With it, you’re better equipped to make sound decisions and help your team succeed.

Influencing decisions

A significant portion of your time will be spent influencing decisions. Some people think of it as more of an organizational politics; others think of it less so, but at its core, influence is simply the ability to help others understand a perspective, evaluate trade-offs, and make informed decisions.

No team or organization operates in isolation. The actions—or inaction—of other teams can directly impact your team’s ability to succeed. You will rarely have sole decision-making authority over everything that affects your team. Instead, you might have ample chance to provide context, information, data, and perspective that help shape decisions.

As your scope grows, your impact depends less on the work you do yourself and more on the decisions you can influence across the organization.

You will need to influence peers to gain alignment across teams and advocate for your team’s needs. You will need to influence senior leaders by providing clear recommendations and helping them understand trade-offs. You will also need to influence the engineers on your team, guiding them toward better approaches while creating space for them to learn and grow.

I often spend a significant portion of my time figuring out how to get another team to prioritize work that advances my team’s goals. The challenge is rarely that people disagree with the objective; it is that every team has competing priorities, limited capacity, and their own pressures. Before asking for support, I need to understand their goals, constraints, incentives, and timelines. Only then can I frame the request in a way that makes sense from their perspective.

Over time, I have learned that many leadership challenges are not fundamentally technical problems. The technical solution may be straightforward, but success depends on aligning the people responsible for implementing it.

Leadership is less about authority and more about influence. The ability to align people around a shared outcome often determines whether an idea becomes a reality.

Leading and setting directions

Leading and setting direction are among your most visible managerial activities. It’s about helping a group of people move toward a shared goal while creating the conditions that enable them to succeed.

You’ll often spend your day defining a clear path toward a destination and helping your team understand why it matters. A compelling vision gives people purpose, helps them make better decisions, and inspires them to work toward a common outcome.

You may spot trends, technical limitations, or emerging business needs before the rest of the team. For example, you might see that a piece of technology the team depends on is becoming difficult to maintain, or that the current architecture will not scale to meet future growth demand. In these situations, your role is to define a clear direction that moves the team from its current state to a better future state. You’ll spend your time explaining the rationale for the change, why it needs to be done, and helping them buy into it.

At other times, the direction is given to you rather than created by you. You might receive a new business goal or organizational priority. You may spend your day translating that goal into something meaningful for your team—helping them understand why it matters, the opportunities it creates, and how their work contributes to achieving it.

Another part of your day-to-day managerial responsibilities is serving as a role model. Teams pay far more attention to what their leaders do than to what they say. If you expect engineers to write thoughtful design documents, participate constructively in reviews, communicate clearly during incidents, or take ownership of problems, demonstrate those behaviors consistently yourself. By modeling the standards you expect, you make those expectations tangible and credible.

For example, imagine your team consistently struggles with incident management. Simply telling people to improve is unlikely to change behavior. Instead, participate actively during incidents and demonstrate effective practices: communicate clearly while triaging issues, coordinate responders, keep stakeholders informed, and facilitate productive post-incident reviews. As the team repeatedly observes these behaviors, they develop a concrete understanding of what good looks like. Over time, these practices become part of the team’s culture, and the team’s overall effectiveness improves.​

Planning

Planning is often viewed as one of the more bureaucratic parts of management. Most people think of quarterly planning, annual roadmaps, and sprint planning. It’s easy to think of planning as something fixed and done within a specific time of the year or month. At least, I used to think about it that way. While those are part of planning, they are not all there is to planning.

I think of planning as deciding how you’ll move from the current state to a desired future state. It could appear in a document that outlines how to approach a task before executing it to achieve a specific goal. It could show up in a document forecasting headcount, budget, and projects for the next quarters.

Every meaningful change requires a plan. Whether you’re improving a process, migrating to a new system, growing your team, or developing your team, planning is the activity that connects today’s reality with tomorrow’s goals.

Once you think about planning this way, you begin to see it everywhere in your day-to-day. And it’s one managerial activity.

Think of the time you wanted to migrate to a new database. You created a plan, right?

Think of the time your team was delivering more slowly than expected? You identified the underlying causes and created a plan with concrete steps to get the execution moving again.

Remember when you were asked to move a goal by X%? You thought hard and developed a way to realistically reach that target and what is needed. That is also a plan.

Remembered when your team was wallowing in incidents? You figured out the root cause, and outlined and discussed how to get the team out of that with your managers? That is still also a plan.

As you lead your team, there will always be a desired future state, no matter how little that is. You will need to articulate what that desired state is, what needs to be done to get there, and who is doing what. This is planning, and a chunk of your day will be spent on it.

In conclusion, many of your day-to-day activities will fall into most of these categorizations, and they’re managerial activities. Activities through which managers lead.

Most people feel pressure to prove they were the right hire. They rush to make decisions, solve problems, and demonstrate impact. By the way, that used to be me.
I have learned that the strongest starts rarely come from moving the fastest. It comes from laying the right foundation and approaching your first few weeks strategically.

Over the years, I’ve found there are a handful of things that consistently set new managers up for success in their first few months.

Following these will help ensure you rock your first few months at a new job, no matter how intense.

Make the Most of Your Onboarding

Depending on the size of the company, onboarding sessions for managers can last for days or weeks. If you’re joining a bigger tech organization with decades of technology and product, you can be sure there will be a lot of information to take in. Companies like Google and Facebook are known for planning six-week-long onboarding.

As a manager, your onboarding isn’t only about learning the business but also about learning how the company expects its managers to lead.

Essentially, what is important at this stage is that you become a sponge. Absorb everything, take notes, and ask questions.

Every company will have something I call a management system, whether written or unwritten. Sometimes it’s documented in leadership principles or management training. More often, it’s reflected in how decisions are made, how managers are expected to lead, and what behaviors are rewarded.

For example, one company may believe managers should invest heavily in coaching low performers before considering an exit. Another may believe that once it’s clear someone isn’t the right fit, it’s better to move quickly. Neither approach is inherently right or wrong, but each reflects a different philosophy about management.

These beliefs influence everything from performance management and decision-making to communication and team culture. Understanding them early will help you navigate the organization more effectively and avoid unintentionally working against its norms.

As you learn the company’s management system, reflect on how it aligns with your own leadership philosophy. You don’t need to abandon your principles, but you do need to understand the environment before trying to change it.

Build Your Network Early

One of the best investments you can make during your first few weeks is in relationships.

As a new manager, you’ll spend a lot of time learning the systems, processes, and technology. But don’t overlook the people. Strong relationships will help you ramp up faster than any documentation (sometimes outdated) ever will.

A great place to start is with the people you met during the interview process. They’ve already met you, which makes it easier to continue the conversation now that you’ve joined.

A simple message on Slack (or whatever your company uses) is often enough:

Hey XYZ,

I’m ABC. You interviewed me for this role, and I just started today. I’m really looking to hit the ground running. If you don’t mind, I would like to schedule a quick chat to learn a bit more about your team and projects, and to hear anything you think would be helpful for me to know to be successful here. Really looking forward to speaking with you again.

Once you’re inside the company, people are often willing to share the context they couldn’t discuss during interviews. They might tell you where teams collaborate well, what challenges the organization is facing, and what they think is important for a new person joining to know.

Don’t stop with your interviewers. Schedule introductory conversations with your peers, product managers, engineering leaders, designers, and other cross-functional partners. Ask thoughtful questions, listen carefully, and take notes. More on this later.

These conversations will help you build a mental map of the organization. More importantly, they’ll establish trust before you need it. Months later, when you need to solve a problem or navigate a complex decision, you’ll already have relationships to lean on.

Resist the Urge to Prove Yourself

“Hey, I’m the new sheriff here now!”, ”Can I quickly chime in?” “Here are my 2 cents!”, “I have something to say”…, blah … blah!.

Calm the heck down, especially in your first few days.

One of the biggest mistakes new managers make is feeling they have to demonstrate their value immediately. You join a new company, open Slack, and suddenly you’re surrounded by conversations you only partially understand. New acronyms, unfamiliar systems, inside jokes, and technical discussions are happening everywhere.

It’s tempting to jump in.

After all, you were hired for your expertise. You want to show you’re capable. You want to prove the company made the right decision.

Resist that temptation. That is self-imposed expectations.

In your first few days, it’s likely you haven’t formed the correct reality to have the best opinions. Every company has its own context, history, constraints, and ways of working. Even if you’ve solved a similar problem before, you don’t yet know why this team does things the way it does. Advice without context is often worse than no advice at all.

One thing that helps is setting expectations early. Tell your team and everybody that your first priority is learning.

You might say something like:

Hey everyone, I’m a new manager here. Here is XYZ about me. My apologies in advance because I’m going to be asking a lot of questions. Most of the time it’s for my own understanding. In the next few days, I’ll focus on learning as much as possible from you all about the project and gathering context. But if you need me to explicitly weigh in on anything, please let me know so I can.

That simple message accomplishes two things: it gives you permission to learn, and it gives your team permission to teach you. By setting this expectation, you’re communicating that you don’t know it all and that you’ll spend some time learning.

Meet Your Manager and Align on Expectations.

One of the biggest sources of anxiety for new managers is not knowing what’s expected of them.

One of the best things my manager did when I joined my previous company was document what success looked like during my onboarding and beyond. It gave me clarity on the organization, the team’s priorities, and, most importantly, what I should focus on first.

Not every manager will do this automatically, so don’t wait for it. Schedule time with your manager early. Essentially, the goal of that call is to quickly touch base on what you should have access to, whether there are any priorities your manager thinks you should address first, and to align on expectations.

This is not all there is to this; you need to be explicit and break expectations down. I often see people ask, “What are your expectations for me in this role ”? By the way, that is not a wrong question to ask. The problem is it’s not just a detailed question to ask.

The expectations for someone in their first two weeks shouldn’t and won’t be the same as those for someone who’s been in the role for six or 12 months.

The questions you want to make sure you ask are:

• What do you expect from my onboarding, and how would you measure the success of my onboarding?

• How would you measure me in my first 3 months?

• What do you want me to focus on in my first 3 months?

• After my onboarding, what does success look like in this role after 1 year?

Now, you are separating onboarding expectations from role expectations, which gives a clearer understanding of how you should spend your time in the first 3 months and what success looks like during those 3 months, and afterward.

Build Trust With the People You Lead

When you start as a new manager, you may have the title and responsibility on day one, but you don’t yet have the trust of the people you lead, and that’s normal.

I like to think of trust as a bank account. When you join a new team, you start with an empty account. Every interaction, decision, and follow-through is a deposit that builds trust over time.

Building that trust starts with getting to know your team.

Depending on the company, you may or may not have met your future team before joining. In big tech companies, interviews are conducted through centralized hiring processes, and successful candidates are matched to teams afterward. That means you may start your first day without having any existing relationships with the people you’ll manage.

Don’t wait for those relationships to form naturally.

Schedule introductory calls with each of your direct reports. These first calls should not be about status updates or project reviews. They should be about understanding the person behind the role.

Learn about their background, what brought them into their career, what motivates them, what they enjoy working on, and what they need from their manager to be successful.

Also take the opportunity to share about yourself:

• Your background and experiences

• How you approach leadership

• How you communicate and make decisions

• What your team can expect from you

A manager README can be a great way to start this conversation. Sharing it before your first meeting gives your team a chance to understand how you work and creates a foundation for a more meaningful discussion.

Remember: your team doesn’t need you to prove everything on day one. They need to understand who you are, how you operate, and whether they can rely on you.

Trust is built one interaction at a time.

Earn Your First Wins Through Listening

Earlier, I mentioned that you shouldn’t put pressure on yourself to prove your value on day one.

That doesn’t mean you should wait around and avoid making an impact.

It means your first impact should come from understanding where your team and organization need you most.

The best early wins will not come from you walking in with a list of changes you want to make. They come from listening, identifying patterns, and focusing your energy on problems that actually matter.

Start by meeting with the people around you:

• Your direct reports

• Your stakeholders

• Product partners

• Design partners

• Data and other cross-functional teams

In each conversation, spend most of your time listening. At the end, ask three questions:

1. What is working well today?

2. What is not working well today?

3. If there was one thing I could change to make the team more effective, what would it be?

These questions force people to reflect. Some answers will be small. Some will be contradictory. But over time, patterns will emerge.

You’ll start hearing the same challenges from multiple people. Those repeated themes are often where your first opportunities are hiding.

Use those insights to build your first roadmap. Prioritize the problems where you can create meaningful improvement while also building credibility with your team.

Your first wins shouldn’t come from proving that you have all the answers instead from understanding the problems worth solving.

I hope this helps someone starting a new management role avoid the pressure to prove themselves too quickly, focus on building the right foundation, and create meaningful impact over time.

The differences can be significant. What counts as performance management or hiring practice in one country can look completely different in another. Even companies on the same street can operate with very different cultures, processes, and expectations.

But despite all that variation, I’ve found that a handful of principles consistently help build high-performing teams and raise the bar on performance. This post focuses on the people side of those principles.

Getting to the basics of a manager’s job is to enable your team to do its best work and, through that, drive the organization’s success. Performance is simply how well and how efficiently that work gets done.

Raising performance is challenging in any environment. It’s even harder when teams are distributed across countries and time zones. In those situations, the place to start is not process or tooling—it’s people.

Get to know individuals and their goals

A great team is built on great relationships.

People are motivated by different things. Some value stability. Others want rapid career growth. Some are driven by learning opportunities, while others are motivated by autonomy or impact.

You won’t discover those motivations by accident. You uncover them by getting to know people individually: understanding their strengths, goals, interests, and what they want from their careers.

Some engineers have exceptional product instincts. Others thrive in infrastructure, architecture, or operational excellence. A good manager understands where people perform best and finds opportunities to align those strengths with business needs.

Once you understand what motivates someone and where they excel, you can make better decisions for both the individual and the organization.

Help paint a path to achieving the goals

Understanding people’s goals is only the beginning. The next step is helping them see a realistic path toward achieving them.

I recommend documenting what you know about each person: their ambitions, strengths, interests, and areas they want to develop. The path won’t always be perfectly clear, but having that understanding helps you recognize opportunities that are best suited for individuals when they arise.

Use your one-on-ones to discuss their goals and collaborate on a plan. People are far more engaged when they can see how today’s work connects to tomorrow’s aspirations.

A good manager understands what each person is reaching for, paints a picture of what they can become, and helps them take meaningful steps in that direction.

Earn trust by showing you care

The managers I’ve enjoyed working with most were the ones I trusted to have my best interests at heart.

Trust grows when people believe you’ll support them, be honest with them, and help them succeed. It doesn’t mean always giving the people you lead what they want. It means consistently acting in a way that shows you care about both their success and their well-being.

Be transparent. Give honest feedback. Advocate for your team when appropriate. Follow through on commitments.

When people know you’re invested in their success, they are far more likely to trust your judgment and go the extra mile when it matters.

Set high standards and hold everyone to them

A high jumper always knows where the bar sits. They can see it, train for it, and immediately tell whether they’ve cleared it.

Performance works the same way.

If you never define what good performance looks like, people have no reliable way to measure themselves, and managers have no consistent way to assess growth.

Career ladders help, but they are often too broad to capture the expectations of a specific team. A set of behaviors required to succeed in an infra team might be different from what is required in a consumer product team. A company-wide ladder can be too general to capture the nuances. Every team should clearly define the behaviors and standards it expects from its members.

This belongs somewhere visible and accessible, such as your team wiki. Examples might include:

• How to react when you’re blocked and how to get unblocked

• What good code reviews look like

• Communication expectations

• Ownership expectations

• What strong collaboration looks like in practice

Defining standards is the easy part. The difficult part is holding everyone accountable to them consistently.

Maintaining a high bar means helping new team members understand expectations, coaching people who are falling short, and making difficult decisions when performance does not improve despite support and clear feedback.

Give feedback from a place of care

Nobody is perfect. Feedback is one of the most effective tools managers have for helping people improve and raising the overall performance of a team.

Many leaders avoid direct feedback because it can feel uncomfortable. As a result, messages become diluted, delayed, or hidden behind vague language.

Effective feedback comes from a place of genuine care. You have to care about the person receiving it, and they have to believe that you do. Even difficult feedback is easier to hear when people understand it is intended to help them succeed.

Be direct. Be specific. Be timely. Focus on behaviors and outcomes rather than personal judgments.

The goal of feedback is not to make people feel good or bad. The goal is to help them grow.

Balance individual needs with business needs

One of the important things you have to do as a manager is aligning what individuals need with what the business needs.

When you understand people’s goals and strengths, you can often create opportunities that benefit both. An engineer who wants broader influence might lead a cross-team initiative. Someone looking to deepen technical expertise might own a critical platform investment.

The strongest performance often comes when personal ambition and business priorities point in the same direction.

People rise to challenges when they understand the path ahead and believe they can succeed. Your role as a manager is to help them see that path, remove obstacles where possible, and support them as they walk it.

High-performing teams are built by high-performing people. And high-performing people are most often the result of managers who understand them, challenge them, support them, and hold them to a clear standard.

When we think of leaders, we think of extraordinary people we have a lot of respect and positive emotions towards.

“Leaders” sound impressive and they are. They inspire people, create movements, and challenge the status quo.

In engineering organizations, they might stand in front of whiteboards drawing bold visions of the future while everyone nods in admiration.

Managers, on the other hand, are often portrayed as people who schedule meetings, update spreadsheets, and ask uncomfortable questions about deadlines.

The former sounds like a visionary and the latter sounds like someone asking whether you’ve updated Jira.

Unsurprisingly, most of us want to be leaders. Very few grow up saying, “One day, I hope to become excellent at resource allocation.” Or should I say “a manager”?

This perception is understandable. Most of us can easily recall examples of inspiring leaders who influenced us. Fewer can point to examples of exceptional managers. When management is done poorly, it often feels bureaucratic, controlling, or disconnected from the work itself.

Over time, this has created a false impression that leadership is inherently more valuable than management.

The reality is more complicated.

Organizations do not succeed because of leadership alone, nor do they succeed because of management alone. They succeed when both skills are developed and are present and working together.

Imagine an engineering organization embarking on a major platform shift. It could be a big migration or a rewrite.

Someone must help the organization understand why the change matters. Someone must set the direction and show others the way. That person might create alignment around a shared vision of the future. They might influence people to move beyond the comfort of the current state. That is leadership at play.

But someone must also determine priorities, allocate resources, coordinate dependencies, manage risks, and ensure that work is executed effectively. This person might even spot an inefficient process and optimize it. This is management at play.

Engineering managers deal with people, and dealing with people requires an ability to influence them (leading).

You will also have to make do with finite resources (capacity, time, budget, servers, or GPUs, whatever they’re called these days) to achieve your set goals. Dealing with finite resources requires an ability to make effective use of them (managing).

Both activities are different and sometimes overlap but serve the same purpose. I like to think of it as leadership creating movement and management turning that movement into results.

Leadership without management creates vision without execution. Teams understand where they are going but struggle to get there.

Management without leadership creates execution without direction. Teams become efficient at delivering work that may not ultimately matter or move them towards the mission.

To be an effective engineering leader, you have to understand that leadership skills and management skills are complementary rather than competing. You’ll need to hone your intuition to recognize when a situation requires leadership and when it requires management. More importantly, you have to develop skills in both.

This distinction is particularly important because leadership and management are often misunderstood as roles rather than skills.

You’re a leader; you just don’t know it yet.

Many people assume leadership begins with authority. That is not true.

One of my favourite authors wrote something in his book that almost felt like blasphemy:

“Leaders are angry people.” He then proceeded to say leadership is born when someone is dissatisfied with a situation and determined to pay the price to change the status quo.

Leadership begins when someone recognizes that the current state can be improved and takes the initiative to do something about it.

Think about a time when you initiated a positive change in your own life, helped a family member overcome a challenge, or guided a colleague toward a better outcome.

In each of these situations, you influenced movement from one state to another. That is leadership skills.

The scale may be small, but the underlying skills and intent is the same.

An engineer who identifies recurring operational failures and proposes a more reliable architecture demonstrates leadership.

A staff engineer who aligns multiple teams around a shared platform initiative demonstrates leadership.

A junior engineer who introduces a better way of working and earns the trust of teammates demonstrates leadership.

You’re a leader just like any other leader. What is different is the context, scale, and level of complexity at which you’re exercising your leadership skills.

In each case, someone is helping others move from the current state toward a better future state.

Leadership is less about authority and more about influence. No formal title is required to lead.

This is why some of the most important acts of leadership in engineering organizations come from people who manage no one. Engineers who mentor colleagues, improve team practices, create shared tools, or influence technical direction are exercising leadership regardless of title.

Management, however, addresses a different challenge.

You’re a manager; you just don’t know it yet.

If leadership helps determine where a group should go, management helps ensure it gets there.

Management is the discipline of organizing resources to achieve desired outcomes. Those resources include people, time, budget, attention, tools, information, and organizational capacity.

When I worked at a small company, I thought about how I wished I worked in a bigger company with a bigger budget and infinite resources at my disposal. I joined a much bigger company and learned that resources are finite everywhere.

Because resources are always limited, management is fundamentally about making choices. Prioritization decisions, staffing decisions, and trade-offs between cost and speed are all about making choices and managing finite resources efficiently.

In engineering organizations, management shows up in planning, staffing, performance management, career development, budgeting, risk management, and cross-functional coordination.

These responsibilities may not appear as exciting as defining a vision, but they are equally important.

A brilliant strategy without execution rarely succeeds. Likewise, a motivated team without coordination will eventually lose momentum.

Like leadership, management exists at every level. And you’re already a manager; you just don’t know it yet.

When you prioritize competing tasks, estimate effort, organize a project, or coordinate work with others, you are exercising management skills.

Managing yourself requires managing your time, energy, and attention. Managing a project requires coordinating tasks and resources. Managing a team requires balancing priorities across multiple people.

Managing an organization requires optimizing resources across entire departments and functions.

The scale at which you manage, the level of complexity, and the blast radius are different, but the underlying capability remains the same: organizing limited resources to achieve desired outcomes efficiently.

You’re more effective when you develop both management and leadership skills.

For engineering managers, the conclusion is straightforward. You do not need to choose between becoming a leader and becoming a manager.

Leadership helps people understand where they are going and why it matters. Management creates the conditions that allow them to get there.

The best engineering managers develop both capabilities and learn to move fluidly between them. They create direction when direction is needed. They create structure when structure is needed.

They help teams move toward a better future while ensuring that future becomes reality.

For context, Keeplys is a hobby project: a simple document manager running on a mobile phone that I’ve been building in my free time. When you store or scan documents via Keeplys, it organises them into a folder locally on the device.

Keeplys is local-first by default. Your documents live on your device but there is an important question: what happens if a device is lost? How do you recover your documents? It wasn’t hard to see that this was a problem worth solving.

Before building the sync feature, the main question on my mind was: how do you sync files to the cloud? I wanted to be able to scan documents on my phone, organise them locally, and have the files backed up to the cloud.

The easy answer would have been to simply upload files, or download updates once a day. This approach quickly fell short when you consider:

• What if I (or a user) edit a document on my phone while editing the same document on their laptop? How does the app resolve conflicts?
• What if the network drops mid-upload while syncing files?
• What if a user runs out of cloud storage?
• What if someone deletes the parent cloud folder used for backup (Keeplys) directly from Google Drive’s web interface?
• What about iCloud, which isn’t even a REST API?

This is a story of how I built a sync engine that handles all of this.

Implementation guiding principles

I’ve come to understand the power of setting guiding principles when exploring choices and narrowing options down. I leaned on this approach and defined a few principles that I wanted the feature to adhere to.

The app must work perfectly offline

Cloud sync is a feature and never should it be a dependency. Users should never see a loading spinner waiting for the sync.

This means:

• Documents are saved locally first, always
• Sync happens asynchronously in the background
• Pending uploads queue up and execute when connectivity returns

Cloud deletion shouldn’t interfere with what is stored locally

The most dangerous operation in any sync system is delete. Google Drive and iCloud should never silently delete local data.

If a user deletes a file from Google Drive’s web interface, it shouldn’t mirror that action locally. Instead, the document should be marked as unsynced and the user should decide what to do.

Drive agnostic

I want to leave room for extension. The first iteration supports Google Drive and iCloud, but I want to allow for other providers like Dropbox, Box, etc. To support this, the design must be drive agnostic. I designed a single sync engine that talks to a DriveProvider interface.

Documents should have observable state

Documents should have visible sync status: synced, pending, conflict, or unsynced. Users should always know what’s happening.

The domain model

To support the above, I created the following core types:

// A local document
interface Document {
  id: string;
  name: string;
  contentHash: string; // SHA-256 of the document
  modifiedAt: number;
  // ...
}

// A reference to a docume in a cloud drive or storage
interface RemoteRef {
  provider: 'google-drive' | 'apple-icloud';
  providerFileId: string;
  etag?: string; // The cloud's version identifier
}

// A queued operation
interface Operation {
  id: string; // UUID for idempotency
  type: 'create' | 'update' | 'delete' | 'move';
  localId: string;
  retryCount: number;
}

// The sync state for a provider
interface SyncState {
  provider: string;
  changeCursor?: string; // Delta token for incremental sync
  pendingOps: Operation[];
}

One key thing to note is the changeCursor. Both Google Drive and iCloud support the concept of a delta token. Delta token is a cursor that lets you ask “what changed since last time?” instead of listing every file on every sync. This is the difference between syncing 3 files and re-scanning 3,000.

The provider interface

I created a contract that every cloud provider must implement. As I add new providers (e.g. Dropbox), each one must conform to DriveProvider.

interface DriveProvider {
  id: string;
  platforms: ('ios' | 'android')[]; // iCloud is iOS-only

  // Auth
  authenticate(): Promise<AuthResult>;

  // The delta API (critical for efficiency)
  getChanges(cursor: string): Promise<ChangeResult>;

  // CRUD operations
  upload(params: UploadParams): Promise<RemoteRef>;
  download(ref: RemoteRef, destination: string): Promise<void>;
  delete(ref: RemoteRef): Promise<void>;

  // Storage quota
  getQuota(): Promise<{ used: number; limit?: number }>;
}

The platforms field is important. iCloud uses a native iOS API (the Ubiquity Container), so it simply doesn’t exist on Android. platforms gives me a way to control platform-specific availability and features.

Here’s how it’s used:

export function getPlatformProviders(): DriveProvider[] {
  return allProviders.filter(p => p.platforms.includes(Platform.OS));
}

On Android, users see only Google Drive in settings. On iOS, they see both Google Drive and iCloud.

Building Google Drive Sync Feature : REST API Cloud Provider

Starting with the Google Drive integration, I quickly learned that its implementation is very different from iCloud’s.

Scoped access To access Google Drive, the app requests the drive.scope, which limits access to files the app itself created. The app can never see a user’s vacation photos or tax documents.

The delta API Google’s changes.list endpoint accepts a pageToken and returns only files that have changed since that token. On the first sync, I list everything. On subsequent syncs, I might get an empty response—meaning nothing changed. This is massively more efficient than re-listing thousands of files.

Conflict detection via ETag Every file has an etag, which acts as a version identifier. When uploading, the app sends:

PUT /upload/drive/v3/files/{fileId}
If-Match: "existing-etag"

If the etag doesn’t match (for example, if the user edited the file elsewhere), Google returns 412 Precondition Failed. The app catches this and routes the document to a conflict resolver.

Resumable uploads For files over 5 MB, I used Google’s resumable upload protocol. This means if the connection drops at 80%, the app resumes from 80%—not from 0%.

Building iCloud Sync: The Filesystem Provider

iCloud was a completely different beast. It’s not a REST API. It’s a synchronized filesystem managed by iOS itself.

The ubiquity container

iCloud gives each app a special folder called the Ubiquity Container. Files you put there are automatically synced by iOS in the background. To “upload” a file, you simply copy it into this folder:

let source = localDocumentURL
let destination = ubiquityContainerURL.appendingPathComponent("document.pdf")

try FileManager.default.copyItem(at: source, to: destination)

// iOS handles the rest

This sounds straightforward, but it’s not quite that simple. Getting entitlements set up is very different from how Google Drive works.

The config plugin

Since I’m building the app in React Native, it doesn’t know about iCloud entitlements. I had to build an Expo Config Plugin (a big thanks to Claude Code here) that:

• Enables the iCloud capability in Xcode
• Adds the correct entitlements to the app
• Exposes the ubiquity container path to JavaScript

No delta API

Unlike Google Drive, iCloud has no cursor-based delta API. Instead, I used NSMetadataQuery to watch the container for changes:

let query = NSMetadataQuery()
query.predicate = NSPredicate(format: "%K LIKE '*'", NSMetadataItemPathKey)
query.searchScopes = [NSMetadataQueryUbiquitousDocumentsScope]
query.start()

This gives live notifications when files change — more real-time than polling, but also harder to implement.

The sync engine

With providers abstracted, I built a sync engine that:

• Transforms local changes into operations
  • Document saved → create or update operation queued
  • Document deleted → delete operation queued
• Persists the operation queue
  • Stored in AsyncStorage as a “sync journal”
  • Survives app crashes and restarts
• Executes operations with retry logic
  • Transient failures: exponential backoff (1s → 2s → 4s → 8s…)
  • Rate limits: respect Retry-After headers
  • Auth failures: pause sync and prompt re-authentication
• Detects and routes conflicts
  • ETag mismatch → conflict state
  • User resolves: keep local, keep remote, or keep both

The sync journal

Every operation gets a UUID for idempotency. If the app crashes after uploading but before updating local metadata, it can safely replay the operation. The cloud will recognize it as a duplicate.

interface SyncJournal {
  providers: {
    [providerId: string]: {
      cursor?: string;
      pendingOps: Operation[];
      failedOps: Operation[];
    };
  };
}

The failedOps array is crucial. After five retries, I stop retrying and surface the error to the user.

Handling deletion in the cloud

Deletion has a dangerous edge case: a remote delete can destroy user data.

Imagine this flow:

1. User links Google Drive
2. All documents sync correctly
3. User opens Google Drive on the web and deletes the Keeplys folder
4. App syncs and sees files are “deleted” remotely
5. App mirrors the delete locally
6. User’s documents are gone forever

I handled this with a soft-delete with confirmation:

Handling the merge problem

All data backups are synced in a folder named “Keeplys”. One edge case I needed to handle was a case where a user links a provider and there’s already a /Keeplys folder in their cloud drive. Maybe from another device, maybe from a previous install.

I used hash-based deduplication as follows:

• List all remote files
• For each local file, compute a SHA-256 hash
• Compare local and remote files by name + hash

This handles the common case of reinstalling the app on a new phone without re-uploading gigabytes of documents.

Handling sync in the background

I didn’t want users to open the app just to sync. I implemented background sync using expo-background-fetch:

BackgroundFetch.registerTaskAsync('KEEPLYS_BACKGROUND_SYNC', {
  minimumInterval: 15 * 60, // iOS minimum: 15 minutes
  stopOnTerminate: false,
  startOnBoot: true,
});

On iOS, this wraps BGAppRefreshTask. On Android, it wraps WorkManager. Both respect battery optimization and network constraints.

I also added a “Sync on Wi-Fi only” toggle. When enabled, background sync checks NetInfo.type === 'wifi' before proceeding.

Over the past few years, I’ve led growth product teams focusing on retention and growing daily active users (DAU). Working on a myriad of challenges aimed at widening access to a global audience has deeply reshaped my understanding of inclusivity and what it truly means to build for a global audience and leave no one behind.

For a long time, I had a narrow view of inclusivity. I saw it as a nice-to-have, rather than a holistic principle that should influence many engineering decisions. Through experience, I’ve learned that even well-intentioned technical trade-offs can unintentionally exclude certain potential users.

Think of it. Most products start small. They begin as simple ideas, gradually evolve, and eventually find market fit. At that point, the business identifies its core users—the cohort that drives the most value and gives the most feedback. Naturally, the focus shifts toward serving them better.

Teams start listening closely to these users’ pain points, prioritizing them, building new features to address their needs, and tailoring the product experience to them. This approach often feels smart and efficient. After all, it follows the familiar “80/20 rule”—what 20% of effort can deliver 80% of the impact?

It seems like a no-brainer to prioritize the 20% of users who drive most, or should I say 80% of the revenue. But over time, this focus creates an unintended consequence: the remaining 80% of users are quietly left behind. They’re excluded.

They might still show up occasionally—out of curiosity or necessity—but each time they do, the product feels a little more distant, a little less for them. The product speaks a quiet truth: “I know you need me, but I wasn’t really built for you.”

Too often, we default to “design and build for core users” or for users with fewer constraints because it promises better ROI. But the deeper, more important questions are:

• What’s standing in the way of users who should be using your product but aren’t?

• Are your design choices excluding people before they even have a chance to become “core users”?

Throughout my career, I’ve often heard (and said) statements like:

“Let’s prioritize the core users who matter most.”

At face value, these statements seem reasonable—and in many cases, they are. Prioritizing core users often feels like a sound, data-driven decision. But it becomes problematic when the entire development process is optimized solely around these users. Over time, the product begins to diverge, unintentionally excluding minority or edge users. The deeper we go down this path, the more self-reinforcing the cycle becomes.

The Engineer’s Bubble

It’s not just in product decisions; it applies equally to engineering. The context we live in profoundly shapes how we build, how we think, and how we perceive reality.

I’m reminded of a story from one of my German classes. It described how everyone is born wearing an invisible pair of glasses through which they view the world. Some have red lenses, others blue, and so on. Each pair of glasses colors our perception—what we see feels real and objective, but it’s deeply shaped by our lens.

The same is true for us as engineers. We build from within our own cultural and technological bubbles, assuming our experiences and environments are universal—but they’re not. Our daily context is one of privilege:

• We use high-end laptops and modern devices.

• We work with fast, reliable internet and constant power.

• We test our applications on the latest browsers and operating systems.

• We are early adopters of new technologies and easily embrace change.

A few years ago, when React was becoming mainstream, my team was thrilled to migrate our legacy app to this shiny new framework. We couldn’t stop talking about how much better the developer experience would be—faster builds, cleaner code, modern patterns. We imagined the productivity gains and the delight of finally using something elegant and efficient.

We rebuilt core parts of the product, added sleek new features, and gave the UI a fresh, modern look. Everything ran beautifully in Chrome, Firefox, and every modern browser. I remember thinking to myself, “How could anyone not love this?” I couldn’t wait for customers to experience it.

Then we launched.

We checked the logs and usage metrics; excitement quickly turned to confusion. Adoption was down, and a sizable portion of our users weren’t making it past the first few screens. What went wrong?

It didn’t take long to find the answer. It was Internet Explorer. They were using that specific browser. We had tested on every major browser except Internet Explorer. Some of our users were still on IE 8 and 9. The app broke completely for them.

We had assumed, without question, that “everyone” used modern browsers by now. But we were wrong.

That experience was a wake-up call. It reminded me that our reality as engineers is not the same as our users’ reality. The tools, devices, and networks we take for granted are luxuries many don’t know or have access to.

A large portion of the world still uses entry-level devices with limited processing power. Many experience intermittent connectivity or rely on expensive, slow internet in their day to day. When we’re not aware of this reality, our products become less and less usable to certain users who should be using them.

Inclusive product engineering is about widening access. It’s about being intentional in how we design and build software. It’s about writing code that runs efficiently and work for all users. It’s about optimizing yours so it has low battery, data, and memory usage. It’s about making sure no one is left behind.

With constant pressure to ship features quickly, inclusive software design can sometimes feel expensive or time-consuming. But the truth is, inclusion doesn’t always require much from the start. It requires more awareness, discipline, and intentionality from us.

How can you start making your product more accessible to the world?

Start with awareness

• Begin with questions, not code. Challenge assumptions early and often.

• Support accessibility from the start.

• Who are you building for, how diverse are they, and how do they engage in the digital world?

• Who might this exclude?

• If you’re improving an existing feature, where are users dropping off—and why?

• What happens if the user has a slower device or poor connectivity?

Test your product under different constraints

Just as automated tests shouldn’t stop at the “happy path,” feature testing shouldn’t either.

• Throttle network speeds to simulate unreliable internet. Does your app handle it gracefully?

• Test on older, entry-level devices or budget phones.

• Use accessibility tools like color-blindness simulators or voiceover modes to check usability.

• How does the app work with LTR and RTL languages?

Design and code with constraints in mind

• Use lightweight assets and reduce dependency on heavy libraries.

• Prioritize clarity and the primary jobs users need to get done.

• Respect data-saving mode in your code.

Involve diverse voices early

Encourage your research and design teams to broaden their participant groups and ensure diverse representation.

If that’s not possible, ask for feedback from teammates in different regions, on different devices, or with different cultural perspectives. Even small variations in context can surface assumptions you didn’t realize you were making.

In conclusion, focusing on the “core 20%” might feel efficient in the short term, but it risks alienating the very users who could expand your product’s reach and impact. As engineers, our code shapes the world. Making the world more equitable begins by stepping outside our bubble, questioning our assumptions, and designing for the full spectrum of users—not just the ones who look, think, or live like us.

We all learn through feedback, from the searing pain of a hot stove to the warm praise of a job well done. It’s the cornerstone of growth, both in life and especially in the world of work.
As a senior engineer or tech lead, feedback is one of the most effective tools in your toolbox to help level up people around you.

But how do you give feedback effectively? For many, it’s shrouded in misconceptions and uncomfortable emotions.

It took me some time to understand some misconceptions I held about giving feedback. As an example, I once asked a report in a 1:1, What change can I make now to better support you? He responded, “I would love to receive more feedback from you.” I thought inwardly that I’m always on the lookout to see where I can give you feedback. You’re doing excellently well, and I can hardly find areas to suggest improvement.

One of the common misconceptions about giving feedback is that it should be given only when an opportunity for improvement is spotted. And I held on to that misconception for a while.

Ditch the myths

Let’s debunk some myths about giving feedback and explore the four simple steps to getting good at giving feedback.

Myth 1: Feedback is only for negative corrections. Not true! Feedback celebrates strengths, reinforces positive behaviors, and guides future actions; it is not always about fixing mistakes.

Myth 2: Instant feedback is always best. Emotions run high in the moment, leading to potentially hurtful and unproductive exchanges. Wait until you’re calm and collected.

Myth 3: Sandwiching is the way to go. Sugarcoating feedback dilutes its impact. Be direct, yet respectful.

Myth 4: You need a laundry list of examples. You don’t need a laundry list of examples to make your feedback effective. Focus on one or two key observations for clarity and impact.

Myth 5: Feedback is a one-way street. It’s a conversation! Encourage questions, active listening, and shared understanding.

Myth 6: Feedback flows top-down only. High-performing teams have feedback flowing everywhere: leader-to-team, team-to-leader, and peer-to-peer.

Believing that you have to give feedback instantly can make you give feedback when you don’t have your emotions in check. You want to avoid giving feedback when you’re angry because it’s far more likely the recipient will feel hurt or judged—and thus defensive.

Feedback shouldn’t be given only when you want a change from the receiver. Feedback is not only about correcting mistakes and driving change but also about recognizing positive contributions and strengths and reinforcing positive behaviors.

Feedback is a two-way dialogue where both the giver and the receiver share their perspectives, ask questions, listen actively, and seek to understand each other.

Anyone can be both a giver and a receiver of feedback. It should not be a top-down thing. In a high-performing organization, feedback flows from top down (from managers to reports), down up (from reports to managers), and laterally (from peers to peers).

Four simple steps to get better at giving feedback.

Follow these four steps to offer better feedback.

1. Ask

Before diving in, ensure the recipient is open to hearing your thoughts. When you ask, it gives the receiver control, ensures that you have the attention of the person, and sets expectations for what the conversation will be about.

A simple question like “Can I share some feedback with you?” can do wonders.

2. Describe the behavior or action

Avoid judging intentions. As a feedback giver, your job is not to judge the intentions of others. You’re not in a position to put a label on the action. Your role is to express the impact their actions had on you, the team, or the project.

When feedback is focused on the past, the other party can’t do anything about it; defensiveness on the side of the receiver is sometimes inevitable. Good feedback should focus less on the past and more on the future, making the receiver feel that the purpose of the conversation is for the future.

The book Effective Managers describes a simple way of describing actions that I found useful. It starts with the “when” question. For example:

• “When you took the initiative to…”

• “When you presented today…”

• “When you don’t communicate progress like you did yesterday…”

• “When you spoke about…”

• “When you stay an extra hour to find the root cause,”

• “When you respond politely after the customer…”

• “When you were presenting your ideas…”

3. Highlight the impact

Once you have described the action, proceed to explain how the behavior affected you, the team, or the project. For example, when you did that to XYZ, this is the impact (insert impact) it had on the team or the project.

Example:”

Behavior: When you took the initiative to…

Impact: This helped the team move forward efficiently and be able to deliver on time.

4. Encourage future action

If the feedback is negative, suggest what could be done differently in the future. If it’s positive, express appreciation and encourage continued behavior.

Building trust is key

Trust is like the air we breathe; when it’s present, nobody really notices; when it’s absent, everybody notices. – Warren Edward Buffett.

When people trust the giver, they’re more receptive and open to learning. Invest time in building genuine connections with your team. This fosters an environment where feedback is seen as a tool for growth, not criticism.

In conclusion, giving feedback is a skill that can be learned, not a magic trick. By practicing these steps and fostering a culture of trust, you can unlock the true power of feedback.

Have you ever been on a team that’s stuck? A team where everyone is moving in different directions? You spoke to three people, and there is a lack of consensus on what the team is trying to achieve. There are different perceptions of the team’s priorities and direction. Priorities shift from time to time, and you could feel a sense of instability or unpredictability in the team’s work. If you stick a little longer, you start to witness signs of disengagement, low morale, or a lack of enthusiasm.

If you notice several of the above, it’s an indication that the team lacks clarity of direction. Where there is no clarity, a different narrative ensues. If you’re a manager, one of the most important things you can do is drive clarity of purpose, direction, plan, and responsibilities.

I have come to believe that clarity in the context of software development is about giving people a practical North Star to guide their thinking and actions in the right direction, even when the conditions get cloudy. I have to admit, it took me a while to learn how important this is when you lead engineering teams.

A team is a function of its environment. If the environment a team is operating in lacks clarity on its purpose, structure, expectations, and boundaries, it becomes really difficult to achieve sustained high performance.

Clarity provides the vision, the direction, the rallying cry, the strategy, and the goals of your business and team. Clarity grounds organisations and teams into something bigger.

Why does driving clarity matter?

• A collective sense of clarity of purpose within an organization has a direct impact on its performance.

• Clarity drives focus. There’s a reason why startups can move swiftly and outpace larger organizations with significantly more resources. Besides the absence of bureaucracy, successful startups embody a strong sense of purpose and instill a radical focus in every team member.

• A lack of clarity impacts commitment. If it’s not clear, it’s hard to commit to.

• Clarity drives behaviors and actions that propel organizations forward, all aligned toward common goals.

There are three major areas a leader should consistently drive clarity on to get a team to their peak performance.

Clarity of direction and purpose

Charles Garfield, author of several books on peak performance, spent decades studying peak performers and found that intense commitment to what they do is one of the single most dramatic differences between peak performers and their less productive colleagues.

Having hired tens of software engineers and interviewed dozens of software engineers, I have found that engineers who are passionate about what they do and the problems they solve perform better and retain better than those who are not.

What really makes people develop intense commitment and passion for what they do? It is rooted in a deep understanding of “purpose.” An intense attachment to why they’re doing that thing.

The differentiation of performance in engineering teams is how the members feel about what they’re doing. The “how” stems from the purpose. To better put, the “why” behind that thing.

Effective leaders master how to communicate purpose and direction that inspire others and provoke actions.

McKinsey asked over 5000 executives to write down their peak experience as team members and the descriptions of the environment. The first of the three things was alignment in direction, which is a shared belief about what the organization is striving towards.

When there is clarity of purpose, a team has a navigation aid that helps set the direction of collective actions.

Clarity of purpose stems from everyone on the team understanding why they’re doing whatever they’re doing in the first place.

A northstar to aim for is a crucial part of motivating your team. When a team can’t understand why they’re doing what they’re doing, it becomes a losing battle. A small litmus test to understand if a team has clarity of purpose is:

• Does the team understand where the company is headed?

• Does the team understand why the company is headed there?

• Does the team understand how they’re contributing to that?

• Does the team understand how success is measured?

Clarity of purpose drives strategy and resources. To formulate a solid plan, you first need to be clear about what you want to achieve.

Clarity of plan

When you don’t have a plan, randomness ensues. A team member pushes a new activity, and another team member pushes an unrelated task on the side.

There are activities and tasks being done. But the effectiveness of the collective actions falls short. You wonder why the impact was so low. You don’t need to look far to notice an absence of clarity in the destination and the plan to get there.

When you have a plan, it limits your choices and helps people focus on the specific path to get to the intended destination.

Having a plan is the beginning. The end is to get to a place where everyone knows the plan, what they’re to do in the plan, and how that thing they’ll do contributes toward reaching their destination.

A team has clarity of plan when they’re able to understand the pieces and how those pieces will help them get to the destination.

To create plan clarity, start by connecting your overall mission (North Star) to actionable steps that get you to your destination—and creatively distill that into your team.

There are a few steps to take to help you understand and have clarity about your plans.

1. Together with your team, do focused planning by collectively mapping out the important pieces required to get to where you want to get to. People remember what they co-create better.

2. Establish a set of measurable key results that you aim to achieve by specific dates.

3. Map out the big projects that your team will take on to achieve those key results, and then the specific tasks to achieve them.

4. Actively and consistently overcommunicate the plan.

Clarity of roles and responsibilities

In a soccer game, players play with a clear understanding of their roles. The goalkeeper knows his or her role. The strikers and defenders all play with a clear understanding of what role they have to play in the game. A lack of role clarity impedes high performance.

When I picked up my first job as a software engineer, I was handed a contract of engagement with clauses on my roles and a description of what I would be doing.

Months into the job, I saw how different my day-to-day job was from the 8 bullet points listed in the paper as a job description. There were implicit role expectations and responsibilities that were not in the job descriptions, as I learned the hard way.

When individuals in a team have role clarity, everyone knows what they’re responsible for and accountable for. They know the role that they have to play in the plan. They know what is expected, which tasks in the plan they have to accomplish, and how their performance on the tasks will be evaluated.

If you’re leading engineering teams, pay attention to whether individuals on the team have a full understanding of their roles and responsibilities in your overall plan.

You should consistently ask yourself:

• Does everyone understand their roles and what they have to contribute to the team’s plan?

• Do they understand the details of the results to be achieved?

If you’re an IC in a team and you lack role clarity, be proactive by asking your manager detailed questions about your role expectations, and your performance will be measured. Put the expectations in writing and send them to your manager for review.

A few months ago, I spoke at the WeAreDevelopers World Congress on building high-performing software teams. The feedback from that talk has been overwhelming, and I’m thrilled it resonated with many people. In this post, I will unpack one of the key points I discussed: empowering your team and getting out of the way.

I like to think of empowerment as involving a team in decision-making, giving them a participatory role that leverages their skills and judgment and enhances their sense of individual worth and commitment to the team’s goals.

When you begin leading a team, you might have the illusion that your role in the team is what keeps the lights on. You may even think that your decisions and skills alone are the primary factors responsible for getting the team to where it is. Nothing could be further from the truth. No one can lead a team or an organisation to success without the collective excellence of many.

The most effective teams I have seen are autonomous. Every team member works, contributes, and is equally committed to the team’s goals. The higher you advance in an organisation, the more you have to rely on diverse skills and talents in the people you’re leading to make your team thrive.

It’s no coincidence that the primary responsibilities of leaders include coaching, mentoring, and empowering those they lead, giving them the opportunity to take on significant initiatives, make mistakes, and continue to develop and grow along the way.

One of the fundamental changes you can make as a manager is reinforcing the belief that leadership is not just a function of the title someone holds on the team but that everyone can be a leader.

Empowering individuals on the team to lead and influence is crucial. If individuals on your team don’t believe they have the power to lead as you do, hierarchy sets in, as stated by Michael Loop in his book, “The Art of Leadership.” This situation can lead people to feel they need to seek permission to drive change or wait for someone to dictate what to do.

For new managers, there are subtle ways you can disempower your team. I fell into one a few years ago when I began leading a team of engineers. I would jump into facilitating inter-team communications, chase down team dependencies, and shield my team from non-technical work.

As my team’s scope increased, it became increasingly challenging to handle all of these tasks alone.

I decided it was time for me to start involving my team by showing them how I handle these tasks. I thought I would demonstrate how it’s done so they could learn from it. Implicitly, I also expected them to contribute in the areas where I involved them.

I tried hard to lead by setting good examples. However, the more I tried to show them, the less I saw the team replicating what I was doing. In the meetings I involved them, they acted as observers rather than participants. I wanted them to be active contributors, but instead, they took a back seat and simply tagged along.

I realised that I had not only communicated my intent and expectations but had also not provided enough space for them to take on responsibilities and lead. Consequently, the more I took the lead, the more the team felt crowded out. They believed it was easier and faster for me to handle those tasks than for them. They felt like they had nothing to add, which increased their disempowerment.

The example from my experience illustrates subtle ways in which a team or reports can become disempowered. Instead, a leader’s goal should be to work themselves out of their job.

The goal of all leaders should be to work themselves out of a job – Jocko Willink.

Empowering engineers takes different forms in engineering organisations. Sometimes, it could mean providing autonomy, sharing the big picture, and giving your team all the details they need to make the right decisions.

At other times, it could mean refraining from providing your team with solutions and dictating what to build. Instead, give them the problems you want to solve and provide the space for them to find solutions.

In other instances, it could mean empowering engineers to solve problems rather than just completing tasks, and giving them ownership over it end-to-end.

Empower your team through autonomy, purpose and mastery

I often observe striking similarities between motivation and empowerment. When a team lacks the drive or motivation to take ownership of a piece of work for which they are responsible from start to finish, most of the time, it is a sign that the team has not been empowered.

Show me an intrinsically motivated team brimming with passion to tackle problems, and I will show you an empowered team.

Intrinsic motivation is behaviour driven by an internal desire to do something. It’s the motivation to engage that arises from within the individual rather than from the externals.

One of the ways to evoke intrinsic motivation is to provide autonomy. In the book “Drive,” intrinsic motivation is based on three factors: autonomy, mastery, and purpose. Tapping into the intrinsic motivation of individuals on your team leads to more sustainable performance and teamwork and consequently empowerment.

Give your team the right amount of autonomy

Autonomy is one’s ability to control one’s work or make decisions regarding one’s work. Empower your team by providing them with the freedom to choose how they approach their tasks, granting them a sense of ownership and accountability for the outcomes you want them to achieve.

You impede autonomy when you hand down feature requirements to your team without providing the context they need to comprehend the business problem and why they are doing what they’re doing.

A disempowered engineering team lacks the decision-making power regarding “what” they are building and “how” they are building it. The path to building an empowered team is to ensure that everyone has a say in what they are constructing and that they have the appropriate context that enables them to make effective micro-decisions.

Give them space to struggle through stretch projects

We’ve all experienced the joy of learning something new: the excitement that follows learning a new language or the satisfaction of finally mastering how to solve a persistent problem. That feeling of continuous improvement on something that felt nearly impossible a month ago is a crucial source of motivation that can keep you going.

Empower individuals on your team by providing them with opportunities to work on tasks that stretch their abilities—tasks that take them slightly out of their comfort zone but offer growth prospects.

Is there a team member who hasn’t yet had the chance to develop their skills in handling cross-functional projects? Create an opportunity that exposes that engineer to cross-functional projects where they can acquire new skills and collaborate with a diverse group of colleagues.

When you don’t allow your team to take on challenging tasks or attempt to shield them from challenges, you deny them the chance to master those challenges. By doing so, you’re not only disempowering those individuals, and you might also be depriving them of coaching opportunities.

Give them a sense of purpose

Purpose signifies understanding how one’s work contributes to something meaningful and impactful. When a team can connect their efforts to a larger purpose or mission, their motivation and engagement increase. To empower your team to make informed judgment calls, provide them with a sense of purpose, which offers the context they need to do so effectively.

An engineering team with clarity about why they are doing what they’re doing will make better implementation decisions that align with the “why.” When an engineering team’s purpose is clear, they understand what they are working on, why they are working on it, and how their work will have an impact. By making sure your team is clear on their purpose, you’re empowering them.

You’re overwhelmed. You have an endless list of different tasks. Your schedule is stuffed with meetings and you need more time for focused work. Each meeting you attend brings more to-do items to your unending list. You’re busy throughout the day but can hardly point to something concrete you have done yourself.

If this describes you, you’re likely in a lead role or managing teams. This post is for you.

If you recently transitioned to a lead or manager role, you may start to feel like you’re not doing your job. That feeling of achievement you once derived from shipping tangible stuff disappears. At this same junction, some leaders slip back to doing IC work, like churning out features while ignoring the more significant part of their role –leading.

There are two things you need to understand as a new lead. You need a new approach to frame what you do. And you need a new way to prioritise them effectively.

As a software lead or manager:

• You will mostly be working through people (your team).

• You will do small little tasks but many instead of a whole big chunk of work.

• You will frequently context-switch throughout the day.

• The impact of some of the work you will do will be long-term, which will make it hard to get that instant feeling of progress. The effect of that extra email you sent to finally close a strong talent will not materialise instantly.

All of the above run counter-parallel to what you did as an individual contributor. Hence you will need to change how you view your work. Rather than what you contribute single-handedly, you need to reframe the way you see the organisation and your contribution to its success.

Your role is no longer about your individual contribution and your individual impact.

Your role is about what the team does, about how the team contributes to the company’s success. Those little boring tasks you do to make your team better and improve their efficiency now matter more than the features you can code alone – that is your job.

But you’re still drowning in tasks. So, how do you stay on top of your not-ending to-do list and still get things done?

The answer is simple: know how to pick your battles. You need to understand tasks you should do and tasks that others should do. You need to know what should be done in the moment and what needs to wait a little bit longer.

One of the mistakes I made in the past was making every task that came my way a task I must do. While a leader may be responsible, it does not mean that the leader should do everything. I was clearly missing “What I should do vs. what others should do.” I was equating being responsible to being the one doing it.

Being responsible for a task does not mean you should do it. There will be hundreds of things demanding your attention because the scope of your work has increased. It’s important to constantly differentiate what you should actually be doing from what others should be doing.

Once you find out what you should do vs. what others should do, you can delegate what others should do and focus on what you should do.

How to know what you should do vs. what others should do

A delegation matrix is an effective tool you can use to decide which tasks to delegate and which to retain. When you use this tool effectively, you’ll be able to free up some time to chase bigger things. A delegation matrix is a 2×2 table measuring the competence of reports on the y-axis and the competence of a manager on the x-axis.

Let’s look at a case where a delegation matrix can be applied. You have just received a request to review a new design document from another product team to ensure approaches are aligned. There are a few options you have.

• Spring into action to review and unblock the team while other things suffer.

• Add it to your overflowing to-do list and block this team for a couple of days or a week until some time is freed up for the review.

Using a delegation matrix, you may assess that a report on your team possesses the skills to review the design doc. Clearly, both you and your report can do the review. This task goes into quadrant 3 in the image above. Everything in quadrants 1 and 3 should be delegated to your reports.

There will be times when you have a task neither you or your report can do. Such tasks should either be delegated to your manager or solicit assistance from your manager to do it.

Everything that falls into Quadrant 4 should be done by you.

Tasks in quadrants 1 and 3 are:

1. Tasks that can be handled adequately by reports

2. Tasks for which team members have all the information for decision-making.

3. Tasks that don’t require skills unique to you or your position.

4. Tasks for which an individual other than you can have direct control over the task.

5. Tasks and/or projects that will contribute to the growth and development of the individual.

Tasks that fall in Quadrant 4

• The delegation process itself: Any work to be delegated should be delegated and explained by you.

• Performance evaluations and disciplinary actions: These are managerial responsibilities.

• Coaching, and moral problems.

• Planning and forecasting: Some of the detailed work can be done by engineers, such as breaking tasks down and estimations. However, you alone are in a position to decide team goals how they fit in with the overall company’s goals.

• Confidential tasks.

Separate urgent, important tasks from non-urgent tasks

Have you categorised your tasks into what you should do vs. what others should do and delegated what others should do, and yet you still feel like you’re drowning in assignments? Then, the next step is to separate urgent and important tasks from non-urgent and important tasks.

Do urgent and important tasks now.

Urgent and important tasks have clear deadlines and consequences if immediate action is not taken. Urgent tasks include responding to incidents, onboarding a new engineer, resolving conflicts, and unblocking a report.

Schedule important tasks that are not urgent.

Important but not urgent tasks are tasks without a specific deadline but they bring you closer to your goals. These are tasks that are easy to procrastinate on. Examples are defining your team’s strategy and vision. Schedule these tasks for the future. Put a block in your calendar in the future to do them.

Delegate urgent but unimportant tasks.

You may still find a couple of urgent but unimportant tasks in your todo list. There are studies that show that we’re naturally biased towards picking urgent tasks even if they’re less important. Tasks that are urgent but are not important are best delegated.

To conclude, if you’re going to take one thing from this post it should be being responsible doesn’t mean doing it yourself. Understanding what you should do vs. what others should do is key to staying on top of your schedule.

Additional resources

Five valuable lessons for a new tech lead
Feeling productive

Will Larson provided three ways to do this that I have found useful.

• First, you need to understand what’s happening on indirectly managed teams below you.

• Second, add things necessary for execution.

• Third, remove things getting in the way of execution.

The biggest puzzle to solve is understanding what is happening in the indirectly managed teams.

As one goes up in an organisation, the less the details you know about day-to-day jobs of people below and the unique challenges they face. An engineering director will have more knowledge about how each piece of work connects with teams under her purview but less knowledge about the codebase or the nitty-gritty of the environment frontline engineers work in and the impact it has on execution.

To draw an analogy, it’s like zooming in and out on maps. In a zoomed out mode, you see the big picture. When you’re in a zoomed in mode, you get a close look at select details. Frontline engineers have a zoomed in experience. And the higher you go in an organisation, the more zoomed out your view becomes. Folks at the top of an organisation tend to see and experience things in a broader sense than those who are layers away under them who may see things narrower but in-depth.

There are a few areas engineering leaders can look at to have visibility required to foster execution and remove roadblocks from frontline teams. That starts from finding a way to get quantitative and qualitative data that will help you:

• Understand how your teams are building what they’re building on time and within the budget you have.

• Understand what your teams have built are running at the desired level of performance and reliability and that it will continue to do so within the foreseeable future.

• Understand if the teams building that stuff are engaged and happy to continue building what they’re building.

• Understand if the users you’re building for are getting value and deriving satisfaction from what you’re building.

Understanding how your teams are building what they’re building.

Understanding how your teams build requires you to understand the process your teams take to build that thing they’re building. Let’s call this “process“. Process is the way your teams habitually do things. It is how your teams get work done on time and within budget. If you’re a Pizza company, process is how you turn dough into edible Pizzas delivered to consumers.

For a software company, your process is ideally your approach to building what you’re building.

If your organisation is like most organisations, your time and budget are finite. You always have to get value delivered to users within a certain budget and within a specific time. Due to this, you have to find an optimum way to optimise value delivery to users within the budget that you have.

To gain an understanding of how your team builds stuff and the possible roadblocks, take a look at where engineers spend their time, things that affect execution, how they collaborate, the tooling you have that enables them, the codebase, systems and architectures they work in.

• Does architecture/codebase allow teams to move as fast as possible?

• Do your teams have the right tooling?

• Do they have autonomy to spin up new services, build, scale and deploy them without hand-offs? How long does it take them to do this?

• Are your teams empowered to deliver values independently, own their backlog, ideate and figure out the most important thing to build?

• How do different functions within the teams work together? Do they have a shared understanding of what’s being built?

• Does everyone know where they fit in and what roles they need to play in what is being built.

• Do they have necessary tools

Answers to questions like these will point you in the right direction in figuring out what may be standing in the way of execution.

3 Ways to have the visibility required

You want to gain a better understanding of unique challenges your indirectly managed teams face? Include the following in your data gathering process.

1. Run developers experience surveys quarterly

Running surveys is one way to understand the experience your engineers have. Engineers’ development experience can reveal a ton about how they’re building what they’re building as well as what is standing in the way of getting things done.

I have seen engineering surveys centred on tools and technology only. Surveys like that often fall short. Sometimes the biggest source of roadblocks could be unrelated to tech or tools. A process or a social contract that has nothing to do with tech can be equally a source of friction for engineers.

I like the way C J Silverio puts it in this post.

If people are regularly doing any end-run around a process to get work done (say, regularly asking for rubber-stamp PRs so they can be unblocked), you have a process that’s not earning back its energy cost. Fix it.

A simple survey with questions like the ones below could reveal a ton about your engineering processes:

• How easy is it to find documentation and access documentation?

• What manual work can be automated that the team does?

• What process do you find valuable?

• What process would you improve?

• How satisfied are you with the tooling used daily e.g linters, IDE?

• What would you add or improve in our tooling?

If you’re looking for where to start, developer experience survey questions by Laura Tacho is a good one.

2. Hold 1-on-1s and skip-level 1-on-1s

When 1-on-1 is used effectively, it can be a platform for gathering information that allows you to know what’s going on in the team and with each engineer. 1-on-1 is a great time to offer your perspectives on what is going well and not going well. But it is also a good time to gather qualitative data, disproof or proof what you already know that is not going well.

If you’re leading from 2 or many layers away, hold skip-level 1-on-1s with frontline engineers.

• It will help you to gather information about how your managers are really doing beyond what they tell you.

• It will help you get a pulse on what’s happening on the front lines.

• It will help you learn where there is dysfunction, insufficient communication, or confusion within parts of your organisation.

3. Track DORA metrics

DORA metrics came to be after six years worth of surveys conducted by the DORA team and identified four metrics that elite-performing software teams use to measure their performance.

• Deployment Frequency—How often do you release code to production? How often do you deliver value to users?

• Lead Time for Changes— The amount of time it takes a commit to get into production. How long does it take to deliver value to users?

• Change Failure Rate— What is the percentage of this value that is defective?

• Time to Restore Service—How fast can we get back up when we fail? How long does it take an organisation to recover from a failure in production?

Measuring productivity metrics is a hard topic in software engineering because companies have attempted to measure developers’ productivity but ended up measuring the wrong thing. A few companies learned from this and abstained totally from ever measuring anything.
One thing is certain: developers’ productivity can not be reduced to a single metric or dimension. There are multiple dimensions to it.

Personally, I find DORA metrics effective because they can show you trends that allow you to ask crucial questions. When a deployment frequency trends downward, it puts a lot of questions you can ask your team.

One of my experiences with using DORA started at TIER. We were tracking how well we were doing WoW and MoM. Once every two weeks, I would sit down with Engineering Leads to discuss the metrics.

When we saw deployment frequency trending downwards or upwards, I would ask questions to understand what was going on. When we got better at putting code to production frequently, I wanted to know what changes we made.

In one of the review meetings with engineering leads, deployment frequency was trending downward. As I sat down with them to understand the root cause and how I could better support them, we saw that there were issues with our deployment infrastructure. The infrastructure team made changes to the base infrastructure that this team was yet to update to.

Most of the engineers in this team were new and had missed out on comms that had gone out earlier on why the team should upgrade. On finding the root cause, we were able to take a more actionable step, work together with the infrastructure team and get supports needed.

Additional resources

If you’re looking for more inputs? The following links are great resources.

• Reduce friction

• Skip Level Meetings: Everything You Need To Know About Skip Level 1 On 1s

• How engineering leaders can lead with visibility

How do I build a high-performing team?

This is one of the common questions managers ask themselves. No one wants a low-performing team. We all want to build effective and high-performing teams – a team that is highly interdependent, bound with a common goal, plans work, makes decisions, solves problems and delivers superior results.

But you can’t just pull a bunch of people together in a room and expect outstanding performance.

Building a high-performing team requires deliberate efforts. You have to create an environment that allows and breed outstanding performance.

Over the years, I have experienced underperforming and high-performing teams. I saw the kind of environment where high-performing teams thrive, things high-performing teams do differently and the culture that enables them.

But how do you create a high-performing team and what levers are there to pull?

There’s more than one way to skin a cat. But certainly, there are a few levers you can pull that will put your team on the way to superior performance.

Hire strong talents. Strong talents make a strong team.

You want to build a strong-performing team? Hire strong people. Why hire a strong senior when you can have 2 medior for the exact cost? Well, superior performance doesn’t come cheap.

Strong engineers are high achievers, trail blazers and catalysts for improvements. It’s worth the cost.

If you want to build a strong software team, hire strong people in product, data, design and engineering. Hire people who constantly challenge themselves and work to better or improve themselves.

Hiring a group of people and expecting outstanding performance doesn’t work. There have to be some thoughts behind the group composition and why they exist.

Usually, we often fail in two ways. Suppose we don’t fail at composing the right team by carefully hiring folks that bring complementary skills, we fail at defining why the team exists and ensuring every team member understands why they exist and why they’re doing what they’re doing.

Empower them

Have you hired a strong team? Coach, empower and get out of the way as necessary. A high-performing team is an empowered team that makes decisions within a defined boundary. They are able to make effective micro-decisions in their day-to-day work.

An engineering team that relies on a single decision-maker, albeit a product manager or an engineering manager is not empowered. Progress stalls when a team depends on a single person to make decisions.

Coach your team to the point that everyone knows what the expected outcome is. Coach them to a point where they know what they should be doing and what’s not.

An environment where one person is solely responsible for figuring out what to build, which is then handed over to the team creates a situation where a team fails to see why they’re doing what they’re doing. It becomes a game of “they ask us, we do it”.

Create synergy

Synergy is when two or more people work together to produce something of value. The more people you have, the more difficult it’s to create synergy – this is one of the reasons why the concept of two pizzas garnered popularity.

Synergy, at its core, is about team working, connecting and collaborating effectively together.

Communication and collaboration are key to maintaining high synergy. Folks having to chase information they need to do their work drains motivation and depletes trust between teams.

When communication breaks down, synergy suffers. And it could feel like every other person is out there to prevent you from doing your job. Often, it’s just a misalignment between functions, as rightly said in this post:

Every team in an org has its own pain points as well as its objective. While it seems that one department refuses to fix the problem of the other, it is also true that most people in the company work with good intentions. Hence, if everyone is trying to do a good job and they also complain about each other, the problem therefore lies in communication (or more specifically, communication breakdown).

If you want to build a high-performing team, pay attention to how different functions collaborate and the willingness of people to share information that benefits the wider group.

One of the best ways I have seen this play out is having a “default open” culture where ideas, information and feedback flow freely.

Create a culture of learning and improvements

Invented by psychologist Bruce Tuckman, a team goes through four stages of psychological development known as forming, storming, norming and performing. For a team to get to the performing stage, there has to be a lot of mistakes and learnings along the way.

A high-performing team is not formed in a day. High-performing teams go through periods when they’re low performing. It’s through mistakes and learning from those mistakes that make the team become better and better as days go by.

A team that reflects on how to become more effective and then tunes and adjusts its way of working gets better as time passes.

As Jocko and Leif stated in their book titled the dichotomy of leadership:

No team can deliver flawless performance. No one can achieve perfection. What makes the best team great is that when they make mistakes, they acknowledge them, take ownership and make corrections to upgrade their performance. With each iteration, they enhance its performance.

High-performing teams make incremental changes in processes (way of working) to improve quality and efficiency. They have a mindset that whatever is good today might not be good tomorrow. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

To build a culture of learning and improvement, you have to create an environment where people feel it’s okay to fail and not be afraid to voice out their shortcomings.

This is where psychological safety plays a crucial role.

Build psychology safety

Psychology safety is the belief that you have the freedom to speak your mind without being punished for that. Such belief is rooted in deep trust between peers and leadership. In a software team with high psychological safety, members are more open to taking risks, trying new ways and not afraid to fail. It’s this tolerance for mistakes and risks that allows the team to be more innovative, develop muscle to try new things, fail quickly and learn.

As a leader, it’s your responsibility to build an environment that doesn’t make people feel they can’t freely share their opinions but also an environment that lets them feel their opinions are welcome and you love to hear them.

In summary, building high-performing teams will take time to happen. It’s through deliberate effort but it’s worth it. To build high-performing teams, hire strong talents, empower them, create synergy and build a culture of learning and improvement.

You were not in control. You had no visibility: maybe there was a car in front of you, maybe not. –Alain Prost

As tech leaders, external factors, culture, and engineers’ personal life can affect delivery flow or your team’s ability to deliver on business outcomes and goals. You will need to equip yourself with data that allows you to get faster feedback to stay on top of these factors and maintain predictable and consistent delivery of value to users.

Whether you’re managing from 1 or 2 layers away, there are four main areas you need to get both quantitative and qualitative data from to have the visibility you need to lead your team effectively. I categorize these four areas you need to get data from as process, operation, people and product.

You need to get both quantitative and qualitative data that will help you:

• Understand how your team is building what they’re building on time and within the budget you have – Process

• Understand if what has been built is running at the desired level of performance and reliability and that it will continue to do so within the foreseeable future – Operation.

• Understand if the folks building that stuff are engaged and happy to continue building it – People.

• Understand if the folks (users) you’re building for are getting value and deriving satisfaction from what you’re building – Product

Process – Understanding how your team is building what they’re building

Process is the way you habitually do things. It is how your team gets work done on time and within budget. If you’re a Pizza company, process is how you turn dough into edible Pizzas delivered to consumers. As software company, your process is ideally your approach to building what you’re building.

If your organization is like most organizations, your time and budget is finite. I’m yet to see an organization that has enough of both. We always have to get value delivered within a certain budget and within a specific time – this calls for a balancing act.

One question most managers have is how their teams are building what they’re building within that time and the budget they have. To answer this question, you need visibility into where engineers spend their time and things that affect execution, like how collaboration works across different functions, the tooling you have that enables your team, the codebase, systems and architectures your engineers work in.

• Does architecture/codebase allow teams to move as fast as possible?

• Does your team have the right tooling?

• Do they have autonomy to start new services, build, scale and deploy them independently?

• Are your teams empowered to deliver values independently, own their backlog, ideate and figure out the most important thing to build?

• How do different functions within the team work together? Do they have a shared understanding of what’s being built? Does everyone know where they fit in and what roles they need to play in what is being built.

Answers to questions like these will point you in the right direction in figuring this out.

3 Ways to have the visibility required

Here are ways engineering leaders can have the visibility required.

Developers Experience/Productivity Survey

One of the ways to understand how your team is building what they’re building is through regular engineering surveys to understand engineers’ experience. What engineers experience while developing can reveal a ton about how they’re building what they’re building and where bottlenecks of pain points exist.

If your organization has a way of measuring employee engagement, you should still run surveys centered on developer experience. Organizational-wide engagement surveys are sometimes too general and may not effectively capture engineering experience.

A simple survey with questions like this could reveal a ton about engineering processes:

• How easy is it to find documentation and access documentation?

• What manual work can be automated that the team does?

• What process do you find valuable?

• What process would you improve?

• How satisfied are you with the tooling used daily e.g linters, IDE?

• What would you add or improve in our tooling?

When we ran some of these surveys at TIER, we realized that documentation and tooling were issues that needed to be addressed. We started investing in making documentation accessible and consolidating them. Along the way, we had to solve a challenge of making documentation close to the code base where engineers work.

If you’re looking for where to start, developer experience survey questions by Laura Tacho is a good one.

1-on-1s

When 1-on-1 is used effectively, it can be a platform for information that allows you to know what’s going on in the team and with each engineer individually.

When I first started having 1-on-1s, one thing I wanted to avoid was making it a meeting to get project updates and progress reports. In my first meeting with a report, I would set expectations of what the meeting was and was not about. One thing I always made clear was to refrain from talking about projects or getting project updates in the meeting.

With time, I realized that talking about projects was unavoidable to support my team effectively. As I listened to concerns from my teams, they always pointed back to the project they were working on.

If a report has an issue with another engineer on a team, it’s due to a project or task they’re working on. If a report is not getting what she wants from another team, it’s due to a task or project she’s working on.

So I realized it’s okay to talk about projects in 1-on-1s if there is an opportunity to unblock or coach a report.

You should use 1-on-1s to pick signals of things going well or not going well in your team. This is why 1-on-1s can be effective in helping you understand how your team is building what they’re building.

Here is an example of a conversation with a report.

*Manager: What do you think about how project XYZ is going?

Report: Well, it’s going fine except that we underestimated the project. There was ABC we didn’t think about or account for when we committed to the project. To do XYZ, we need to do ABC which will take 3 more weeks. In fact we don’t even know if it can be done yet. We’re currently undertaking tech discovery on how to do ABC.*

From the above conversation, this team is probably not investing in proper product and technical discovery before taking on projects if it becomes a pattern.

DORA Metrics

Measuring productivity metrics is a hard topic in software engineering. Historically, companies have attempted to measure developers’ productivity but ended up measuring the wrong thing. One thing is certain: developers’ productivity can not be reduced to a single metric or dimension. There are multiple dimensions to it.

One approach that has worked for me in the past is by measuring DORA metrics.

DORA metrics came to be after six years worth of surveys conducted by the DORA team and identified four metrics that elite-performing software teams use to measure their performance.

• Deployment Frequency—How often do you release code to production? How often do you deliver values to users?

• Lead Time for Changes— The amount of time it takes a commit to get into production. How long does it take to deliver value to users?

• Change Failure Rate— What is the percentage of this value that is defective?

• Time to Restore Service—How fast can we get back up when we fail? How long does it take an organization to recover from a failure in production?

I have found DORA metrics effective because they can show you trends that allow you to ask crucial questions. When a deployment frequency trends downward, it puts a lot of questions you can ask your team.

My teams at TIER adopted DORA metrics. We were tracking how well we were doing WoW and MoM. Once every two weeks, I would sit down with Engineering Leads to discuss the metrics.

When we saw deployment frequency trending downwards or upwards, I would ask questions to understand what was going on. When we got better at putting code to production frequently, I also wanted to know what changes we made.

In one of the review meetings with engineering leads, deployment frequency was trending downward. As I sat down with the them to understand the root cause and how I could better support them, we saw that there were issues with our deployment infrastructure. The infrastructure team made changes to the base infrastructure that this team was yet to update to.

Most of the engineers in this team were new and had missed out on comms that had gone out earlier on why the team should upgrade. On finding the root cause, we were able to take a more actionable step, work together with the infrastructure team and get the support needed.

Engineering onboarding entails communicating your company’s expectations, such as technical, process, product, cultural, and professional, to new hires in a way that makes the knowledge useful and practical.

The time it takes for a new engineer to be a productive member of a team is a function of how effective your onboarding process is.

I do realize there is a lot that goes into onboarding a new engineer. Usually, there are roles played by Procurement, HR team, and more. Usually, an HR onboarding will contain training programs designed to help new hires get acquainted with a company and its culture. The HR onboarding is usually generalized and may not include engineering onboarding. As a result, the focus of this post will be onboarding from the standpoint of a hiring manager.

Start from the hiring manager’s interview.

Engineering onboarding starts from the hiring manager’s interview. This is the first opportunity to help a candidate see her potential impact on the job. Don’t only assess the candidate’s fit during the hiring manager interview but also help the candidate connect to your team’s mission and purpose.

You’re doing onboarding work if you’re able to help a candidate understand what the role is about and how she can possibly fit in. The hiring manager’s interview can serve as an onboarding foundation that will be built on later if the candidate makes it to an offer stage.

Use the candidate’s notice period to your advantage

Candidates drop out at every stage of the hiring funnel. For a very strong talent, dropping out after signing an offer is a possibility. Some candidates have long notice periods. It’s important to keep the communication going.

You should leverage the notice period to continue to build on the candidate’s excitement about your company, your team, and how you work. My strategies include reaching out to them a week after an offer is signed to share our excitement in a mail.

As the notice period draws to a close, I reach out to see if they need any assistance and make use of the opportunity to send our onboarding guide to the candidate to prepare her ahead.

Have a comprehensive written down onboarding guide

One of the few elements of engineering onboarding that I found to be important when onboarding engineers is having a written down onboarding guide. PowerPoint presentations should not replace a written down employee onboarding guide that new hires can refer to from time to time. Nothing scales better than words written down because it’s always available when people aren’t.

Your onboarding guide should help candidates answer the following questions:

• What’s the team’s culture like?

• What is the company culture like?

• How does the team function or work?

• What are the tools used?

• How should access to tools be requested?

• Who are the people to meet?

• What are the company, product, and tech strategies?

• What are the engineering principles?

• Where to find documentation?

• What is the current team roadmap?

• Faqs and more.

Don’t leave engineers to figure out the tech part.

I had the opportunity to work with six different companies on three continents in the last ten years. Each of these companies has its own unique way of shipping software. They don’t all use the same techniques or approaches to software development.

Regardless of how experienced a new hire is, they should be onboarded into how you build software.

The “She is an engineer, and she will figure it out” will not work.

She needs to get familiar with your workflows and processes, and practices. She needs to know how things get shipped, what metrics are important, e.t.c.

On average, it’s estimated that it takes 3-5 months for a new engineer to be productive. A well-designed tech onboarding will ramp up new hires to a point where they become a productive member of the team.

The times it takes a new hire to be productive is a measure of how well your onboarding is.

An effective onboarding for engineers should not ignore tech. So, ensure your tech onboarding helps new hires understand:

• How software gets shipped

• What are the technical expectations from the role

• What are the non-technical expectations

• What is the release process like

• How to set up development environments

• What tooling is used

• Your code review process

• Your reliability metrics

• Incident management

• Existing and adopted ADR decisions

• What is the current state of your tech landscape, and what is the expected state?

Set onboarding and role expectations.

Without clear expectations, it’s impossible to drive accountability. Setting clear onboarding expectations helps new hires take responsibility for their own onboarding. It gives them a north star of where they’re headed.

When I onboard engineers, I break down onboarding expectations into 6 categories. Each category has a list of expectations that are required during onboarding.

• What is expected during the first week of onboarding

• What is expected from 2- 3 weeks of joining the team

• What is expected by the end of month 1

• What is expected by the end of month 3

• What is expected by month 6 and beyond

Conduct regular check-ins

When I joined Architrave, I relocated from Nigeria to Germany. Fortunately, I had an amazing people team and a fantastic manager who conducted a series of check-ins with me during my first few months.

The check-ins helped me receive early feedback and helped me navigate the change. The check-ins left a positive impression on me even after leaving the company. My onboarding period at Architrave made me realize the importance of frequent check-ins with new hires.

New hires may find it difficult to ask for support and may struggle silently. This makes it harder for managers to catch up and intervene quickly to get them back on track. By having regular check-ins with new hires, you’re creating an opportunity to ask crucial questions about how their onboarding is going. And how you could provide the support they need.

I follow structured check-ins with new hires as follows:

1. Check-in after 2 weeks of joining

2. Check-in after 30 days

3. Check-in after 60 days

4. Check-in after 90 days

Before going for onboarding check-ins with new hires, it’s important to have some feedback ready.

• Is the engineer doing the work you hired her to do?

• What is the new engineer doing well, and what can she change to make working with them easier?

• How confident are you that the engineer is going to be a top performer six months from now?

The importance of feedback cannot be overestimated and it is well captured in this post.

New hires will need frequent feedback to feel safe in their onboarding and understand if they are doing things right. This is both important for their self-confidence, lowers their stress levels, and makes sure they are indeed in the right direction.

Assign an onboarding buddy

Assigning an onboarding buddy to a new hire is a good way to provide an informal support system for the new hire.

It’s impossible to cover all new hires’ questions in the onboarding handbook or presentations.

When you assign an onboarding buddy, it allows the new hires to build a fast rapid connection with a co-engineer. And help the new hire get fast support for things not covered in the employee handbook.

Finally, obtain feedback and continuously use the feedback to improve your onboarding process.

But the moment you take a leap into a leadership role, moving beyond leading oneself to lead others, you start to realize that succeeding in this new venture takes more than being technically sound. Suddenly, there are non-technical skills that seem to matter.

Unfortunately, most people don’t get a manual handed over to them to help with the transition. In this post, we will look at five valuable lessons for a new tech lead. Whether you’re an individual contributor who is thinking (or in the process) of stepping into a tech leadership role, or a manager who is overseeing such transitions, this article is for you.

1. Your output is measured differently than you’re used to

Moving to a tech leadership role implies that you’ve done excellently well as an IC. You’ve earned your way to be a rockstar individual contributor, churning out features in the blink of an eye. But in this new role, what made you a good IC is now slightly different from what will make you a good tech lead. Realizing and adjusting to this new reality may feel strange at first. After a few weeks, or months especially, you may start feeling you’re not as productive as you were. This is because as an IC, you derived a sense of accomplishment from purely technical work such as writing code, refactoring an existing codebase, or optimizing an existing feature.

But as you step into your new role, your responsibilities diversify. You now have a responsibility to carve a technical path for your team. Your responsibilities now include having an overview of the entire system your team is building, establishing technical visions, and driving alignment.

Your role has changed, and how your output is (or will be) measured has changed too. It no longer depends on how much you can single-handedly deliver. Instead, it depends partly on what you do and partly on what you enable others (your team) to do. The quicker you’re able to tune your mind to this new reality, the greater your impact will be.

2. Aligning priorities is now part of your job

Aligning priorities means helping your team to focus on what’s most important and impactful. It means making sure your team’s efforts are all in sync with one another, supporting and reinforcing each other on the right thing.

As an IC, you were part of a team; you adopted the team’s goals and priorities. But as you become a tech lead, you start to take an active role in helping to shape your team’s priorities. You start to develop a wider understanding of your team’s goal and use that to decide what’s most urgent and important in terms of collaborating with other stakeholders.

There are always hundreds of tasks to be done, hundreds of features to build, experiments to run, and ideas to test. Being effective in this new journey requires a focus on value and impact and knowing how to choose which results to deliver.

You have to train yourself to look at every project or idea through this lens: how can we use 20% of our efforts to unlock 80% of our business goals, and how can I convince others and get them on board?

There won’t be enough time to build every idea. And there won’t be enough resources to create the ideal, ultra-polished product. The three variables you must now constantly play with are time, resources, and scope:

Figure 1: The triple constraints

Scope measures the work to be done. Resources include budget, team members, and everything at your disposal to execute the work to be done. Time refers to deadlines and how long you have to deliver on the work. You need to learn how to negotiate project scopes and requirements based on available time and resources.

3. How you communicate matters more

One of the most important skills to master is communicating effectively. Not all tech leads are good communicators, but all good tech leads communicate well.

Building software is a complicated process with numerous options and trade-offs that can easily be overlooked when communication is poor. As you step into a tech lead role, the way you communicate begins to have greater consequences and impact. Rather than simply making information available for people, you need to think carefully about what and how you communicate to people in order to achieve the desired outcome.

Sometimes, the curse of knowledge (where you develop a cognitive bias that makes it difficult to remember what it was like to be a novice) might kick in as you have access to broader information. You might think your team knows everything that you do or has the same context that you have on a project, when they don’t.

The effect of miscommunication could lead your team to veer away from building the right thing or making technical decisions that don’t align with the project’s direction, which can be expensive to fix later on.

You have to communicate and overcommunicate. Be creative about it at times; this could be as simple as creating a diagram or mock-up screen to pass the information across. When expressing complex ideas, it’s important to take a couple of minutes to think about how you can make it easy for your audience to understand. The worst mistake you can make is to assume people already know. Sometimes people forget things or are lost in other thoughts. So don’t be afraid to repeat yourself to get information to stick.

4. You should avoid premature fixes

As you step into your new role, you have to learn to wait for things to unfold in order to have a clear picture of what is going on before implementing a fix.

This can be very hard to grasp because it goes against our very instincts as engineers. As makers, we know what it means when we didn’t build to counter various edge cases that could make our application behave in unexpected ways. We know users of software can’t be trusted, so we have to think and anticipate problems in advance and put constraints to guard against them.

As you step from coding software for people to use, to leading people that code software for people to use, rushing to implement constraints to prevent or guard against problems you anticipate can sometimes have unintended consequences.

You can think of constraints as processes, practices, or guidelines enforced and intended to prevent certain problems or lead to certain outcomes.

While a lack of processes can affect your team’s performance and ultimately lead to the development of low-quality products, on other hand, too many processes created in anticipation of problems that are not yet clear can slow and hamper your team’s productivity.

‘The Master allows things to happen. She shapes events as they come. She steps out of the way and lets the Tao speak for itself.’ – The Daodejing, Lao Tzu

Instead of rushing in to create tons of processes right away, wait for problems to surface due to a lack of a process that fixes that problem. You’ll not only be able to craft a more appropriate solution, but you will also easily get your team’s buy-in, and collectively and collaboratively you will be able to define a better approach to fixing the problem.

Not only will your team see the need for it, but they will also understand why it is important. Since they’re involved in providing a solution, they will feel a sense of accountability.

5. You need to learn to manage projects

One important skill that feels very different from your previous role is dipping your feet into a bit of product management.

The Project Management Institute (PMI) defines project management as the ’use of specific knowledge, skills, tools, and techniques to deliver something of value to people.’ Because you and your team are responsible for delivering something of value (software) to users and stakeholders, you have an obligation towards your users as well as your stakeholders, which means you need to make sure work gets done.

There will be non-engineering bottlenecks that raise their heads to prevent you from delivering on time or block your team’s progress. These could include dependencies on people, corporate processes, or mismanaged expectations. Now it’s on you to identify and tackle these problems.

Now you need to think about planning and breaking tasks down. You need to inform stakeholders outside your team of what is going on in engineering and, likewise, inform your team of what is going on in other teams.

Management is now a part of your job. Rather than simply handing off anything that has to do with project management to project managers, you now need to work with them and compliment their efforts.

Reflections

The transition from IC to tech lead is never easy, and every journey is unique. There’s no manual and you’ll need to figure things out as you go. Take time to read, experiment, learn, and reflect. You will make mistakes, but what matters most is learning from them. That’s how you’ll grow as a leader.

Originally published on www.leaddev.com.

In this post, I’ll share some of these blog posts. I believe there are ideas and new learnings in them that will make you a better engineer or engineering manager.

Making the Most of These Posts

Frequently, I find myself in a situation where I could use some insights from what I’ve learned, and I completely forgot them. I’ve developed a process that helped me remember what I read and make them stick.

As you read through each post, I recommend doing the following to get the best out of each post.

1. Know what you’re reading. Read what the introduction is about and note down what you’ll learn at the end. This will prepare your mind for the insights to come.

2. Identify and jot down key ideas and information in the post. Write down learnings, insights, and ideas. Taking notes of what you learn prompts you to think about your learnings and also improves your memory.

3. Make your notes searchable.

4. Re-read your notes often.

Effective Communication Is Not About What You Say – Melanie Ensign

Communication is very crucial to leading effectively. But there is a clear difference between merely passing a message across and communicating effectively. The outcome of any communication is what determines how effective it is. Simply pushing out a message or publishing information is not enough to measure its impact. Effective communication is focused on what people need to hear in order to reach the outcome you desire and less on what you say.

You Are Your Own Best Hype Person – Marie Chatfield

No one lights a candle and puts it under a bushel. At some point in any engineer’s career, you’re going to have to prove to someone that you’re worth it: worth hiring, worth promoting, worth the raise, worth taking a risk on. The better you’re at making known what you do to others (that’s hyping yourself), the higher your chances of succeeding.

Don’t Lead By Example – James Coling

We’re often told to lead by example. Yes, we all need to set good examples for people to follow. Often time, we overstep our boundaries by continuously taking the lead without setting expectations leaving reports feeling crowded out or disempowered. This post helped me to understand that setting a good example is necessary, but isn’t sufficient for strong technical leadership. Acting like a tech lead means setting clear expectations and embracing direct communication.

Leading Without Managing – David Golden

Julie Zhuo, in her book, defines Leadership as being able to guide and influence other people. When most think of leading, they think of it as something that people in authority or engineers who wear manager’s or tech lead’s hats do. Contrary to that, anyone can be a leader, and it’s a skill anyone can learn. One of the key points I learned from this post was that anyone could lead without authority or coercive power and how to do just that.

Three Crucial Skills Leaders Must Develop to Become Executives – Nikhyl Singhal

“Executive skills are subtle and can be elusive to managers, demanding a great deal of focus, courage, and dedication. Becoming a great executive requires a set of skills beyond what’s needed to be a leader”. Nikhyl Singhalin, in this post, shares crucial skills that leaders must develop to become executives. Some of the key takeaways are:

• Learn to take risks and constantly put yourself in new, challenging, and increasingly ambiguous settings.

• Learn to build strong relationships with other senior people who work in entirely different functions.

• Learn to build teams and trust the people in them.

10 Admirable Attributes of a Great Technical Lead – Elye (Medium Paywall)

“It takes a lot of effort to be a great tech lead. It’s a delicate balancing act between two poles of the same attribute. If there is too much weight on one side, the person may fall”. This post outlines ten attributes that will make any tech lead a better one.

Being a Chief Technology Officer: Lessons Learned in My First Year – Shekhar Gulati

Most people are not lucky to have a manual handed over to them when stepping into a leadership role. As one climbs the ladder, one begins to notice what is required to succeed at the role is slightly different from what makes one an excellent individual contributor. One of the lessons I found valuable in this post includes picking your battles wisely and getting things done without actually doing them.

Building Mental Models of Ideas That Don’t Change – Hammad Khalid

For Individual Contributors and Engineering Managers, there is always new stuff to learn. If you’re at the beginning of your career, it can be overwhelming. One approach to stay above the water is finding a way to prioritise what to learn and a way of keeping track of principles or ideas that don’t change (that’s mental models) to make better and informed decisions. This post outlines a list of engineering principles and management principles, some of which took me years to learn.

A Collection of How-Tos, Templates, and Articles to Help You Be the Manager and Leader Your Team Needs

This link contains a collection of todos, how-tos, templates, and articles for engineering managers and tech leads. The categories covered include techniques and templates for:

• Conducting interviews

• Conducting meetings

• Building diversity and inclusion

• Recruiting

• Building Culture

• Growth and more.

11 Top Responsibilities and 10 Common Mistakes of a Technical Leader– Lorenzo Pasqualis

“Similar to building a Lego tower, building software requires making decisions from the beginning to the end. The first responsibility of a technical leader is to define the engineering reality: What needs to be built, the general technical direction, an example of the golden standard for productivity and technical excellence, the business and technological context, and the time and resource constraints.” If you’re a Tech Lead or an Engineering Manager, you’ll find this post interesting.

First Day Success Manual for New Managers – John Reh

One of the few ways to set yourself for failure when joining a new team is to start by criticizing the existing codebase, past practices and implementing tons of their processes from day one without having a clear picture of how things came to be. Criticizing past practices no matter how ineffective they seem is one of the many ways new hires set themselves up for failure. I love this post because it outlines how to prepare and make a really good impression, from meeting the team to attitude and culture when joining a new team.

Where To Go From Here: Join Software Leads’ Newsletter

To take your learnings further, Join the Software Leads’ newsletter, a monthly newsletter on software engineering and leadership.

Every month, I publish a new issue that includes links to interesting articles, use cases, and insights from experts on leading teams and solving common software engineering challenges at scale. Together, we learn and become better software engineers, better software leads, better engineering managers, and CTOs by making fewer mistakes as we learn from one another.

Building a good Application Programming Interface (API) is more than returning responses. Being a developer and having integrated with tons of APIs, I have noticed a pattern between successful API products and those that are not. It’s about solving problems with great affordance. One might ask the key to building good API products. In this post, I’ll provide some tips that I have found essential when doing so.

After landing my first software engineer role seven years ago, my first big project was building an online book marketplace. In the project, I integrated with multiple Application Programming Interfaces (API). From pre-filling book information using an International Standard Book Number (ISBN) to charging customer credit cards, integration with APIs was key to this project.

Working with APIs

Reflecting on my career, I have integrated, built, and exposed multiple APIs. With time, I began to notice two distinct patterns about APIs. Some have certain characteristics that make developers fall in love with them. There is something about them that draws you in, and you feel like drawing others in too. You just want to tell or recommend it to others at no cost.

Likewise, the universe is not also free of APIs that make you feel otherwise. They appear not to live up to their promises. It’s either the documentation provides a generic list of actions you can perform with the API, or it’s hard to understand how the API can solve your problem. Sometimes, If you manage to scale through the documentation hurdle without issues, the implementation is lurking around the corner to bite you. If you overcome the implementation difficulties, security or support are just around the corner waiting to pounce on you.

Guidelines and Categories

I realized there are two categories of APIs: well-designed APIs and poorly designed ones. Until a particular time in my career, I assumed poorly designed APIs were a function of developers who built them. While this may be true in some cases, I have also learned that the fate of an API is not decided during the development phase only, it starts from the time you conceive an idea that needs to expose some interfaces and beyond. I like to see APIs as not just some mere interface or endpoint, at least in this post. I prefer to think of them as products that expose some interfaces and solve a problem irrespective of the technology or protocols. Therefore I will refer to APIs as API products interchangeably.

While there are so many API guidelines and best practices for implementing them, most are geared towards code implementation details. Little has been said on the business side of things. It’s important to note that well-designed API products ride on a rock-solid business strategy.

The API growth rate is increasing at an explosive rate. According to a recent Postman API report, between January 2019 and January 2020, there was an increase of more than 100% in API growth. This growth has been mainly due to the significant shift in the tech industry and consumer behavior — the rise in cloud adoption, the increase in microservice architecture adoption, and consumers’ love for multi-device experience.

To put these in a better way:

• Consumers are moving from a single to multi-device experience, which drives a need for systems to be integrated.

• Businesses are moving away from the monolithic way of building applications to a more decentralized architecture.

• Organizations are shifting from on-premise data-center to the cloud.

The tech industry’s significant shifts and consumer behavior further buttress that the API economy has come to stay, and businesses benefiting the most tend to create API products that developers love to use.

Every successful product either solves a problem or makes solving a problem easy. Such is true with API products. Good API products are sufficiently capable of solving the problem they’re designed to solve without compromising availability and security. They are easy to use, easy to evolve, and hard to misuse.

Understand the “Why” and “How”

It’s not too uncommon to see APIs that no one understands the problems they are trying to solve. Undeniably, API products are no different from other great products. Great products have a rock-solid strategy — the “why” and “how” are clearly defined and understood. Before you set out to build an API, you first need to ask yourself: why do I want to build an API? What problem will the API solve or make it easy to solve? How will the API help me achieve my goals? If you’re unable to understand the business needs for an API, you should engage the right people and figure them out before designing it. Your strategy will transverse down from the business to implementation details. When your strategy is clear, It will challenge developers to think creatively in achieving the end goals.

Design Matters

Mehdi Medjaoui, in his book Continuous API Management, describes the design as something you do when you make decisions about how something you’re creating will look, feel, and be used. Each time you add or update an API, you’re actively making design decisions for better or worse. This is why you have to make a deliberate effort at the design stage — you can’t rush it.

When designing your API, you should pay attention to the following:

• Vocabularies: Are your words and terms easy to understand for your users?

• Styles: What protocols are you supporting, Rest, or GraphQL?

• Naturalness: Do your users have to change their usual ways of solving their problems significantly? Did you follow established standards and conventions?

• Consistency: What level of familiarity will you provide? Are your APIs similar to what your users may have used in the past?

Get Good at Documentation

Documentation is an important aspect of building API products that people love to use. The year 2020 state of the API report showed that documentation is the highest obstacle to consuming APIs. If you’re a developer, you understand how frustrating a lack of sufficient documentation can be when integrating with APIs. Be aware that people will read your API documentation for three primary reasons: evaluation, integration, and debugging.

Evaluation is what people do when they are trying to see if your API product solves their problem. Once your API is evaluated, users will need your documentation for integration. Once your API is embedded in their application, they will visit it when something goes wrong.

If you’re providing only one type of documentation, you’re undeserving your users. Teams must understand that documentation isn’t just a “dump” of API parameters. There are marketing and customer support aspects that should be included. Besides your endpoints, errors, requests, and response structures, your documentation should also tell the story, how you see the problems, and your solutions to solve those problems.

Get Development Right

This is one of the most important phases in API development. This is where your ideas, strategies are turned into implementations. The development phase contains many vital decisions that are hard to reverse if not made right. You will need to make some internal decisions like what technologies to use, protocols, runtimes, and where to run the software that powers your API. End users don’t care whether you use NoSQL or MySQL as your datastore. What’s important when you make technology decisions is that you choose a technology your team can support.

The quality of tech decisions you make transcends what is needed to launch your product’s first version. Launching the first version of your API is one part of the job. You have to make decisions about your API’s quality, reliability, changeability, and maintainability over a lifetime. API performance and uptime are considered to be some of the major determinants of whether an API product meets consumers’ expectations. This sheds light on how consumers will assess your API.

Tracking Effectiveness

You also have to decide what to test and how you will test it to build confidence in your API. In summary, your tests should be able to cover the following:

• Can the API Product deliver on the strategic goals?

• Is the API Product quality enough to support the strategic goals?

APIs are meant to be simple, they hide complexities and do not expose them. I like to compare APIs to an electric plug — simple to use without any complicated manuals — an electric plug does not care about the source of electricity, neither does it care about the change over switch installed or the electricity provider. It’s a conduit for electricity–plug, and it just works. Admittedly, modern systems are complex, but the complexity should not leak to the interfaces.

In Conclusion

Building a good API is more than accepting requests and returning responses. That’s far from the goal. It’s about solving problems with great affordance. You have to align usability with purpose continuously. It’s important to note that people will use your API differently. There will be different problems people use your API to solve. You can’t change that. What you can not do is be everything to everyone.

Finally, you have to go out there and let developers understand your products’ benefits by connecting with them at various events. You have to let them see how your products make their lives easier. The tech community is competitive. It’s no longer a matter of if you build it, they will come. But instead, build, take it to them and beyond.

Originally published by me on www.mindtheproduct.com, [03/02/2021].

Like numerous engineers, I’ve built a working code on my machine that refused to then run in other environments due to the disparity of environments, that is, different configurations and dependencies.

Editorial note: This post was initially published on Iamondemand’s blog. You can check out the original here, at their site.

Docker defines containers as “a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.”

By bundling an entire runtime environment, including an application plus all its configurations, binary files, dependencies, and libraries needed to run it into a single image, container technology eliminates the “but it runs on my local machine” problem and allows you to adopt the “build once, deploy anywhere” principle.

Containers provide a consistent deployment environment that development teams can use across software development and delivery pipelines.

The demand today for tolerant, reliable, and scalable software has given rise to thousands of containers running in a typical production environment, which brings a new problem: management. Container orchestration tools like Kubernetes solve this issue.

Kubernetes lets you run and manage containers and deploy stateful and stateless applications with multiple cloud providers. With Kubernetes, you can run a single application across a fleet of machines or different computing environments. Furthermore, you can either deploy your applications to a Kubernetes cluster in the cloud or on-premises. Of course, the major challenge with cloud deployments is selecting the right provider out of the many out there that offer managed Kubernetes as a service.

Before making a decision, it’s important you examine each service provider to identify which one best suits your organization’s needs. In this post, I’ll discuss the top Kubernetes as a Service offerings, their similarities, key differences, and how to choose the right one.

How to Choose the Right Kubernetes Provider

Most cloud providers have their own set of Kubernetes hosting environments for managing containers. Although Kubernetes offerings vary wildly among providers, here are some factors to consider as you shop for the right platform to manage your container workloads:

Data Security

Before selecting a Kubernetes service provider, you should understand each provider’s security governance process, measures, and mechanisms for preserving your data and application. An ideal Kubernetes service provider should align with strict security compliance and industry best practices. It should also implement some sort of confidentiality for sensitive data and provide administrative rights for restricting, monitoring, blocking, and approving users’ access to data.

Performance and Reliability

Reliability is an important requirement of any application running in the cloud. You should ascertain the reliability of cloud service providers by comparing their performance against their service-level agreements for the last 6 to 12 months. Cloud service providers usually publish this information, and even if they don’t, you can always request it.

You shouldn’t expect 100% perfection because every provider will experience downtime at some point. A good example was the recent Google incident. What matters the most is how often downtimes happen and how a provider handles them when they do occur. An ideal Kubernetes service provider should have documented, proven, and established processes for handling unplanned and planned downtime.

Standards and Certifications

Before going forward with a Kubernetes provider, you need to ensure that the provider adheres to the best industry practices and recognized standards. It’d be helpful if you also understood how the provider plans to continuously adhere to these standards. An ideal Kubernetes provider should have good knowledge management, effective data management, service status visibility, and structured processes.

Now that you’re familiar with the criteria to consider while searching for the right Kubernetes Managed Service Provider, the next step is to identify the market’s biggest players.

Top Options for Kubernetes as a Service

There are many cloud services for running container workloads using Kubernetes, but let’s closely examine the Kubernetes solution from the Big 3 players: Amazon’s Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Microsoft’s Azure Kubernetes Service (AKS).

Amazon Elastic Kubernetes Service

Amazon Elastic Kubernetes Service (EKS) is AWS’s managed container offering that allows developers to start, run, and scale Kubernetes apps on-premises or in the AWS cloud. EKS also allows you to plan, schedule, and execute your container workloads across compute services and features available on AWS, such as Spot Instances, Amazon Fargate, and EC2.

With Amazon EKS, you can easily manage your applications and Kubernetes clusters across hybrid environments too, plus you can use the Kubernetes jobs to run parallel or sequential batch workloads on your EKS cluster. Common use cases for Amazon EKS include hybrid development, batch processing, machine learning, and web applications. The fee charged by AWS is $0.10/hour for each EKS cluster created.

Strengths

• Highly configurable and customizable

• Efficiently provisions and scales your Kubernetes application

Limitations

• Complexity involved with addition and customization of node pools

• No automatic node updates

Google Kubernetes Engine (GKE)

Google Kubernetes Engine (GKE) is Google’s managed offering that allows you to deploy, manage, and scale containerized applications. The GKE environment comprises multiple Compute Engine instances grouped together to form clusters that are powered by Kubernetes. By running a GKE cluster, you get to benefit from Google’s advanced cluster management features, including automatic upgrades and scaling, logging and monitoring, node auto-repair, load-balancing, and more.

With Google Kubernetes Engine, you can create clusters based on your budget and the availability requirements of your workload. The two available clusters include: regional and zonal (multi-zonal and single-zonal). GKE has a financially backed Service Level Agreement that guarantees 99.5% availability and 99.95% availability for zonal clustersand regional clusters, respectively. Just like Amazon, Google charges $0.10/hour for every GKE cluster that you create, although there is no charge for Anthos GKE Clusters.

Strengths

• Easy Cluster creation and management

• Automatic cluster patching and upgrades

Limitations

• Difficulty in customizing server configurations

• No adequate documentation for supportive information

Microsoft Azure Kubernetes Service (AKS)

Azure Kubernetes Service (AKS) is a highly secure, available, and fully managed Kubernetes service that allows you to deploy and manage containerized apps more easily. AKS offers an integrated CI/CD experience, serverless Kubernetes, enterprise-grade governance, and security. Your DevOps team can leverage AKS to build, deliver, and scale applications fast.

AKS has a financially backed Service Level Agreement that guarantees 99.95% availability for Kubernetes clusters using Azure Availability Zone and 99.9% availability for Kubernetes clusters that don’t. Unlike other providers, Azure doesn’t charge for Kubernetes cluster management. However, you pay for the resources you use, like networking, storage resources, and virtual machine instances.

Strengths

• Faster end-to-end development experience via Azure DevOps, Kubernetes tools, Visual Studio Code, and Azure Monitor

• Advanced access and identity management using Azure Active Directory

Limitations

• No automatic node updates

• Inability to update availability zone settings after a cluster is created

Comparison Criteria

Google, Amazon, and Microsoft Azure all offer Managed Kubernetes solutions, but you’re at liberty to select the provider whose offerings meet your organization’s needs and requirements. If you don’t want to use EKS, AKS, and GKE, the three cloud providers do have other alternatives that you can use to run container workloads.

Alternative to Kubernetes for Running Container Workloads

There are two ways to run container workloads on the cloud: using managed Kubernetes services like AKS, GKE, EKS or by leveraging the serverless container services provided by the cloud providers. If you don’t plan to run on the cloud, you can also adopt a self-hosted solution like OpenShift to run container workloads.

By implementing serverless container services, you can manage the life cycle and availability of your container workloads. Examples of serverless container services include Google Cloud Run, AWS Fargate, and Azure Container Services.

AWS Fargate

AWS Fargate is a serverless compute engine that lets you run containers without the need to provision, configure, or scale clusters of your virtual machines. With Fargate, you don’t have to choose server types, decide when to optimize cluster packing, or scale your cluster. Development teams can thus focus on building and operating applications while Fargate manages all the infrastructure and scaling needed to run the app with high availability.

Google Cloud Run

Cloud Run is a serverless compute platform that allows you to build and deploy highly scalable applications without the need for Kubernetes-based or VM deployments. With Cloud Run, you can build applications with your favorite tools/dependencies and in your favorite language and deploy the apps in seconds.

Cloud Run offers several benefits, including auto-scaling, no up-front provisioning, and zero server management. It’s ideal for use cases like mobile backends, web applications, stateless HTTP applications, streaming, batch data processing, etc.

Azure Container Instances

Azure Container Instances (ACI) is a serverless compute platform that enables you to easily run containers without having to handle the management of the virtual servers. ACI lets you deploy containers in the cloud with remarkable speed and simplicity. It also enables you to secure applications with hypervisor isolation.

By running your application in Azure Container Instances, you can focus on developing applications without worrying about managing the infrastructure that runs the app. ACI is suitable for data-processing jobs, building event-driven applications, and elastic bursting.

Wrapping Up

Containers solve the problem associated with the multiple, different environments that development teams must deal with today, letting them build and deploy the same application for development, testing, and production environments. And Kubernetes allows you to orchestrate these containers.

Although there are several cloud service providers for your container workload orchestration, you need to compare what each provider brings to the table. By understanding the strengths and limitations of the various managed Kubernetes services, you’ll be well positioned to select the right provider that perfectly suits your organization’s needs.

Editorial note: This post was initially published on Iamondemand’s blog. You can check out the original here, at their site.

When your systems are distributed, various things can go wrong. Even if you’ve written the perfect code, a node may fail, a connection may timeout, or participant servers may act arbitrarily. The bottom line is that things will break. And when they do, you want to be able to identify and fix the problem as soon as possible before it alters the entire system’s performance, or affects customers or your organization’s reputation. For this reason, we need observability to run today’s services and infrastructure.

People have varying knowledge of what observability means. For some engineers, it’s the old wine of monitoring in a pristine bottle. For others, it’s an umbrella concept that includes log analysis, trace analysis for distributed systems, visualization, and alerts management.

Honeycomb, in its Guide to Achieving Observability, also defines observability as the ability to ask arbitrary questions about your production environment without having to know beforehand what you wanted to ask. Despite the variability in these definitions, they both explain the overarching goal of observability, which is to achieve better, unprecedented visibility into systems.

Observability is a system that enables you to understand what’s really happening in your software, from the outside. An observable system provides all the information you need in real time to address the day-to-day questions you might have about a system. It also enables you to navigate from effect to cause whenever the system develops a fault.

An effective observability solution may address questions like:

• Why is “y” broken?
• What went wrong during the release of feature “x”?
• Why has system performance degraded over the past few months?
• What did my service look like at point “y”?
• Is this system issue affecting specific users or all of them?

You may ask why you should put in the effort to make your systems observable. I discuss the reasons in the following sections.

The Unpredictability of Distributed Systems

A distributed system comprises multiple components located on different networked computers, communicating and coordinating their actions by passing messages from one to another. The inherent integrations and nature of distributed systems lead to “layers of distinct ownership,” which are sometimes challenging to manage. By implementing observability across a development environment, you’re able to understand your system’s failure modes and trace issues to their root cause.

The Need to Always Maintain Software

After an application is designed, implemented, and launched, it also has to be maintained. Development teams need to continuously adapt to changing customer behavior and ensure that the application runs at peak performance levels and works as expected. An observable system makes application maintenance easier. You’re able to easily identify errors, fix bugs that might arise, customize the application to users’ needs, and eventually improve the app’s performance.

The Need to Speed Up Code Development and Deployment

DevOps is about speed: faster application development and shipments, more immediate updates, and continuous development—all of which lead to a shortened development lifecycle and provide faster and continuous delivery with high software quality. If your development teams cannot identify and address issues before they occur, or they cannot react quickly to changes, it may be difficult to speed up the time to market. By leveraging a strong observability platform, software development teams can increase speed and effectiveness in deploying code, updating, and tracking changes.

In a nutshell, observability offers many benefits to managers and engineers. It also plays a vital role in an organization—especially in maintaining and enhancing a system’s overall architecture and performance. As distributed systems become more complex, engineers can infer the system’s internal states through externally available knowledge, including tracing data, log data, monitoring data, and so on.

Pillars of Observability

Observability is about instrumenting systems to gather useful data that tells you when and why systems are behaving in a certain way. Whenever a system develops a fault, you should first examine telemetry data to better understand why the error occurred.

The following telemetry data—commonly referred to as the pillars of observability—are key to achieving observability in distributed systems:

• Logs
• Metrics
• Tracing

Figure 1: Three pillars of observability

Logs

Logs are structured and unstructured lines of text a system produces when certain codes run. In general, you can think of a log as a record of an event that happened within an application. Logs help uncover unpredictable and emergent behaviors exhibited by components of microservices architecture.

They’re easy to generate and instrument. And most application frameworks, libraries, and languages come with support for logging. Virtually every component of a distributed system generates logs of actions and events at any point.

Log files provide comprehensive system details, such as a fault, and the specific time when the fault occurred. By analyzing the logs, you can troubleshoot your code and identify where and why the error occurred. Logs are also useful for troubleshooting security incidents in load balancers, caches, and databases.

Metrics

Metrics are a numerical representation of data that can be used to determine a service or component’s overall behavior over time. Metrics comprise a set of attributes (e.g., name, value, label, and timestamp) that convey information about SLAs, SLOs, and SLIs.

Unlike an event log, which records specific events, metrics are a measured value derived from system performance. Metrics are real time-savers because you can easily correlate them across infrastructure components to get a holistic view of system health and performance. They also enable easier querying and longer retention of data.

You can gather metrics on system uptime, response time, the number of requests per second, and how much processing power or memory an application is using, for example. Typically, SREs and ops engineers use metrics to trigger alerts whenever a system value goes above a specified threshold.

For example, let’s say you want to monitor the requests per second in an HTTP service. You notice an abrupt spike in traffic, and you want to know what’s happening in your system. Metrics provide deeper visibility and insight that help you understand the cause of the spike. The spike could be due to an incorrect service configuration, malicious behavior, or issues with other parts of your system. In addition to providing visibility, you can also use the information to detect and determine the severity of issues.

Trace

Although logs and metrics might be adequate for understanding individual system behavior and performance, they rarely provide helpful information for understanding the lifetime of a request in a distributed system. To view and understand the entire lifecycle of a request or action across several systems, you need another observability technique called tracing.

A trace represents the entire journey of a request or action as it moves through all the nodes of a distributed system. Traces allow you to profile and observe systems, especially containerized applications, serverless architectures, or microservices architecture. By analyzing trace data, you and your team can measure overall system health, pinpoint bottlenecks, identify and resolve issues faster, and prioritize high-value areas for optimization and improvements.

Traces are an essential pillar of observability because they provide context for the other components of observability. For instance, you can analyze a trace to identify the most valuable metrics based on what you’re trying to accomplish, or the logs relevant to the issue you’re trying to troubleshoot.

Tracing is better suited for debugging and monitoring complex applications that contend for resources (e.g., a mutex, disk, or network) in a nontrivial manner. Tracing provides quick answers to the following questions in distributed software environments:

• Which services have inefficient or problematic code that should be prioritized for optimization?
• How is the health and performance of services that make up a distributed architecture?
• What are the performance bottlenecks that could affect the overall end-user experience?

Although logs, traces, and metrics each serve their own unique purpose, they all work together to help you better understand the performance and behavior of distributed systems. If your organization already uses microservice or serverless architecture, or if they plan to adopt containers and microservices, a combination of all the telemetry data will provide the detailed information needed to understand and debug your system.

Implementing Observability

The fundamental process of implementing an observability-centric culture across your development environment relies on adopting tools that enable you to support and derive insights from the pillars of observability outlined above.

First, you need to select a modern observability platform that perfectly fits your specific needs. A full-stack platform that condenses all the telemetry data from metrics, logs, and traces into a visual, intuitive, and easy-to-understand dashboard can be a good starting point. (Just be careful not to choose a platform that locks you into a specific cloud platform.)

Then, you need to identify and monitor metrics related to issues you’ve already experienced and those you could likely encounter in the future.

After integrating observability into your incident management process, the next step is to establish an observability-centric culture across your organization. Absent of this, you won’t be able to get the best out of tracing, metrics, and log management. No amount of observability tooling can serve as a substitute for sound engineering instincts and intuition.

By selecting the right observability platform, getting a connected view of your system’s performance data in a singular view, and establishing an observability culture, you’ll be able to identify issues faster, understand what caused them, and eventually, build customer-focused products at greater speeds.

Conclusion

As the adoption of microservices and containers increases, engineers and managers need to develop a culture of observability. Doing so will help your teams get the best out of your investments in the cloud. It will help you form a continuous innovation culture that delivers high-end software to customers.

It doesn’t matter if you’re a seasoned DevOps pro or just getting started with observability. Understanding the three pillars of observability is an essential part of developing an observability-centric culture. Practice what you’ve learned in this post, and you’ll be on your way to achieving observability in distributed systems.

Editorial note: This post was initially published on Iamondemand’s blog. You can check out the original here, at their site.

Machine Learning in the Cloud

Cloud-based machine learning enables organizations to access high-performance resources and infrastructures that they couldn’t use (or even afford) on their own. Today, cloud use involves networking, storage, and computing, but when cloud computing is combined with ML to form the “intelligent cloud,” its capabilities increase and it is much easier to protect, scale, and handle. With the great strides taking place in cloud and machine learning development, their future seems ever-more tied together.

Although cloud computing is not new, some organizations have yet to embrace it. The question that comes to mind is: What do you gain by running ML workloads in the cloud? Below are some answers.

Reduced Costs

Hosting your learning workloads on the cloud will save you a considerable amount in capital costs because you won’t need any physical hardware investments.

Also, you won’t need to hire professional personnel to maintain the hardware, since cloud service providers buy and manage the hardware equipment themselves.

Availability

In an on-premise work environment, anything can happen. Things break apart: processors burn, hard drives get corrupted, and networks get interrupted.

The cloud aims to deliver abstract computing space, memory, and power, making you forget about the challenges of the physical world. Fortunately, most cloud service providers are reliable and maintain 99.9% uptime, so your Machine Learning team or engineers can always get their work done on the cloud. And of course, you can access ML services on the cloud from anywhere at any time.

Elasticity

Often, businesses don’t predict their future needs correctly and purchase more hardware than they need or else don’t have enough servers to handle computing tasks. Sadly, as they try to correct their errors, the process becomes more and more expensive.

The cloud enables businesses to experiment with different ML technologies and scale up or down as and when needed. With nominal monthly fees, they can expand capacity.

Of course, the resources of every cloud service provider are limited, but they’re still far beyond what 99% of businesses will need. A good cloud computing system has many different tools, bandwidths, and storage space options.

On-Demand Self-Service

When machine learning workloads are hosted in the cloud, you can access them at any time without administrator approval, as you would need with on-premise data centers. Cloud computing achieves this through automation and self-service.

On-demand self-service also helps businesses to plan. For instance, you can request new virtual machines or storage capacities at any time and expect to have them within seconds.

Broad Network Access

With machine learning workloads hosted in the cloud, you’ll have unlimited data access privileges. Machine learning uses data as its input for smarter decision-making and improved performance, and cloud environments can hold massive volumes of data. That means that using ML in the cloud will help you overcome the delay in traditional silos and accessibility. ML and AI are also great instruments for transferring data between the on-premise infrastructure and cloud environment.

Amazon SageMaker Cloud ML Service

Amazon SageMaker is a fully-managed service that helps data scientists, analysts, and software developers to build, train, and deploy machine learning models on the cloud.

SageMaker provides an integrated Jupyter notebook environment for easy, seamless access to your data sources for analysis and exploration, as well as optimized ML algorithms that can run effectively against a huge amount of data in a distributed environment. SageMaker helps to remove any obstacle that might prevent you from building ML solutions or cause a delay.

How SageMaker Works

SageMaker has a three-step process that simplifies machine learning modelling. Let’s take a look:

Figure 1: AWS SageMaker: Build, train, deploy

Build

SageMaker has a completely integrated development environment you can leverage to build production-ready ML models. With its hosted and easy-to-use Jupyter notebooks, you can visualize and explore data stored in Amazon S3. Besides, SageMaker also comes installed with tons of optimized ML algorithms.

Train

You can use SageMaker’s Experiments to organize, track, assess, and compare every iteration to ML models. You can train your ML model on a managed infrastructure by making a simple API call or via a single click. With the help of its automatic model tuning capabilities, you can select the best set of hyperparameters from your preferred algorithm.

Deploy

After model training and hyperparameter tuning, your next step is to deploy the ML model to generate predictions for batch data or new data. Once the model is deployed, it will be hosted on auto-scaling Amazon ML instances across several availability zones for high availability and performance.

Amazon SageMaker Features

SageMaker has many useful features that make building, training, and deployment of ML models easy for users. These include the following:

Inbuilt Machine Learning Algorithms

Amazon SageMaker comes installed with many ML algorithms that you can use to train big datasets. These algorithms are always available and optimized for accuracy, scale, and speed.

SageMaker includes supervised ML algorithms such as logistic/linear regression and XGBoost. These algorithms are used to solve time series prediction or recommendation problems. SageMaker also includes support for unsupervised learning, such as principal component analysis and k-means. These unsupervised algorithms are used to address problems like clustering and customer segmentation.

End-to-End Machine Learning Platform

An end-to-end machine learning platform is a platform designed to speed up modeling and deployment and ensure scalability and reliability in production. Amazon SageMaker supports the end-to-end lifecycle of ML applications, from data collection to model building, training, deployment, and scaling.

Zero Setup

Research shows that there’s an increasing demand for ML, and the future of ML is promising. Yet according to a 2018 US Census Bureau survey of nearly 600,000 businesses, a mere 2.8% were using machine learning, and only 8.9% were using any form of AI. This shows that many enterprises are facing a crucial knowledge gap—but the cloud offers a solution.

Amazon SageMaker lets you implement ML features without some of the operational overheads experienced with on-premise setup. SageMaker also provides the APIs and SDKs, freeing you from responsibility for setup, and you can embed ML functionalities on the go.

Pay for What You Use

Using the cloud gives you cost-effective resourcing and increased agility. With AWS SageMaker, you pay only for resources that you use. When building, training, and deploying your ML models on SageMaker, you’ll be billed by the second, with no upfront commitments and no minimum fees. Pricing is broken down by on-demand ML storage, ML instances, and data processing.

The pay-as-you-use model is perfect for bursting ML workloads and helps you leverage the power and speed of GPUs for training without incurring the costs of hardware investment.

Flexible Model Training

ML model development can be an iterative, expensive, and complicated process. Also, model training jobs can take anywhere from minutes to hours or even days. With SageMaker, you can efficiently train and build ML models at scale. This means you can experiment with ML capabilities and services and automatically scale based on resource demands.

Training Your First Model on Sagemaker

On a cool Monday morning, you get an email from your boss saying, “Our focus for the next quarter is building an application that predicts whether a customer will sign up for a Certificate of Deposit (CD).” Our target customers are people who are looking to invest their capital. Your job is to build a machine learning model capable of predicting if a customer will enroll, given the features and the data from the marketing team. This is a typical end-to-end ML project that you can efficiently build and deploy on SageMaker. In this section, we’ll learn how to get started with SageMaker and to train and deploy ML models on it.

Note: To use the service, you’ll need to create an AWS account.

Now let’s dive right in.

Step 1: Create a Notebook Instance

1. Log into the AWS Management console, type “SageMaker” in the search bar and select it.

2. In the SageMaker console, click “Notebook instances,” then “Create notebook instance.”

1. Fill in the required information (like Notebook instance name or Notebook instance type), then click “Create notebook instance.”

1. When the server status changes from “Pending” to “Inservice,” click “Open Jupyter.”

Step 2: Load, Visualize, and Prepare Data

1. Once Jupyter opens, click “New” and select “Conda Python 3.”

1. Import the relevant libraries, define some environment variables in the notebook environment (as shown below), and run the cell.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import boto3, re, sys, math, json, os, sagemaker, urllib.request
from sagemaker import get_execution_role
from IPython.display import Image
from IPython.display import display
from time import gmtime, strftime
from sagemaker.predictor import csv_serializer

# Define the IAM role 
role = get_execution_role()
prefix = "sagemaker/DEMO-xgboost-dm"
containers = {"eu-west-1": "685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:latest",
              "us-west-2": "433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest",
              "us-east-1": "811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest",
              "us-east-2": "825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest"
              }
# every region has its XGBoost container 
my_region = boto3.session.Session().region_name
# set the region of the instance
print("Success - the MySageMakerInstance is in the " + my_region + " region. You will use the " + containers[
    my_region] + " container for your SageMaker endpoint.")

1. Create an S3 bucket. The training data and model artifacts will be saved in the bucket. (In the screen capture below, the bucket name is “awsexperimentbucket1000.”)

If the S3 bucket is created successfully, your code will run without any errors (as shown below).

bucket_name = 'awsexperimentbucket1000' # you can always use any name as your bucket name
s3 = boto3.resource('s3')
try: 
    if my_region == 'us-east-1':  
    s3.create_bucket(Bucket=bucket_name) 
    else:  
    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={ 'LocationConstraint': my_region }) print('S3 bucket created successfully')

except Exception as e: print('S3 error: ',e)

1. Download the data, then load it into a dataframe.

   If everything is successful, your code will run without any errors (as shown below).

'''this is the link to the datasethttps://d1.awsstatic.com/tmt/build-train-deploy-machine-learning-model-sagemaker/bank_clean.27f01fbbdf43271788427f3682996ae29ceca05d.csv", "bank_clean.csv"'''

try:
    urllib.request.urlretrieve ("https://d1.awsstatic.com/tmt/build-train-deploy-machine-learning-model-sagemaker/bank_clean.27f01fbbdf43271788427f3682996ae29ceca05d.csv", "bank_clean.csv")
print('Data downloaded successfully')
except Exception as e:print('Data load error: ',e)

try:
    df = pd.read_csv('./bank_clean.csv',index_col=0)print('Data loaded into dataframe successfully')
except Exception as e: print('Data load error: ',e)

You can use the Pandas head, shape, or columns function to explore the dataset.

1. Divide the dataset into a train set and a test set.

The train set (75% of the data) will be used to build the model, while the test set (25% of the data) will be used to evaluate the model’s performance.

### the np.split() function splits the dataset into train and test 

settrain_df, test_df = np.split(df.sample(frac=1, random_state=42), [int(0.75 * len(df))])
print(train_df.shape, test_df.shape)

Step 3: Train the Model

In this step, you’ll train your ML model with the train_df dataset.

1. To use SageMaker’s prebuilt XGBoost model, reformat the dataset structure and load the data from the AWS S3 bucket. See code below:

pd.concat([train_df['y_yes'], train_df.drop(['y_no', 'y_yes'], axis=1)],
          axis=1).to_csv('train.csv', index=False, header=False)
boto3.Session().resource('s3').Bucket(bucket_name).Object(os.path.join(prefix,
                                                                       'train/train.csv')).upload_file('train.csv')
s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket_name, prefix),
                                    content_type='csv')

1. Set up an Amazon SageMaker session, instantiate the estimator (XGBoost model), then define its parameters. See code below:

xgbsession = sagemaker.Session()
xgboost_model = sagemaker.estimator.Estimator(containers[my_region], role, train_instance_count=1,
                                              train_instance_type='ml.m4.xlarge',
                                              output_path='s3://{}/{}/output'.format(bucket_name, prefix),
                                              sagemaker_session=session)
xgboost_model.set_hyperparameters(max_depth=5, eta=0.2, gamma=4, min_child_weight=6, subsample=0.8, silent=0,
                                  objective='binary:logistic', num_round=100)
_predictor = xgboost_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

1. Now that the dataset has been loaded and you’ve set up the estimator, use gradient optimization to train the model on the “ml.m4.xlarge” instance. See code below:

xgboost_model.fit({'train': s3_input_train})

Violà! The model training is successful.

Step 4: Deploy the Model

Deploy the model built to an endpoint. See code below:

xgb_predictor = xgboost_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

Once the deployment is successful, the code above will run without any errors.

Now that you’ve deployed the model, you can use the test set to generate a set of predictions. See code below.

  # load the data into an array
test_df_array = test_df.drop(['y_no', 'y_yes'],axis=1).values 
# set the data type for an inference
xgb_predictor.content_type = 'text/csv'
# set the serializer type
xgb_predictor.serializer = csv_serializer
predictions = xgb_predictor.predict(test_df_array).decode('utf-8')
# predict!
predictions_array = np.fromstring(predictions[1:], sep=',')
# and turn the prediction into an array
pd.DataFrame(predictions_array).rename(columns={0: "predicted_values"})

Conclusion

Cloud computing and ML technologies are transforming the world we live in. Amazon SageMaker minimizes the need for maintenance and streamlines the machine learning pipelines while cutting costs. It’s a valuable tool for building end-to-end ML solutions at scale. I hope this article helps you get started.