For context, Keeplys is a hobby project: a simple document manager running on a mobile phone that I’ve been building in my free time. When you store or scan documents via Keeplys, it organises them into a folder locally on the device.

Keeplys is local-first by default. Your documents live on your device but there is an important question: what happens if a device is lost? How do you recover your documents? It wasn’t hard to see that this was a problem worth solving.

Before building the sync feature, the main question on my mind was: how do you sync files to the cloud? I wanted to be able to scan documents on my phone, organise them locally, and have the files backed up to the cloud.

The easy answer would have been to simply upload files, or download updates once a day. This approach quickly fell short when you consider:

• What if I (or a user) edit a document on my phone while editing the same document on their laptop? How does the app resolve conflicts?
• What if the network drops mid-upload while syncing files?
• What if a user runs out of cloud storage?
• What if someone deletes the parent cloud folder used for backup (Keeplys) directly from Google Drive’s web interface?
• What about iCloud, which isn’t even a REST API?

This is a story of how I built a sync engine that handles all of this.

Implementation guiding principles

I’ve come to understand the power of setting guiding principles when exploring choices and narrowing options down. I leaned on this approach and defined a few principles that I wanted the feature to adhere to.

The app must work perfectly offline

Cloud sync is a feature and never should it be a dependency. Users should never see a loading spinner waiting for the sync.

This means:

• Documents are saved locally first, always
• Sync happens asynchronously in the background
• Pending uploads queue up and execute when connectivity returns

Cloud deletion shouldn’t interfere with what is stored locally

The most dangerous operation in any sync system is delete. Google Drive and iCloud should never silently delete local data.

If a user deletes a file from Google Drive’s web interface, it shouldn’t mirror that action locally. Instead, the document should be marked as unsynced and the user should decide what to do.

Drive agnostic

I want to leave room for extension. The first iteration supports Google Drive and iCloud, but I want to allow for other providers like Dropbox, Box, etc. To support this, the design must be drive agnostic. I designed a single sync engine that talks to a DriveProvider interface.

Documents should have observable state

Documents should have visible sync status: synced, pending, conflict, or unsynced. Users should always know what’s happening.

The domain model

To support the above, I created the following core types:

// A local document
interface Document {
  id: string;
  name: string;
  contentHash: string; // SHA-256 of the document
  modifiedAt: number;
  // ...
}

// A reference to a docume in a cloud drive or storage
interface RemoteRef {
  provider: 'google-drive' | 'apple-icloud';
  providerFileId: string;
  etag?: string; // The cloud's version identifier
}

// A queued operation
interface Operation {
  id: string; // UUID for idempotency
  type: 'create' | 'update' | 'delete' | 'move';
  localId: string;
  retryCount: number;
}

// The sync state for a provider
interface SyncState {
  provider: string;
  changeCursor?: string; // Delta token for incremental sync
  pendingOps: Operation[];
}

One key thing to note is the changeCursor. Both Google Drive and iCloud support the concept of a delta token. Delta token is a cursor that lets you ask “what changed since last time?” instead of listing every file on every sync. This is the difference between syncing 3 files and re-scanning 3,000.

The provider interface

I created a contract that every cloud provider must implement. As I add new providers (e.g. Dropbox), each one must conform to DriveProvider.

interface DriveProvider {
  id: string;
  platforms: ('ios' | 'android')[]; // iCloud is iOS-only

  // Auth
  authenticate(): Promise<AuthResult>;

  // The delta API (critical for efficiency)
  getChanges(cursor: string): Promise<ChangeResult>;

  // CRUD operations
  upload(params: UploadParams): Promise<RemoteRef>;
  download(ref: RemoteRef, destination: string): Promise<void>;
  delete(ref: RemoteRef): Promise<void>;

  // Storage quota
  getQuota(): Promise<{ used: number; limit?: number }>;
}

The platforms field is important. iCloud uses a native iOS API (the Ubiquity Container), so it simply doesn’t exist on Android. platforms gives me a way to control platform-specific availability and features.

Here’s how it’s used:

export function getPlatformProviders(): DriveProvider[] {
  return allProviders.filter(p => p.platforms.includes(Platform.OS));
}

On Android, users see only Google Drive in settings. On iOS, they see both Google Drive and iCloud.

Building Google Drive Sync Feature : REST API Cloud Provider

Starting with the Google Drive integration, I quickly learned that its implementation is very different from iCloud’s.

Scoped access To access Google Drive, the app requests the drive.scope, which limits access to files the app itself created. The app can never see a user’s vacation photos or tax documents.

The delta API Google’s changes.list endpoint accepts a pageToken and returns only files that have changed since that token. On the first sync, I list everything. On subsequent syncs, I might get an empty response—meaning nothing changed. This is massively more efficient than re-listing thousands of files.

Conflict detection via ETag Every file has an etag, which acts as a version identifier. When uploading, the app sends:

PUT /upload/drive/v3/files/{fileId}
If-Match: "existing-etag"

If the etag doesn’t match (for example, if the user edited the file elsewhere), Google returns 412 Precondition Failed. The app catches this and routes the document to a conflict resolver.

Resumable uploads For files over 5 MB, I used Google’s resumable upload protocol. This means if the connection drops at 80%, the app resumes from 80%—not from 0%.

Building iCloud Sync: The Filesystem Provider

iCloud was a completely different beast. It’s not a REST API. It’s a synchronized filesystem managed by iOS itself.

The ubiquity container

iCloud gives each app a special folder called the Ubiquity Container. Files you put there are automatically synced by iOS in the background. To “upload” a file, you simply copy it into this folder:

let source = localDocumentURL
let destination = ubiquityContainerURL.appendingPathComponent("document.pdf")

try FileManager.default.copyItem(at: source, to: destination)

// iOS handles the rest

This sounds straightforward, but it’s not quite that simple. Getting entitlements set up is very different from how Google Drive works.

The config plugin

Since I’m building the app in React Native, it doesn’t know about iCloud entitlements. I had to build an Expo Config Plugin (a big thanks to Claude Code here) that:

• Enables the iCloud capability in Xcode
• Adds the correct entitlements to the app
• Exposes the ubiquity container path to JavaScript

No delta API

Unlike Google Drive, iCloud has no cursor-based delta API. Instead, I used NSMetadataQuery to watch the container for changes:

let query = NSMetadataQuery()
query.predicate = NSPredicate(format: "%K LIKE '*'", NSMetadataItemPathKey)
query.searchScopes = [NSMetadataQueryUbiquitousDocumentsScope]
query.start()

This gives live notifications when files change — more real-time than polling, but also harder to implement.

The sync engine

With providers abstracted, I built a sync engine that:

• Transforms local changes into operations
  • Document saved → create or update operation queued
  • Document deleted → delete operation queued
• Persists the operation queue
  • Stored in AsyncStorage as a “sync journal”
  • Survives app crashes and restarts
• Executes operations with retry logic
  • Transient failures: exponential backoff (1s → 2s → 4s → 8s…)
  • Rate limits: respect Retry-After headers
  • Auth failures: pause sync and prompt re-authentication
• Detects and routes conflicts
  • ETag mismatch → conflict state
  • User resolves: keep local, keep remote, or keep both

The sync journal

Every operation gets a UUID for idempotency. If the app crashes after uploading but before updating local metadata, it can safely replay the operation. The cloud will recognize it as a duplicate.

interface SyncJournal {
  providers: {
    [providerId: string]: {
      cursor?: string;
      pendingOps: Operation[];
      failedOps: Operation[];
    };
  };
}

The failedOps array is crucial. After five retries, I stop retrying and surface the error to the user.

Handling deletion in the cloud

Deletion has a dangerous edge case: a remote delete can destroy user data.

Imagine this flow:

1. User links Google Drive
2. All documents sync correctly
3. User opens Google Drive on the web and deletes the Keeplys folder
4. App syncs and sees files are “deleted” remotely
5. App mirrors the delete locally
6. User’s documents are gone forever

I handled this with a soft-delete with confirmation:

Handling the merge problem

All data backups are synced in a folder named “Keeplys”. One edge case I needed to handle was a case where a user links a provider and there’s already a /Keeplys folder in their cloud drive. Maybe from another device, maybe from a previous install.

I used hash-based deduplication as follows:

• List all remote files
• For each local file, compute a SHA-256 hash
• Compare local and remote files by name + hash

This handles the common case of reinstalling the app on a new phone without re-uploading gigabytes of documents.

Handling sync in the background

I didn’t want users to open the app just to sync. I implemented background sync using expo-background-fetch:

BackgroundFetch.registerTaskAsync('KEEPLYS_BACKGROUND_SYNC', {
  minimumInterval: 15 * 60, // iOS minimum: 15 minutes
  stopOnTerminate: false,
  startOnBoot: true,
});

On iOS, this wraps BGAppRefreshTask. On Android, it wraps WorkManager. Both respect battery optimization and network constraints.

I also added a “Sync on Wi-Fi only” toggle. When enabled, background sync checks NetInfo.type === 'wifi' before proceeding.

Over the past few years, I’ve led growth product teams focusing on retention and growing daily active users (DAU). Working on a myriad of challenges aimed at widening access to a global audience has deeply reshaped my understanding of inclusivity and what it truly means to build for a global audience and leave no one behind.

For a long time, I had a narrow view of inclusivity. I saw it as a nice-to-have, rather than a holistic principle that should influence many engineering decisions. Through experience, I’ve learned that even well-intentioned technical trade-offs can unintentionally exclude certain potential users.

Think of it. Most products start small. They begin as simple ideas, gradually evolve, and eventually find market fit. At that point, the business identifies its core users—the cohort that drives the most value and gives the most feedback. Naturally, the focus shifts toward serving them better.

Teams start listening closely to these users’ pain points, prioritizing them, building new features to address their needs, and tailoring the product experience to them. This approach often feels smart and efficient. After all, it follows the familiar “80/20 rule”—what 20% of effort can deliver 80% of the impact?

It seems like a no-brainer to prioritize the 20% of users who drive most, or should I say 80% of the revenue. But over time, this focus creates an unintended consequence: the remaining 80% of users are quietly left behind. They’re excluded.

They might still show up occasionally—out of curiosity or necessity—but each time they do, the product feels a little more distant, a little less for them. The product speaks a quiet truth: “I know you need me, but I wasn’t really built for you.”

Too often, we default to “design and build for core users” or for users with fewer constraints because it promises better ROI. But the deeper, more important questions are:

• What’s standing in the way of users who should be using your product but aren’t?

• Are your design choices excluding people before they even have a chance to become “core users”?

Throughout my career, I’ve often heard (and said) statements like:

“Let’s prioritize the core users who matter most.”

At face value, these statements seem reasonable—and in many cases, they are. Prioritizing core users often feels like a sound, data-driven decision. But it becomes problematic when the entire development process is optimized solely around these users. Over time, the product begins to diverge, unintentionally excluding minority or edge users. The deeper we go down this path, the more self-reinforcing the cycle becomes.

The Engineer’s Bubble

It’s not just in product decisions; it applies equally to engineering. The context we live in profoundly shapes how we build, how we think, and how we perceive reality.

I’m reminded of a story from one of my German classes. It described how everyone is born wearing an invisible pair of glasses through which they view the world. Some have red lenses, others blue, and so on. Each pair of glasses colors our perception—what we see feels real and objective, but it’s deeply shaped by our lens.

The same is true for us as engineers. We build from within our own cultural and technological bubbles, assuming our experiences and environments are universal—but they’re not. Our daily context is one of privilege:

• We use high-end laptops and modern devices.

• We work with fast, reliable internet and constant power.

• We test our applications on the latest browsers and operating systems.

• We are early adopters of new technologies and easily embrace change.

A few years ago, when React was becoming mainstream, my team was thrilled to migrate our legacy app to this shiny new framework. We couldn’t stop talking about how much better the developer experience would be—faster builds, cleaner code, modern patterns. We imagined the productivity gains and the delight of finally using something elegant and efficient.

We rebuilt core parts of the product, added sleek new features, and gave the UI a fresh, modern look. Everything ran beautifully in Chrome, Firefox, and every modern browser. I remember thinking to myself, “How could anyone not love this?” I couldn’t wait for customers to experience it.

Then we launched.

We checked the logs and usage metrics; excitement quickly turned to confusion. Adoption was down, and a sizable portion of our users weren’t making it past the first few screens. What went wrong?

It didn’t take long to find the answer. It was Internet Explorer. They were using that specific browser. We had tested on every major browser except Internet Explorer. Some of our users were still on IE 8 and 9. The app broke completely for them.

We had assumed, without question, that “everyone” used modern browsers by now. But we were wrong.

That experience was a wake-up call. It reminded me that our reality as engineers is not the same as our users’ reality. The tools, devices, and networks we take for granted are luxuries many don’t know or have access to.

A large portion of the world still uses entry-level devices with limited processing power. Many experience intermittent connectivity or rely on expensive, slow internet in their day to day. When we’re not aware of this reality, our products become less and less usable to certain users who should be using them.

Inclusive product engineering is about widening access. It’s about being intentional in how we design and build software. It’s about writing code that runs efficiently and work for all users. It’s about optimizing yours so it has low battery, data, and memory usage. It’s about making sure no one is left behind.

With constant pressure to ship features quickly, inclusive software design can sometimes feel expensive or time-consuming. But the truth is, inclusion doesn’t always require much from the start. It requires more awareness, discipline, and intentionality from us.

How can you start making your product more accessible to the world?

Start with awareness

• Begin with questions, not code. Challenge assumptions early and often.

• Support accessibility from the start.

• Who are you building for, how diverse are they, and how do they engage in the digital world?

• Who might this exclude?

• If you’re improving an existing feature, where are users dropping off—and why?

• What happens if the user has a slower device or poor connectivity?

Test your product under different constraints

Just as automated tests shouldn’t stop at the “happy path,” feature testing shouldn’t either.

• Throttle network speeds to simulate unreliable internet. Does your app handle it gracefully?

• Test on older, entry-level devices or budget phones.

• Use accessibility tools like color-blindness simulators or voiceover modes to check usability.

• How does the app work with LTR and RTL languages?

Design and code with constraints in mind

• Use lightweight assets and reduce dependency on heavy libraries.

• Prioritize clarity and the primary jobs users need to get done.

• Respect data-saving mode in your code.

Involve diverse voices early

Encourage your research and design teams to broaden their participant groups and ensure diverse representation.

If that’s not possible, ask for feedback from teammates in different regions, on different devices, or with different cultural perspectives. Even small variations in context can surface assumptions you didn’t realize you were making.

In conclusion, focusing on the “core 20%” might feel efficient in the short term, but it risks alienating the very users who could expand your product’s reach and impact. As engineers, our code shapes the world. Making the world more equitable begins by stepping outside our bubble, questioning our assumptions, and designing for the full spectrum of users—not just the ones who look, think, or live like us.

We all learn through feedback, from the searing pain of a hot stove to the warm praise of a job well done. It’s the cornerstone of growth, both in life and especially in the world of work.
As a senior engineer or tech lead, feedback is one of the most effective tools in your toolbox to help level up people around you.

But how do you give feedback effectively? For many, it’s shrouded in misconceptions and uncomfortable emotions.

It took me some time to understand some misconceptions I held about giving feedback. As an example, I once asked a report in a 1:1, What change can I make now to better support you? He responded, “I would love to receive more feedback from you.” I thought inwardly that I’m always on the lookout to see where I can give you feedback. You’re doing excellently well, and I can hardly find areas to suggest improvement.

One of the common misconceptions about giving feedback is that it should be given only when an opportunity for improvement is spotted. And I held on to that misconception for a while.

Ditch the myths

Let’s debunk some myths about giving feedback and explore the four simple steps to getting good at giving feedback.

Myth 1: Feedback is only for negative corrections. Not true! Feedback celebrates strengths, reinforces positive behaviors, and guides future actions; it is not always about fixing mistakes.

Myth 2: Instant feedback is always best. Emotions run high in the moment, leading to potentially hurtful and unproductive exchanges. Wait until you’re calm and collected.

Myth 3: Sandwiching is the way to go. Sugarcoating feedback dilutes its impact. Be direct, yet respectful.

Myth 4: You need a laundry list of examples. You don’t need a laundry list of examples to make your feedback effective. Focus on one or two key observations for clarity and impact.

Myth 5: Feedback is a one-way street. It’s a conversation! Encourage questions, active listening, and shared understanding.

Myth 6: Feedback flows top-down only. High-performing teams have feedback flowing everywhere: leader-to-team, team-to-leader, and peer-to-peer.

Believing that you have to give feedback instantly can make you give feedback when you don’t have your emotions in check. You want to avoid giving feedback when you’re angry because it’s far more likely the recipient will feel hurt or judged—and thus defensive.

Feedback shouldn’t be given only when you want a change from the receiver. Feedback is not only about correcting mistakes and driving change but also about recognizing positive contributions and strengths and reinforcing positive behaviors.

Feedback is a two-way dialogue where both the giver and the receiver share their perspectives, ask questions, listen actively, and seek to understand each other.

Anyone can be both a giver and a receiver of feedback. It should not be a top-down thing. In a high-performing organization, feedback flows from top down (from managers to reports), down up (from reports to managers), and laterally (from peers to peers).

Four simple steps to get better at giving feedback.

Follow these four steps to offer better feedback.

1. Ask

Before diving in, ensure the recipient is open to hearing your thoughts. When you ask, it gives the receiver control, ensures that you have the attention of the person, and sets expectations for what the conversation will be about.

A simple question like “Can I share some feedback with you?” can do wonders.

2. Describe the behavior or action

Avoid judging intentions. As a feedback giver, your job is not to judge the intentions of others. You’re not in a position to put a label on the action. Your role is to express the impact their actions had on you, the team, or the project.

When feedback is focused on the past, the other party can’t do anything about it; defensiveness on the side of the receiver is sometimes inevitable. Good feedback should focus less on the past and more on the future, making the receiver feel that the purpose of the conversation is for the future.

The book Effective Managers describes a simple way of describing actions that I found useful. It starts with the “when” question. For example:

• “When you took the initiative to…”

• “When you presented today…”

• “When you don’t communicate progress like you did yesterday…”

• “When you spoke about…”

• “When you stay an extra hour to find the root cause,”

• “When you respond politely after the customer…”

• “When you were presenting your ideas…”

3. Highlight the impact

Once you have described the action, proceed to explain how the behavior affected you, the team, or the project. For example, when you did that to XYZ, this is the impact (insert impact) it had on the team or the project.

Example:”

Behavior: When you took the initiative to…

Impact: This helped the team move forward efficiently and be able to deliver on time.

4. Encourage future action

If the feedback is negative, suggest what could be done differently in the future. If it’s positive, express appreciation and encourage continued behavior.

Building trust is key

Trust is like the air we breathe; when it’s present, nobody really notices; when it’s absent, everybody notices. – Warren Edward Buffett.

When people trust the giver, they’re more receptive and open to learning. Invest time in building genuine connections with your team. This fosters an environment where feedback is seen as a tool for growth, not criticism.

In conclusion, giving feedback is a skill that can be learned, not a magic trick. By practicing these steps and fostering a culture of trust, you can unlock the true power of feedback.

Have you ever been on a team that’s stuck? A team where everyone is moving in different directions? You spoke to three people, and there is a lack of consensus on what the team is trying to achieve. There are different perceptions of the team’s priorities and direction. Priorities shift from time to time, and you could feel a sense of instability or unpredictability in the team’s work. If you stick a little longer, you start to witness signs of disengagement, low morale, or a lack of enthusiasm.

If you notice several of the above, it’s an indication that the team lacks clarity of direction. Where there is no clarity, a different narrative ensues. If you’re a manager, one of the most important things you can do is drive clarity of purpose, direction, plan, and responsibilities.

I have come to believe that clarity in the context of software development is about giving people a practical North Star to guide their thinking and actions in the right direction, even when the conditions get cloudy. I have to admit, it took me a while to learn how important this is when you lead engineering teams.

A team is a function of its environment. If the environment a team is operating in lacks clarity on its purpose, structure, expectations, and boundaries, it becomes really difficult to achieve sustained high performance.

Clarity provides the vision, the direction, the rallying cry, the strategy, and the goals of your business and team. Clarity grounds organisations and teams into something bigger.

Why does driving clarity matter?

• A collective sense of clarity of purpose within an organization has a direct impact on its performance.

• Clarity drives focus. There’s a reason why startups can move swiftly and outpace larger organizations with significantly more resources. Besides the absence of bureaucracy, successful startups embody a strong sense of purpose and instill a radical focus in every team member.

• A lack of clarity impacts commitment. If it’s not clear, it’s hard to commit to.

• Clarity drives behaviors and actions that propel organizations forward, all aligned toward common goals.

There are three major areas a leader should consistently drive clarity on to get a team to their peak performance.

Clarity of direction and purpose

Charles Garfield, author of several books on peak performance, spent decades studying peak performers and found that intense commitment to what they do is one of the single most dramatic differences between peak performers and their less productive colleagues.

Having hired tens of software engineers and interviewed dozens of software engineers, I have found that engineers who are passionate about what they do and the problems they solve perform better and retain better than those who are not.

What really makes people develop intense commitment and passion for what they do? It is rooted in a deep understanding of “purpose.” An intense attachment to why they’re doing that thing.

The differentiation of performance in engineering teams is how the members feel about what they’re doing. The “how” stems from the purpose. To better put, the “why” behind that thing.

Effective leaders master how to communicate purpose and direction that inspire others and provoke actions.

McKinsey asked over 5000 executives to write down their peak experience as team members and the descriptions of the environment. The first of the three things was alignment in direction, which is a shared belief about what the organization is striving towards.

When there is clarity of purpose, a team has a navigation aid that helps set the direction of collective actions.

Clarity of purpose stems from everyone on the team understanding why they’re doing whatever they’re doing in the first place.

A northstar to aim for is a crucial part of motivating your team. When a team can’t understand why they’re doing what they’re doing, it becomes a losing battle. A small litmus test to understand if a team has clarity of purpose is:

• Does the team understand where the company is headed?

• Does the team understand why the company is headed there?

• Does the team understand how they’re contributing to that?

• Does the team understand how success is measured?

Clarity of purpose drives strategy and resources. To formulate a solid plan, you first need to be clear about what you want to achieve.

Clarity of plan

When you don’t have a plan, randomness ensues. A team member pushes a new activity, and another team member pushes an unrelated task on the side.

There are activities and tasks being done. But the effectiveness of the collective actions falls short. You wonder why the impact was so low. You don’t need to look far to notice an absence of clarity in the destination and the plan to get there.

When you have a plan, it limits your choices and helps people focus on the specific path to get to the intended destination.

Having a plan is the beginning. The end is to get to a place where everyone knows the plan, what they’re to do in the plan, and how that thing they’ll do contributes toward reaching their destination.

A team has clarity of plan when they’re able to understand the pieces and how those pieces will help them get to the destination.

To create plan clarity, start by connecting your overall mission (North Star) to actionable steps that get you to your destination—and creatively distill that into your team.

There are a few steps to take to help you understand and have clarity about your plans.

1. Together with your team, do focused planning by collectively mapping out the important pieces required to get to where you want to get to. People remember what they co-create better.

2. Establish a set of measurable key results that you aim to achieve by specific dates.

3. Map out the big projects that your team will take on to achieve those key results, and then the specific tasks to achieve them.

4. Actively and consistently overcommunicate the plan.

Clarity of roles and responsibilities

In a soccer game, players play with a clear understanding of their roles. The goalkeeper knows his or her role. The strikers and defenders all play with a clear understanding of what role they have to play in the game. A lack of role clarity impedes high performance.

When I picked up my first job as a software engineer, I was handed a contract of engagement with clauses on my roles and a description of what I would be doing.

Months into the job, I saw how different my day-to-day job was from the 8 bullet points listed in the paper as a job description. There were implicit role expectations and responsibilities that were not in the job descriptions, as I learned the hard way.

When individuals in a team have role clarity, everyone knows what they’re responsible for and accountable for. They know the role that they have to play in the plan. They know what is expected, which tasks in the plan they have to accomplish, and how their performance on the tasks will be evaluated.

If you’re leading engineering teams, pay attention to whether individuals on the team have a full understanding of their roles and responsibilities in your overall plan.

You should consistently ask yourself:

• Does everyone understand their roles and what they have to contribute to the team’s plan?

• Do they understand the details of the results to be achieved?

If you’re an IC in a team and you lack role clarity, be proactive by asking your manager detailed questions about your role expectations, and your performance will be measured. Put the expectations in writing and send them to your manager for review.

A few months ago, I spoke at the WeAreDevelopers World Congress on building high-performing software teams. The feedback from that talk has been overwhelming, and I’m thrilled it resonated with many people. In this post, I will unpack one of the key points I discussed: empowering your team and getting out of the way.

I like to think of empowerment as involving a team in decision-making, giving them a participatory role that leverages their skills and judgment and enhances their sense of individual worth and commitment to the team’s goals.

When you begin leading a team, you might have the illusion that your role in the team is what keeps the lights on. You may even think that your decisions and skills alone are the primary factors responsible for getting the team to where it is. Nothing could be further from the truth. No one can lead a team or an organisation to success without the collective excellence of many.

The most effective teams I have seen are autonomous. Every team member works, contributes, and is equally committed to the team’s goals. The higher you advance in an organisation, the more you have to rely on diverse skills and talents in the people you’re leading to make your team thrive.

It’s no coincidence that the primary responsibilities of leaders include coaching, mentoring, and empowering those they lead, giving them the opportunity to take on significant initiatives, make mistakes, and continue to develop and grow along the way.

One of the fundamental changes you can make as a manager is reinforcing the belief that leadership is not just a function of the title someone holds on the team but that everyone can be a leader.

Empowering individuals on the team to lead and influence is crucial. If individuals on your team don’t believe they have the power to lead as you do, hierarchy sets in, as stated by Michael Loop in his book, “The Art of Leadership.” This situation can lead people to feel they need to seek permission to drive change or wait for someone to dictate what to do.

For new managers, there are subtle ways you can disempower your team. I fell into one a few years ago when I began leading a team of engineers. I would jump into facilitating inter-team communications, chase down team dependencies, and shield my team from non-technical work.

As my team’s scope increased, it became increasingly challenging to handle all of these tasks alone.

I decided it was time for me to start involving my team by showing them how I handle these tasks. I thought I would demonstrate how it’s done so they could learn from it. Implicitly, I also expected them to contribute in the areas where I involved them.

I tried hard to lead by setting good examples. However, the more I tried to show them, the less I saw the team replicating what I was doing. In the meetings I involved them, they acted as observers rather than participants. I wanted them to be active contributors, but instead, they took a back seat and simply tagged along.

I realised that I had not only communicated my intent and expectations but had also not provided enough space for them to take on responsibilities and lead. Consequently, the more I took the lead, the more the team felt crowded out. They believed it was easier and faster for me to handle those tasks than for them. They felt like they had nothing to add, which increased their disempowerment.

The example from my experience illustrates subtle ways in which a team or reports can become disempowered. Instead, a leader’s goal should be to work themselves out of their job.

The goal of all leaders should be to work themselves out of a job – Jocko Willink.

Empowering engineers takes different forms in engineering organisations. Sometimes, it could mean providing autonomy, sharing the big picture, and giving your team all the details they need to make the right decisions.

At other times, it could mean refraining from providing your team with solutions and dictating what to build. Instead, give them the problems you want to solve and provide the space for them to find solutions.

In other instances, it could mean empowering engineers to solve problems rather than just completing tasks, and giving them ownership over it end-to-end.

Empower your team through autonomy, purpose and mastery

I often observe striking similarities between motivation and empowerment. When a team lacks the drive or motivation to take ownership of a piece of work for which they are responsible from start to finish, most of the time, it is a sign that the team has not been empowered.

Show me an intrinsically motivated team brimming with passion to tackle problems, and I will show you an empowered team.

Intrinsic motivation is behaviour driven by an internal desire to do something. It’s the motivation to engage that arises from within the individual rather than from the externals.

One of the ways to evoke intrinsic motivation is to provide autonomy. In the book “Drive,” intrinsic motivation is based on three factors: autonomy, mastery, and purpose. Tapping into the intrinsic motivation of individuals on your team leads to more sustainable performance and teamwork and consequently empowerment.

Give your team the right amount of autonomy

Autonomy is one’s ability to control one’s work or make decisions regarding one’s work. Empower your team by providing them with the freedom to choose how they approach their tasks, granting them a sense of ownership and accountability for the outcomes you want them to achieve.

You impede autonomy when you hand down feature requirements to your team without providing the context they need to comprehend the business problem and why they are doing what they’re doing.

A disempowered engineering team lacks the decision-making power regarding “what” they are building and “how” they are building it. The path to building an empowered team is to ensure that everyone has a say in what they are constructing and that they have the appropriate context that enables them to make effective micro-decisions.

Give them space to struggle through stretch projects

We’ve all experienced the joy of learning something new: the excitement that follows learning a new language or the satisfaction of finally mastering how to solve a persistent problem. That feeling of continuous improvement on something that felt nearly impossible a month ago is a crucial source of motivation that can keep you going.

Empower individuals on your team by providing them with opportunities to work on tasks that stretch their abilities—tasks that take them slightly out of their comfort zone but offer growth prospects.

Is there a team member who hasn’t yet had the chance to develop their skills in handling cross-functional projects? Create an opportunity that exposes that engineer to cross-functional projects where they can acquire new skills and collaborate with a diverse group of colleagues.

When you don’t allow your team to take on challenging tasks or attempt to shield them from challenges, you deny them the chance to master those challenges. By doing so, you’re not only disempowering those individuals, and you might also be depriving them of coaching opportunities.

Give them a sense of purpose

Purpose signifies understanding how one’s work contributes to something meaningful and impactful. When a team can connect their efforts to a larger purpose or mission, their motivation and engagement increase. To empower your team to make informed judgment calls, provide them with a sense of purpose, which offers the context they need to do so effectively.

An engineering team with clarity about why they are doing what they’re doing will make better implementation decisions that align with the “why.” When an engineering team’s purpose is clear, they understand what they are working on, why they are working on it, and how their work will have an impact. By making sure your team is clear on their purpose, you’re empowering them.

You’re overwhelmed. You have an endless list of different tasks. Your schedule is stuffed with meetings and you need more time for focused work. Each meeting you attend brings more to-do items to your unending list. You’re busy throughout the day but can hardly point to something concrete you have done yourself.

If this describes you, you’re likely in a lead role or managing teams. This post is for you.

If you recently transitioned to a lead or manager role, you may start to feel like you’re not doing your job. That feeling of achievement you once derived from shipping tangible stuff disappears. At this same junction, some leaders slip back to doing IC work, like churning out features while ignoring the more significant part of their role –leading.

There are two things you need to understand as a new lead. You need a new approach to frame what you do. And you need a new way to prioritise them effectively.

As a software lead or manager:

• You will mostly be working through people (your team).

• You will do small little tasks but many instead of a whole big chunk of work.

• You will frequently context-switch throughout the day.

• The impact of some of the work you will do will be long-term, which will make it hard to get that instant feeling of progress. The effect of that extra email you sent to finally close a strong talent will not materialise instantly.

All of the above run counter-parallel to what you did as an individual contributor. Hence you will need to change how you view your work. Rather than what you contribute single-handedly, you need to reframe the way you see the organisation and your contribution to its success.

Your role is no longer about your individual contribution and your individual impact.

Your role is about what the team does, about how the team contributes to the company’s success. Those little boring tasks you do to make your team better and improve their efficiency now matter more than the features you can code alone – that is your job.

But you’re still drowning in tasks. So, how do you stay on top of your not-ending to-do list and still get things done?

The answer is simple: know how to pick your battles. You need to understand tasks you should do and tasks that others should do. You need to know what should be done in the moment and what needs to wait a little bit longer.

One of the mistakes I made in the past was making every task that came my way a task I must do. While a leader may be responsible, it does not mean that the leader should do everything. I was clearly missing “What I should do vs. what others should do.” I was equating being responsible to being the one doing it.

Being responsible for a task does not mean you should do it. There will be hundreds of things demanding your attention because the scope of your work has increased. It’s important to constantly differentiate what you should actually be doing from what others should be doing.

Once you find out what you should do vs. what others should do, you can delegate what others should do and focus on what you should do.

How to know what you should do vs. what others should do

A delegation matrix is an effective tool you can use to decide which tasks to delegate and which to retain. When you use this tool effectively, you’ll be able to free up some time to chase bigger things. A delegation matrix is a 2×2 table measuring the competence of reports on the y-axis and the competence of a manager on the x-axis.

Let’s look at a case where a delegation matrix can be applied. You have just received a request to review a new design document from another product team to ensure approaches are aligned. There are a few options you have.

• Spring into action to review and unblock the team while other things suffer.

• Add it to your overflowing to-do list and block this team for a couple of days or a week until some time is freed up for the review.

Using a delegation matrix, you may assess that a report on your team possesses the skills to review the design doc. Clearly, both you and your report can do the review. This task goes into quadrant 3 in the image above. Everything in quadrants 1 and 3 should be delegated to your reports.

There will be times when you have a task neither you or your report can do. Such tasks should either be delegated to your manager or solicit assistance from your manager to do it.

Everything that falls into Quadrant 4 should be done by you.

Tasks in quadrants 1 and 3 are:

1. Tasks that can be handled adequately by reports

2. Tasks for which team members have all the information for decision-making.

3. Tasks that don’t require skills unique to you or your position.

4. Tasks for which an individual other than you can have direct control over the task.

5. Tasks and/or projects that will contribute to the growth and development of the individual.

Tasks that fall in Quadrant 4

• The delegation process itself: Any work to be delegated should be delegated and explained by you.

• Performance evaluations and disciplinary actions: These are managerial responsibilities.

• Coaching, and moral problems.

• Planning and forecasting: Some of the detailed work can be done by engineers, such as breaking tasks down and estimations. However, you alone are in a position to decide team goals how they fit in with the overall company’s goals.

• Confidential tasks.

Separate urgent, important tasks from non-urgent tasks

Have you categorised your tasks into what you should do vs. what others should do and delegated what others should do, and yet you still feel like you’re drowning in assignments? Then, the next step is to separate urgent and important tasks from non-urgent and important tasks.

Do urgent and important tasks now.

Urgent and important tasks have clear deadlines and consequences if immediate action is not taken. Urgent tasks include responding to incidents, onboarding a new engineer, resolving conflicts, and unblocking a report.

Schedule important tasks that are not urgent.

Important but not urgent tasks are tasks without a specific deadline but they bring you closer to your goals. These are tasks that are easy to procrastinate on. Examples are defining your team’s strategy and vision. Schedule these tasks for the future. Put a block in your calendar in the future to do them.

Delegate urgent but unimportant tasks.

You may still find a couple of urgent but unimportant tasks in your todo list. There are studies that show that we’re naturally biased towards picking urgent tasks even if they’re less important. Tasks that are urgent but are not important are best delegated.

To conclude, if you’re going to take one thing from this post it should be being responsible doesn’t mean doing it yourself. Understanding what you should do vs. what others should do is key to staying on top of your schedule.

Additional resources

Five valuable lessons for a new tech lead
Feeling productive

Will Larson provided three ways to do this that I have found useful.

• First, you need to understand what’s happening on indirectly managed teams below you.

• Second, add things necessary for execution.

• Third, remove things getting in the way of execution.

The biggest puzzle to solve is understanding what is happening in the indirectly managed teams.

As one goes up in an organisation, the less the details you know about day-to-day jobs of people below and the unique challenges they face. An engineering director will have more knowledge about how each piece of work connects with teams under her purview but less knowledge about the codebase or the nitty-gritty of the environment frontline engineers work in and the impact it has on execution.

To draw an analogy, it’s like zooming in and out on maps. In a zoomed out mode, you see the big picture. When you’re in a zoomed in mode, you get a close look at select details. Frontline engineers have a zoomed in experience. And the higher you go in an organisation, the more zoomed out your view becomes. Folks at the top of an organisation tend to see and experience things in a broader sense than those who are layers away under them who may see things narrower but in-depth.

There are a few areas engineering leaders can look at to have visibility required to foster execution and remove roadblocks from frontline teams. That starts from finding a way to get quantitative and qualitative data that will help you:

• Understand how your teams are building what they’re building on time and within the budget you have.

• Understand what your teams have built are running at the desired level of performance and reliability and that it will continue to do so within the foreseeable future.

• Understand if the teams building that stuff are engaged and happy to continue building what they’re building.

• Understand if the users you’re building for are getting value and deriving satisfaction from what you’re building.

Understanding how your teams are building what they’re building.

Understanding how your teams build requires you to understand the process your teams take to build that thing they’re building. Let’s call this “process“. Process is the way your teams habitually do things. It is how your teams get work done on time and within budget. If you’re a Pizza company, process is how you turn dough into edible Pizzas delivered to consumers.

For a software company, your process is ideally your approach to building what you’re building.

If your organisation is like most organisations, your time and budget are finite. You always have to get value delivered to users within a certain budget and within a specific time. Due to this, you have to find an optimum way to optimise value delivery to users within the budget that you have.

To gain an understanding of how your team builds stuff and the possible roadblocks, take a look at where engineers spend their time, things that affect execution, how they collaborate, the tooling you have that enables them, the codebase, systems and architectures they work in.

• Does architecture/codebase allow teams to move as fast as possible?

• Do your teams have the right tooling?

• Do they have autonomy to spin up new services, build, scale and deploy them without hand-offs? How long does it take them to do this?

• Are your teams empowered to deliver values independently, own their backlog, ideate and figure out the most important thing to build?

• How do different functions within the teams work together? Do they have a shared understanding of what’s being built?

• Does everyone know where they fit in and what roles they need to play in what is being built.

• Do they have necessary tools

Answers to questions like these will point you in the right direction in figuring out what may be standing in the way of execution.

3 Ways to have the visibility required

You want to gain a better understanding of unique challenges your indirectly managed teams face? Include the following in your data gathering process.

1. Run developers experience surveys quarterly

Running surveys is one way to understand the experience your engineers have. Engineers’ development experience can reveal a ton about how they’re building what they’re building as well as what is standing in the way of getting things done.

I have seen engineering surveys centred on tools and technology only. Surveys like that often fall short. Sometimes the biggest source of roadblocks could be unrelated to tech or tools. A process or a social contract that has nothing to do with tech can be equally a source of friction for engineers.

I like the way C J Silverio puts it in this post.

If people are regularly doing any end-run around a process to get work done (say, regularly asking for rubber-stamp PRs so they can be unblocked), you have a process that’s not earning back its energy cost. Fix it.

A simple survey with questions like the ones below could reveal a ton about your engineering processes:

• How easy is it to find documentation and access documentation?

• What manual work can be automated that the team does?

• What process do you find valuable?

• What process would you improve?

• How satisfied are you with the tooling used daily e.g linters, IDE?

• What would you add or improve in our tooling?

If you’re looking for where to start, developer experience survey questions by Laura Tacho is a good one.

2. Hold 1-on-1s and skip-level 1-on-1s

When 1-on-1 is used effectively, it can be a platform for gathering information that allows you to know what’s going on in the team and with each engineer. 1-on-1 is a great time to offer your perspectives on what is going well and not going well. But it is also a good time to gather qualitative data, disproof or proof what you already know that is not going well.

If you’re leading from 2 or many layers away, hold skip-level 1-on-1s with frontline engineers.

• It will help you to gather information about how your managers are really doing beyond what they tell you.

• It will help you get a pulse on what’s happening on the front lines.

• It will help you learn where there is dysfunction, insufficient communication, or confusion within parts of your organisation.

3. Track DORA metrics

DORA metrics came to be after six years worth of surveys conducted by the DORA team and identified four metrics that elite-performing software teams use to measure their performance.

• Deployment Frequency—How often do you release code to production? How often do you deliver value to users?

• Lead Time for Changes— The amount of time it takes a commit to get into production. How long does it take to deliver value to users?

• Change Failure Rate— What is the percentage of this value that is defective?

• Time to Restore Service—How fast can we get back up when we fail? How long does it take an organisation to recover from a failure in production?

Measuring productivity metrics is a hard topic in software engineering because companies have attempted to measure developers’ productivity but ended up measuring the wrong thing. A few companies learned from this and abstained totally from ever measuring anything.
One thing is certain: developers’ productivity can not be reduced to a single metric or dimension. There are multiple dimensions to it.

Personally, I find DORA metrics effective because they can show you trends that allow you to ask crucial questions. When a deployment frequency trends downward, it puts a lot of questions you can ask your team.

One of my experiences with using DORA started at TIER. We were tracking how well we were doing WoW and MoM. Once every two weeks, I would sit down with Engineering Leads to discuss the metrics.

When we saw deployment frequency trending downwards or upwards, I would ask questions to understand what was going on. When we got better at putting code to production frequently, I wanted to know what changes we made.

In one of the review meetings with engineering leads, deployment frequency was trending downward. As I sat down with them to understand the root cause and how I could better support them, we saw that there were issues with our deployment infrastructure. The infrastructure team made changes to the base infrastructure that this team was yet to update to.

Most of the engineers in this team were new and had missed out on comms that had gone out earlier on why the team should upgrade. On finding the root cause, we were able to take a more actionable step, work together with the infrastructure team and get supports needed.

Additional resources

If you’re looking for more inputs? The following links are great resources.

• Reduce friction

• Skip Level Meetings: Everything You Need To Know About Skip Level 1 On 1s

• How engineering leaders can lead with visibility

How do I build a high-performing team?

This is one of the common questions managers ask themselves. No one wants a low-performing team. We all want to build effective and high-performing teams – a team that is highly interdependent, bound with a common goal, plans work, makes decisions, solves problems and delivers superior results.

But you can’t just pull a bunch of people together in a room and expect outstanding performance.

Building a high-performing team requires deliberate efforts. You have to create an environment that allows and breed outstanding performance.

Over the years, I have experienced underperforming and high-performing teams. I saw the kind of environment where high-performing teams thrive, things high-performing teams do differently and the culture that enables them.

But how do you create a high-performing team and what levers are there to pull?

There’s more than one way to skin a cat. But certainly, there are a few levers you can pull that will put your team on the way to superior performance.

Hire strong talents. Strong talents make a strong team.

You want to build a strong-performing team? Hire strong people. Why hire a strong senior when you can have 2 medior for the exact cost? Well, superior performance doesn’t come cheap.

Strong engineers are high achievers, trail blazers and catalysts for improvements. It’s worth the cost.

If you want to build a strong software team, hire strong people in product, data, design and engineering. Hire people who constantly challenge themselves and work to better or improve themselves.

Hiring a group of people and expecting outstanding performance doesn’t work. There have to be some thoughts behind the group composition and why they exist.

Usually, we often fail in two ways. Suppose we don’t fail at composing the right team by carefully hiring folks that bring complementary skills, we fail at defining why the team exists and ensuring every team member understands why they exist and why they’re doing what they’re doing.

Empower them

Have you hired a strong team? Coach, empower and get out of the way as necessary. A high-performing team is an empowered team that makes decisions within a defined boundary. They are able to make effective micro-decisions in their day-to-day work.

An engineering team that relies on a single decision-maker, albeit a product manager or an engineering manager is not empowered. Progress stalls when a team depends on a single person to make decisions.

Coach your team to the point that everyone knows what the expected outcome is. Coach them to a point where they know what they should be doing and what’s not.

An environment where one person is solely responsible for figuring out what to build, which is then handed over to the team creates a situation where a team fails to see why they’re doing what they’re doing. It becomes a game of “they ask us, we do it”.

Create synergy

Synergy is when two or more people work together to produce something of value. The more people you have, the more difficult it’s to create synergy – this is one of the reasons why the concept of two pizzas garnered popularity.

Synergy, at its core, is about team working, connecting and collaborating effectively together.

Communication and collaboration are key to maintaining high synergy. Folks having to chase information they need to do their work drains motivation and depletes trust between teams.

When communication breaks down, synergy suffers. And it could feel like every other person is out there to prevent you from doing your job. Often, it’s just a misalignment between functions, as rightly said in this post:

Every team in an org has its own pain points as well as its objective. While it seems that one department refuses to fix the problem of the other, it is also true that most people in the company work with good intentions. Hence, if everyone is trying to do a good job and they also complain about each other, the problem therefore lies in communication (or more specifically, communication breakdown).

If you want to build a high-performing team, pay attention to how different functions collaborate and the willingness of people to share information that benefits the wider group.

One of the best ways I have seen this play out is having a “default open” culture where ideas, information and feedback flow freely.

Create a culture of learning and improvements

Invented by psychologist Bruce Tuckman, a team goes through four stages of psychological development known as forming, storming, norming and performing. For a team to get to the performing stage, there has to be a lot of mistakes and learnings along the way.

A high-performing team is not formed in a day. High-performing teams go through periods when they’re low performing. It’s through mistakes and learning from those mistakes that make the team become better and better as days go by.

A team that reflects on how to become more effective and then tunes and adjusts its way of working gets better as time passes.

As Jocko and Leif stated in their book titled the dichotomy of leadership:

No team can deliver flawless performance. No one can achieve perfection. What makes the best team great is that when they make mistakes, they acknowledge them, take ownership and make corrections to upgrade their performance. With each iteration, they enhance its performance.

High-performing teams make incremental changes in processes (way of working) to improve quality and efficiency. They have a mindset that whatever is good today might not be good tomorrow. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

To build a culture of learning and improvement, you have to create an environment where people feel it’s okay to fail and not be afraid to voice out their shortcomings.

This is where psychological safety plays a crucial role.

Build psychology safety

Psychology safety is the belief that you have the freedom to speak your mind without being punished for that. Such belief is rooted in deep trust between peers and leadership. In a software team with high psychological safety, members are more open to taking risks, trying new ways and not afraid to fail. It’s this tolerance for mistakes and risks that allows the team to be more innovative, develop muscle to try new things, fail quickly and learn.

As a leader, it’s your responsibility to build an environment that doesn’t make people feel they can’t freely share their opinions but also an environment that lets them feel their opinions are welcome and you love to hear them.

In summary, building high-performing teams will take time to happen. It’s through deliberate effort but it’s worth it. To build high-performing teams, hire strong talents, empower them, create synergy and build a culture of learning and improvement.

You were not in control. You had no visibility: maybe there was a car in front of you, maybe not. –Alain Prost

As tech leaders, external factors, culture, and engineers’ personal life can affect delivery flow or your team’s ability to deliver on business outcomes and goals. You will need to equip yourself with data that allows you to get faster feedback to stay on top of these factors and maintain predictable and consistent delivery of value to users.

Whether you’re managing from 1 or 2 layers away, there are four main areas you need to get both quantitative and qualitative data from to have the visibility you need to lead your team effectively. I categorize these four areas you need to get data from as process, operation, people and product.

You need to get both quantitative and qualitative data that will help you:

• Understand how your team is building what they’re building on time and within the budget you have – Process

• Understand if what has been built is running at the desired level of performance and reliability and that it will continue to do so within the foreseeable future – Operation.

• Understand if the folks building that stuff are engaged and happy to continue building it – People.

• Understand if the folks (users) you’re building for are getting value and deriving satisfaction from what you’re building – Product

Process – Understanding how your team is building what they’re building

Process is the way you habitually do things. It is how your team gets work done on time and within budget. If you’re a Pizza company, process is how you turn dough into edible Pizzas delivered to consumers. As software company, your process is ideally your approach to building what you’re building.

If your organization is like most organizations, your time and budget is finite. I’m yet to see an organization that has enough of both. We always have to get value delivered within a certain budget and within a specific time – this calls for a balancing act.

One question most managers have is how their teams are building what they’re building within that time and the budget they have. To answer this question, you need visibility into where engineers spend their time and things that affect execution, like how collaboration works across different functions, the tooling you have that enables your team, the codebase, systems and architectures your engineers work in.

• Does architecture/codebase allow teams to move as fast as possible?

• Does your team have the right tooling?

• Do they have autonomy to start new services, build, scale and deploy them independently?

• Are your teams empowered to deliver values independently, own their backlog, ideate and figure out the most important thing to build?

• How do different functions within the team work together? Do they have a shared understanding of what’s being built? Does everyone know where they fit in and what roles they need to play in what is being built.

Answers to questions like these will point you in the right direction in figuring this out.

3 Ways to have the visibility required

Here are ways engineering leaders can have the visibility required.

Developers Experience/Productivity Survey

One of the ways to understand how your team is building what they’re building is through regular engineering surveys to understand engineers’ experience. What engineers experience while developing can reveal a ton about how they’re building what they’re building and where bottlenecks of pain points exist.

If your organization has a way of measuring employee engagement, you should still run surveys centered on developer experience. Organizational-wide engagement surveys are sometimes too general and may not effectively capture engineering experience.

A simple survey with questions like this could reveal a ton about engineering processes:

• How easy is it to find documentation and access documentation?

• What manual work can be automated that the team does?

• What process do you find valuable?

• What process would you improve?

• How satisfied are you with the tooling used daily e.g linters, IDE?

• What would you add or improve in our tooling?

When we ran some of these surveys at TIER, we realized that documentation and tooling were issues that needed to be addressed. We started investing in making documentation accessible and consolidating them. Along the way, we had to solve a challenge of making documentation close to the code base where engineers work.

If you’re looking for where to start, developer experience survey questions by Laura Tacho is a good one.

1-on-1s

When 1-on-1 is used effectively, it can be a platform for information that allows you to know what’s going on in the team and with each engineer individually.

When I first started having 1-on-1s, one thing I wanted to avoid was making it a meeting to get project updates and progress reports. In my first meeting with a report, I would set expectations of what the meeting was and was not about. One thing I always made clear was to refrain from talking about projects or getting project updates in the meeting.

With time, I realized that talking about projects was unavoidable to support my team effectively. As I listened to concerns from my teams, they always pointed back to the project they were working on.

If a report has an issue with another engineer on a team, it’s due to a project or task they’re working on. If a report is not getting what she wants from another team, it’s due to a task or project she’s working on.

So I realized it’s okay to talk about projects in 1-on-1s if there is an opportunity to unblock or coach a report.

You should use 1-on-1s to pick signals of things going well or not going well in your team. This is why 1-on-1s can be effective in helping you understand how your team is building what they’re building.

Here is an example of a conversation with a report.

*Manager: What do you think about how project XYZ is going?

Report: Well, it’s going fine except that we underestimated the project. There was ABC we didn’t think about or account for when we committed to the project. To do XYZ, we need to do ABC which will take 3 more weeks. In fact we don’t even know if it can be done yet. We’re currently undertaking tech discovery on how to do ABC.*

From the above conversation, this team is probably not investing in proper product and technical discovery before taking on projects if it becomes a pattern.

DORA Metrics

Measuring productivity metrics is a hard topic in software engineering. Historically, companies have attempted to measure developers’ productivity but ended up measuring the wrong thing. One thing is certain: developers’ productivity can not be reduced to a single metric or dimension. There are multiple dimensions to it.

One approach that has worked for me in the past is by measuring DORA metrics.

DORA metrics came to be after six years worth of surveys conducted by the DORA team and identified four metrics that elite-performing software teams use to measure their performance.

• Deployment Frequency—How often do you release code to production? How often do you deliver values to users?

• Lead Time for Changes— The amount of time it takes a commit to get into production. How long does it take to deliver value to users?

• Change Failure Rate— What is the percentage of this value that is defective?

• Time to Restore Service—How fast can we get back up when we fail? How long does it take an organization to recover from a failure in production?

I have found DORA metrics effective because they can show you trends that allow you to ask crucial questions. When a deployment frequency trends downward, it puts a lot of questions you can ask your team.

My teams at TIER adopted DORA metrics. We were tracking how well we were doing WoW and MoM. Once every two weeks, I would sit down with Engineering Leads to discuss the metrics.

When we saw deployment frequency trending downwards or upwards, I would ask questions to understand what was going on. When we got better at putting code to production frequently, I also wanted to know what changes we made.

In one of the review meetings with engineering leads, deployment frequency was trending downward. As I sat down with the them to understand the root cause and how I could better support them, we saw that there were issues with our deployment infrastructure. The infrastructure team made changes to the base infrastructure that this team was yet to update to.

Most of the engineers in this team were new and had missed out on comms that had gone out earlier on why the team should upgrade. On finding the root cause, we were able to take a more actionable step, work together with the infrastructure team and get the support needed.

Engineering onboarding entails communicating your company’s expectations, such as technical, process, product, cultural, and professional, to new hires in a way that makes the knowledge useful and practical.

The time it takes for a new engineer to be a productive member of a team is a function of how effective your onboarding process is.

I do realize there is a lot that goes into onboarding a new engineer. Usually, there are roles played by Procurement, HR team, and more. Usually, an HR onboarding will contain training programs designed to help new hires get acquainted with a company and its culture. The HR onboarding is usually generalized and may not include engineering onboarding. As a result, the focus of this post will be onboarding from the standpoint of a hiring manager.

Start from the hiring manager’s interview.

Engineering onboarding starts from the hiring manager’s interview. This is the first opportunity to help a candidate see her potential impact on the job. Don’t only assess the candidate’s fit during the hiring manager interview but also help the candidate connect to your team’s mission and purpose.

You’re doing onboarding work if you’re able to help a candidate understand what the role is about and how she can possibly fit in. The hiring manager’s interview can serve as an onboarding foundation that will be built on later if the candidate makes it to an offer stage.

Use the candidate’s notice period to your advantage

Candidates drop out at every stage of the hiring funnel. For a very strong talent, dropping out after signing an offer is a possibility. Some candidates have long notice periods. It’s important to keep the communication going.

You should leverage the notice period to continue to build on the candidate’s excitement about your company, your team, and how you work. My strategies include reaching out to them a week after an offer is signed to share our excitement in a mail.

As the notice period draws to a close, I reach out to see if they need any assistance and make use of the opportunity to send our onboarding guide to the candidate to prepare her ahead.

Have a comprehensive written down onboarding guide

One of the few elements of engineering onboarding that I found to be important when onboarding engineers is having a written down onboarding guide. PowerPoint presentations should not replace a written down employee onboarding guide that new hires can refer to from time to time. Nothing scales better than words written down because it’s always available when people aren’t.

Your onboarding guide should help candidates answer the following questions:

• What’s the team’s culture like?

• What is the company culture like?

• How does the team function or work?

• What are the tools used?

• How should access to tools be requested?

• Who are the people to meet?

• What are the company, product, and tech strategies?

• What are the engineering principles?

• Where to find documentation?

• What is the current team roadmap?

• Faqs and more.

Don’t leave engineers to figure out the tech part.

I had the opportunity to work with six different companies on three continents in the last ten years. Each of these companies has its own unique way of shipping software. They don’t all use the same techniques or approaches to software development.

Regardless of how experienced a new hire is, they should be onboarded into how you build software.

The “She is an engineer, and she will figure it out” will not work.

She needs to get familiar with your workflows and processes, and practices. She needs to know how things get shipped, what metrics are important, e.t.c.

On average, it’s estimated that it takes 3-5 months for a new engineer to be productive. A well-designed tech onboarding will ramp up new hires to a point where they become a productive member of the team.

The times it takes a new hire to be productive is a measure of how well your onboarding is.

An effective onboarding for engineers should not ignore tech. So, ensure your tech onboarding helps new hires understand:

• How software gets shipped

• What are the technical expectations from the role

• What are the non-technical expectations

• What is the release process like

• How to set up development environments

• What tooling is used

• Your code review process

• Your reliability metrics

• Incident management

• Existing and adopted ADR decisions

• What is the current state of your tech landscape, and what is the expected state?

Set onboarding and role expectations.

Without clear expectations, it’s impossible to drive accountability. Setting clear onboarding expectations helps new hires take responsibility for their own onboarding. It gives them a north star of where they’re headed.

When I onboard engineers, I break down onboarding expectations into 6 categories. Each category has a list of expectations that are required during onboarding.

• What is expected during the first week of onboarding

• What is expected from 2- 3 weeks of joining the team

• What is expected by the end of month 1

• What is expected by the end of month 3

• What is expected by month 6 and beyond

Conduct regular check-ins

When I joined Architrave, I relocated from Nigeria to Germany. Fortunately, I had an amazing people team and a fantastic manager who conducted a series of check-ins with me during my first few months.

The check-ins helped me receive early feedback and helped me navigate the change. The check-ins left a positive impression on me even after leaving the company. My onboarding period at Architrave made me realize the importance of frequent check-ins with new hires.

New hires may find it difficult to ask for support and may struggle silently. This makes it harder for managers to catch up and intervene quickly to get them back on track. By having regular check-ins with new hires, you’re creating an opportunity to ask crucial questions about how their onboarding is going. And how you could provide the support they need.

I follow structured check-ins with new hires as follows:

1. Check-in after 2 weeks of joining

2. Check-in after 30 days

3. Check-in after 60 days

4. Check-in after 90 days

Before going for onboarding check-ins with new hires, it’s important to have some feedback ready.

• Is the engineer doing the work you hired her to do?

• What is the new engineer doing well, and what can she change to make working with them easier?

• How confident are you that the engineer is going to be a top performer six months from now?

The importance of feedback cannot be overestimated and it is well captured in this post.

New hires will need frequent feedback to feel safe in their onboarding and understand if they are doing things right. This is both important for their self-confidence, lowers their stress levels, and makes sure they are indeed in the right direction.

Assign an onboarding buddy

Assigning an onboarding buddy to a new hire is a good way to provide an informal support system for the new hire.

It’s impossible to cover all new hires’ questions in the onboarding handbook or presentations.

When you assign an onboarding buddy, it allows the new hires to build a fast rapid connection with a co-engineer. And help the new hire get fast support for things not covered in the employee handbook.

Finally, obtain feedback and continuously use the feedback to improve your onboarding process.

But the moment you take a leap into a leadership role, moving beyond leading oneself to lead others, you start to realize that succeeding in this new venture takes more than being technically sound. Suddenly, there are non-technical skills that seem to matter.

Unfortunately, most people don’t get a manual handed over to them to help with the transition. In this post, we will look at five valuable lessons for a new tech lead. Whether you’re an individual contributor who is thinking (or in the process) of stepping into a tech leadership role, or a manager who is overseeing such transitions, this article is for you.

1. Your output is measured differently than you’re used to

Moving to a tech leadership role implies that you’ve done excellently well as an IC. You’ve earned your way to be a rockstar individual contributor, churning out features in the blink of an eye. But in this new role, what made you a good IC is now slightly different from what will make you a good tech lead. Realizing and adjusting to this new reality may feel strange at first. After a few weeks, or months especially, you may start feeling you’re not as productive as you were. This is because as an IC, you derived a sense of accomplishment from purely technical work such as writing code, refactoring an existing codebase, or optimizing an existing feature.

But as you step into your new role, your responsibilities diversify. You now have a responsibility to carve a technical path for your team. Your responsibilities now include having an overview of the entire system your team is building, establishing technical visions, and driving alignment.

Your role has changed, and how your output is (or will be) measured has changed too. It no longer depends on how much you can single-handedly deliver. Instead, it depends partly on what you do and partly on what you enable others (your team) to do. The quicker you’re able to tune your mind to this new reality, the greater your impact will be.

2. Aligning priorities is now part of your job

Aligning priorities means helping your team to focus on what’s most important and impactful. It means making sure your team’s efforts are all in sync with one another, supporting and reinforcing each other on the right thing.

As an IC, you were part of a team; you adopted the team’s goals and priorities. But as you become a tech lead, you start to take an active role in helping to shape your team’s priorities. You start to develop a wider understanding of your team’s goal and use that to decide what’s most urgent and important in terms of collaborating with other stakeholders.

There are always hundreds of tasks to be done, hundreds of features to build, experiments to run, and ideas to test. Being effective in this new journey requires a focus on value and impact and knowing how to choose which results to deliver.

You have to train yourself to look at every project or idea through this lens: how can we use 20% of our efforts to unlock 80% of our business goals, and how can I convince others and get them on board?

There won’t be enough time to build every idea. And there won’t be enough resources to create the ideal, ultra-polished product. The three variables you must now constantly play with are time, resources, and scope:

Figure 1: The triple constraints

Scope measures the work to be done. Resources include budget, team members, and everything at your disposal to execute the work to be done. Time refers to deadlines and how long you have to deliver on the work. You need to learn how to negotiate project scopes and requirements based on available time and resources.

3. How you communicate matters more

One of the most important skills to master is communicating effectively. Not all tech leads are good communicators, but all good tech leads communicate well.

Building software is a complicated process with numerous options and trade-offs that can easily be overlooked when communication is poor. As you step into a tech lead role, the way you communicate begins to have greater consequences and impact. Rather than simply making information available for people, you need to think carefully about what and how you communicate to people in order to achieve the desired outcome.

Sometimes, the curse of knowledge (where you develop a cognitive bias that makes it difficult to remember what it was like to be a novice) might kick in as you have access to broader information. You might think your team knows everything that you do or has the same context that you have on a project, when they don’t.

The effect of miscommunication could lead your team to veer away from building the right thing or making technical decisions that don’t align with the project’s direction, which can be expensive to fix later on.

You have to communicate and overcommunicate. Be creative about it at times; this could be as simple as creating a diagram or mock-up screen to pass the information across. When expressing complex ideas, it’s important to take a couple of minutes to think about how you can make it easy for your audience to understand. The worst mistake you can make is to assume people already know. Sometimes people forget things or are lost in other thoughts. So don’t be afraid to repeat yourself to get information to stick.

4. You should avoid premature fixes

As you step into your new role, you have to learn to wait for things to unfold in order to have a clear picture of what is going on before implementing a fix.

This can be very hard to grasp because it goes against our very instincts as engineers. As makers, we know what it means when we didn’t build to counter various edge cases that could make our application behave in unexpected ways. We know users of software can’t be trusted, so we have to think and anticipate problems in advance and put constraints to guard against them.

As you step from coding software for people to use, to leading people that code software for people to use, rushing to implement constraints to prevent or guard against problems you anticipate can sometimes have unintended consequences.

You can think of constraints as processes, practices, or guidelines enforced and intended to prevent certain problems or lead to certain outcomes.

While a lack of processes can affect your team’s performance and ultimately lead to the development of low-quality products, on other hand, too many processes created in anticipation of problems that are not yet clear can slow and hamper your team’s productivity.

‘The Master allows things to happen. She shapes events as they come. She steps out of the way and lets the Tao speak for itself.’ – The Daodejing, Lao Tzu

Instead of rushing in to create tons of processes right away, wait for problems to surface due to a lack of a process that fixes that problem. You’ll not only be able to craft a more appropriate solution, but you will also easily get your team’s buy-in, and collectively and collaboratively you will be able to define a better approach to fixing the problem.

Not only will your team see the need for it, but they will also understand why it is important. Since they’re involved in providing a solution, they will feel a sense of accountability.

5. You need to learn to manage projects

One important skill that feels very different from your previous role is dipping your feet into a bit of product management.

The Project Management Institute (PMI) defines project management as the ’use of specific knowledge, skills, tools, and techniques to deliver something of value to people.’ Because you and your team are responsible for delivering something of value (software) to users and stakeholders, you have an obligation towards your users as well as your stakeholders, which means you need to make sure work gets done.

There will be non-engineering bottlenecks that raise their heads to prevent you from delivering on time or block your team’s progress. These could include dependencies on people, corporate processes, or mismanaged expectations. Now it’s on you to identify and tackle these problems.

Now you need to think about planning and breaking tasks down. You need to inform stakeholders outside your team of what is going on in engineering and, likewise, inform your team of what is going on in other teams.

Management is now a part of your job. Rather than simply handing off anything that has to do with project management to project managers, you now need to work with them and compliment their efforts.

Reflections

The transition from IC to tech lead is never easy, and every journey is unique. There’s no manual and you’ll need to figure things out as you go. Take time to read, experiment, learn, and reflect. You will make mistakes, but what matters most is learning from them. That’s how you’ll grow as a leader.

Originally published on www.leaddev.com.

In this post, I’ll share some of these blog posts. I believe there are ideas and new learnings in them that will make you a better engineer or engineering manager.

Making the Most of These Posts

Frequently, I find myself in a situation where I could use some insights from what I’ve learned, and I completely forgot them. I’ve developed a process that helped me remember what I read and make them stick.

As you read through each post, I recommend doing the following to get the best out of each post.

1. Know what you’re reading. Read what the introduction is about and note down what you’ll learn at the end. This will prepare your mind for the insights to come.

2. Identify and jot down key ideas and information in the post. Write down learnings, insights, and ideas. Taking notes of what you learn prompts you to think about your learnings and also improves your memory.

3. Make your notes searchable.

4. Re-read your notes often.

Effective Communication Is Not About What You Say – Melanie Ensign

Communication is very crucial to leading effectively. But there is a clear difference between merely passing a message across and communicating effectively. The outcome of any communication is what determines how effective it is. Simply pushing out a message or publishing information is not enough to measure its impact. Effective communication is focused on what people need to hear in order to reach the outcome you desire and less on what you say.

You Are Your Own Best Hype Person – Marie Chatfield

No one lights a candle and puts it under a bushel. At some point in any engineer’s career, you’re going to have to prove to someone that you’re worth it: worth hiring, worth promoting, worth the raise, worth taking a risk on. The better you’re at making known what you do to others (that’s hyping yourself), the higher your chances of succeeding.

Don’t Lead By Example – James Coling

We’re often told to lead by example. Yes, we all need to set good examples for people to follow. Often time, we overstep our boundaries by continuously taking the lead without setting expectations leaving reports feeling crowded out or disempowered. This post helped me to understand that setting a good example is necessary, but isn’t sufficient for strong technical leadership. Acting like a tech lead means setting clear expectations and embracing direct communication.

Leading Without Managing – David Golden

Julie Zhuo, in her book, defines Leadership as being able to guide and influence other people. When most think of leading, they think of it as something that people in authority or engineers who wear manager’s or tech lead’s hats do. Contrary to that, anyone can be a leader, and it’s a skill anyone can learn. One of the key points I learned from this post was that anyone could lead without authority or coercive power and how to do just that.

Three Crucial Skills Leaders Must Develop to Become Executives – Nikhyl Singhal

“Executive skills are subtle and can be elusive to managers, demanding a great deal of focus, courage, and dedication. Becoming a great executive requires a set of skills beyond what’s needed to be a leader”. Nikhyl Singhalin, in this post, shares crucial skills that leaders must develop to become executives. Some of the key takeaways are:

• Learn to take risks and constantly put yourself in new, challenging, and increasingly ambiguous settings.

• Learn to build strong relationships with other senior people who work in entirely different functions.

• Learn to build teams and trust the people in them.

10 Admirable Attributes of a Great Technical Lead – Elye (Medium Paywall)

“It takes a lot of effort to be a great tech lead. It’s a delicate balancing act between two poles of the same attribute. If there is too much weight on one side, the person may fall”. This post outlines ten attributes that will make any tech lead a better one.

Being a Chief Technology Officer: Lessons Learned in My First Year – Shekhar Gulati

Most people are not lucky to have a manual handed over to them when stepping into a leadership role. As one climbs the ladder, one begins to notice what is required to succeed at the role is slightly different from what makes one an excellent individual contributor. One of the lessons I found valuable in this post includes picking your battles wisely and getting things done without actually doing them.

Building Mental Models of Ideas That Don’t Change – Hammad Khalid

For Individual Contributors and Engineering Managers, there is always new stuff to learn. If you’re at the beginning of your career, it can be overwhelming. One approach to stay above the water is finding a way to prioritise what to learn and a way of keeping track of principles or ideas that don’t change (that’s mental models) to make better and informed decisions. This post outlines a list of engineering principles and management principles, some of which took me years to learn.

A Collection of How-Tos, Templates, and Articles to Help You Be the Manager and Leader Your Team Needs

This link contains a collection of todos, how-tos, templates, and articles for engineering managers and tech leads. The categories covered include techniques and templates for:

• Conducting interviews

• Conducting meetings

• Building diversity and inclusion

• Recruiting

• Building Culture

• Growth and more.

11 Top Responsibilities and 10 Common Mistakes of a Technical Leader– Lorenzo Pasqualis

“Similar to building a Lego tower, building software requires making decisions from the beginning to the end. The first responsibility of a technical leader is to define the engineering reality: What needs to be built, the general technical direction, an example of the golden standard for productivity and technical excellence, the business and technological context, and the time and resource constraints.” If you’re a Tech Lead or an Engineering Manager, you’ll find this post interesting.

First Day Success Manual for New Managers – John Reh

One of the few ways to set yourself for failure when joining a new team is to start by criticizing the existing codebase, past practices and implementing tons of their processes from day one without having a clear picture of how things came to be. Criticizing past practices no matter how ineffective they seem is one of the many ways new hires set themselves up for failure. I love this post because it outlines how to prepare and make a really good impression, from meeting the team to attitude and culture when joining a new team.

Where To Go From Here: Join Software Leads’ Newsletter

To take your learnings further, Join the Software Leads’ newsletter, a monthly newsletter on software engineering and leadership.

Every month, I publish a new issue that includes links to interesting articles, use cases, and insights from experts on leading teams and solving common software engineering challenges at scale. Together, we learn and become better software engineers, better software leads, better engineering managers, and CTOs by making fewer mistakes as we learn from one another.

Building a good Application Programming Interface (API) is more than returning responses. Being a developer and having integrated with tons of APIs, I have noticed a pattern between successful API products and those that are not. It’s about solving problems with great affordance. One might ask the key to building good API products. In this post, I’ll provide some tips that I have found essential when doing so.

After landing my first software engineer role seven years ago, my first big project was building an online book marketplace. In the project, I integrated with multiple Application Programming Interfaces (API). From pre-filling book information using an International Standard Book Number (ISBN) to charging customer credit cards, integration with APIs was key to this project.

Working with APIs

Reflecting on my career, I have integrated, built, and exposed multiple APIs. With time, I began to notice two distinct patterns about APIs. Some have certain characteristics that make developers fall in love with them. There is something about them that draws you in, and you feel like drawing others in too. You just want to tell or recommend it to others at no cost.

Likewise, the universe is not also free of APIs that make you feel otherwise. They appear not to live up to their promises. It’s either the documentation provides a generic list of actions you can perform with the API, or it’s hard to understand how the API can solve your problem. Sometimes, If you manage to scale through the documentation hurdle without issues, the implementation is lurking around the corner to bite you. If you overcome the implementation difficulties, security or support are just around the corner waiting to pounce on you.

Guidelines and Categories

I realized there are two categories of APIs: well-designed APIs and poorly designed ones. Until a particular time in my career, I assumed poorly designed APIs were a function of developers who built them. While this may be true in some cases, I have also learned that the fate of an API is not decided during the development phase only, it starts from the time you conceive an idea that needs to expose some interfaces and beyond. I like to see APIs as not just some mere interface or endpoint, at least in this post. I prefer to think of them as products that expose some interfaces and solve a problem irrespective of the technology or protocols. Therefore I will refer to APIs as API products interchangeably.

While there are so many API guidelines and best practices for implementing them, most are geared towards code implementation details. Little has been said on the business side of things. It’s important to note that well-designed API products ride on a rock-solid business strategy.

The API growth rate is increasing at an explosive rate. According to a recent Postman API report, between January 2019 and January 2020, there was an increase of more than 100% in API growth. This growth has been mainly due to the significant shift in the tech industry and consumer behavior — the rise in cloud adoption, the increase in microservice architecture adoption, and consumers’ love for multi-device experience.

To put these in a better way:

• Consumers are moving from a single to multi-device experience, which drives a need for systems to be integrated.

• Businesses are moving away from the monolithic way of building applications to a more decentralized architecture.

• Organizations are shifting from on-premise data-center to the cloud.

The tech industry’s significant shifts and consumer behavior further buttress that the API economy has come to stay, and businesses benefiting the most tend to create API products that developers love to use.

Every successful product either solves a problem or makes solving a problem easy. Such is true with API products. Good API products are sufficiently capable of solving the problem they’re designed to solve without compromising availability and security. They are easy to use, easy to evolve, and hard to misuse.

Understand the “Why” and “How”

It’s not too uncommon to see APIs that no one understands the problems they are trying to solve. Undeniably, API products are no different from other great products. Great products have a rock-solid strategy — the “why” and “how” are clearly defined and understood. Before you set out to build an API, you first need to ask yourself: why do I want to build an API? What problem will the API solve or make it easy to solve? How will the API help me achieve my goals? If you’re unable to understand the business needs for an API, you should engage the right people and figure them out before designing it. Your strategy will transverse down from the business to implementation details. When your strategy is clear, It will challenge developers to think creatively in achieving the end goals.

Design Matters

Mehdi Medjaoui, in his book Continuous API Management, describes the design as something you do when you make decisions about how something you’re creating will look, feel, and be used. Each time you add or update an API, you’re actively making design decisions for better or worse. This is why you have to make a deliberate effort at the design stage — you can’t rush it.

When designing your API, you should pay attention to the following:

• Vocabularies: Are your words and terms easy to understand for your users?

• Styles: What protocols are you supporting, Rest, or GraphQL?

• Naturalness: Do your users have to change their usual ways of solving their problems significantly? Did you follow established standards and conventions?

• Consistency: What level of familiarity will you provide? Are your APIs similar to what your users may have used in the past?

Get Good at Documentation

Documentation is an important aspect of building API products that people love to use. The year 2020 state of the API report showed that documentation is the highest obstacle to consuming APIs. If you’re a developer, you understand how frustrating a lack of sufficient documentation can be when integrating with APIs. Be aware that people will read your API documentation for three primary reasons: evaluation, integration, and debugging.

Evaluation is what people do when they are trying to see if your API product solves their problem. Once your API is evaluated, users will need your documentation for integration. Once your API is embedded in their application, they will visit it when something goes wrong.

If you’re providing only one type of documentation, you’re undeserving your users. Teams must understand that documentation isn’t just a “dump” of API parameters. There are marketing and customer support aspects that should be included. Besides your endpoints, errors, requests, and response structures, your documentation should also tell the story, how you see the problems, and your solutions to solve those problems.

Get Development Right

This is one of the most important phases in API development. This is where your ideas, strategies are turned into implementations. The development phase contains many vital decisions that are hard to reverse if not made right. You will need to make some internal decisions like what technologies to use, protocols, runtimes, and where to run the software that powers your API. End users don’t care whether you use NoSQL or MySQL as your datastore. What’s important when you make technology decisions is that you choose a technology your team can support.

The quality of tech decisions you make transcends what is needed to launch your product’s first version. Launching the first version of your API is one part of the job. You have to make decisions about your API’s quality, reliability, changeability, and maintainability over a lifetime. API performance and uptime are considered to be some of the major determinants of whether an API product meets consumers’ expectations. This sheds light on how consumers will assess your API.

Tracking Effectiveness

You also have to decide what to test and how you will test it to build confidence in your API. In summary, your tests should be able to cover the following:

• Can the API Product deliver on the strategic goals?

• Is the API Product quality enough to support the strategic goals?

APIs are meant to be simple, they hide complexities and do not expose them. I like to compare APIs to an electric plug — simple to use without any complicated manuals — an electric plug does not care about the source of electricity, neither does it care about the change over switch installed or the electricity provider. It’s a conduit for electricity–plug, and it just works. Admittedly, modern systems are complex, but the complexity should not leak to the interfaces.

In Conclusion

Building a good API is more than accepting requests and returning responses. That’s far from the goal. It’s about solving problems with great affordance. You have to align usability with purpose continuously. It’s important to note that people will use your API differently. There will be different problems people use your API to solve. You can’t change that. What you can not do is be everything to everyone.

Finally, you have to go out there and let developers understand your products’ benefits by connecting with them at various events. You have to let them see how your products make their lives easier. The tech community is competitive. It’s no longer a matter of if you build it, they will come. But instead, build, take it to them and beyond.

Originally published by me on www.mindtheproduct.com, [03/02/2021].

Like numerous engineers, I’ve built a working code on my machine that refused to then run in other environments due to the disparity of environments, that is, different configurations and dependencies.

Editorial note: This post was initially published on Iamondemand’s blog. You can check out the original here, at their site.

Docker defines containers as “a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.”

By bundling an entire runtime environment, including an application plus all its configurations, binary files, dependencies, and libraries needed to run it into a single image, container technology eliminates the “but it runs on my local machine” problem and allows you to adopt the “build once, deploy anywhere” principle.

Containers provide a consistent deployment environment that development teams can use across software development and delivery pipelines.

The demand today for tolerant, reliable, and scalable software has given rise to thousands of containers running in a typical production environment, which brings a new problem: management. Container orchestration tools like Kubernetes solve this issue.

Kubernetes lets you run and manage containers and deploy stateful and stateless applications with multiple cloud providers. With Kubernetes, you can run a single application across a fleet of machines or different computing environments. Furthermore, you can either deploy your applications to a Kubernetes cluster in the cloud or on-premises. Of course, the major challenge with cloud deployments is selecting the right provider out of the many out there that offer managed Kubernetes as a service.

Before making a decision, it’s important you examine each service provider to identify which one best suits your organization’s needs. In this post, I’ll discuss the top Kubernetes as a Service offerings, their similarities, key differences, and how to choose the right one.

How to Choose the Right Kubernetes Provider

Most cloud providers have their own set of Kubernetes hosting environments for managing containers. Although Kubernetes offerings vary wildly among providers, here are some factors to consider as you shop for the right platform to manage your container workloads:

Data Security

Before selecting a Kubernetes service provider, you should understand each provider’s security governance process, measures, and mechanisms for preserving your data and application. An ideal Kubernetes service provider should align with strict security compliance and industry best practices. It should also implement some sort of confidentiality for sensitive data and provide administrative rights for restricting, monitoring, blocking, and approving users’ access to data.

Performance and Reliability

Reliability is an important requirement of any application running in the cloud. You should ascertain the reliability of cloud service providers by comparing their performance against their service-level agreements for the last 6 to 12 months. Cloud service providers usually publish this information, and even if they don’t, you can always request it.

You shouldn’t expect 100% perfection because every provider will experience downtime at some point. A good example was the recent Google incident. What matters the most is how often downtimes happen and how a provider handles them when they do occur. An ideal Kubernetes service provider should have documented, proven, and established processes for handling unplanned and planned downtime.

Standards and Certifications

Before going forward with a Kubernetes provider, you need to ensure that the provider adheres to the best industry practices and recognized standards. It’d be helpful if you also understood how the provider plans to continuously adhere to these standards. An ideal Kubernetes provider should have good knowledge management, effective data management, service status visibility, and structured processes.

Now that you’re familiar with the criteria to consider while searching for the right Kubernetes Managed Service Provider, the next step is to identify the market’s biggest players.

Top Options for Kubernetes as a Service

There are many cloud services for running container workloads using Kubernetes, but let’s closely examine the Kubernetes solution from the Big 3 players: Amazon’s Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Microsoft’s Azure Kubernetes Service (AKS).

Amazon Elastic Kubernetes Service

Amazon Elastic Kubernetes Service (EKS) is AWS’s managed container offering that allows developers to start, run, and scale Kubernetes apps on-premises or in the AWS cloud. EKS also allows you to plan, schedule, and execute your container workloads across compute services and features available on AWS, such as Spot Instances, Amazon Fargate, and EC2.

With Amazon EKS, you can easily manage your applications and Kubernetes clusters across hybrid environments too, plus you can use the Kubernetes jobs to run parallel or sequential batch workloads on your EKS cluster. Common use cases for Amazon EKS include hybrid development, batch processing, machine learning, and web applications. The fee charged by AWS is $0.10/hour for each EKS cluster created.

Strengths

• Highly configurable and customizable

• Efficiently provisions and scales your Kubernetes application

Limitations

• Complexity involved with addition and customization of node pools

• No automatic node updates

Google Kubernetes Engine (GKE)

Google Kubernetes Engine (GKE) is Google’s managed offering that allows you to deploy, manage, and scale containerized applications. The GKE environment comprises multiple Compute Engine instances grouped together to form clusters that are powered by Kubernetes. By running a GKE cluster, you get to benefit from Google’s advanced cluster management features, including automatic upgrades and scaling, logging and monitoring, node auto-repair, load-balancing, and more.

With Google Kubernetes Engine, you can create clusters based on your budget and the availability requirements of your workload. The two available clusters include: regional and zonal (multi-zonal and single-zonal). GKE has a financially backed Service Level Agreement that guarantees 99.5% availability and 99.95% availability for zonal clustersand regional clusters, respectively. Just like Amazon, Google charges $0.10/hour for every GKE cluster that you create, although there is no charge for Anthos GKE Clusters.

Strengths

• Easy Cluster creation and management

• Automatic cluster patching and upgrades

Limitations

• Difficulty in customizing server configurations

• No adequate documentation for supportive information

Microsoft Azure Kubernetes Service (AKS)

Azure Kubernetes Service (AKS) is a highly secure, available, and fully managed Kubernetes service that allows you to deploy and manage containerized apps more easily. AKS offers an integrated CI/CD experience, serverless Kubernetes, enterprise-grade governance, and security. Your DevOps team can leverage AKS to build, deliver, and scale applications fast.

AKS has a financially backed Service Level Agreement that guarantees 99.95% availability for Kubernetes clusters using Azure Availability Zone and 99.9% availability for Kubernetes clusters that don’t. Unlike other providers, Azure doesn’t charge for Kubernetes cluster management. However, you pay for the resources you use, like networking, storage resources, and virtual machine instances.

Strengths

• Faster end-to-end development experience via Azure DevOps, Kubernetes tools, Visual Studio Code, and Azure Monitor

• Advanced access and identity management using Azure Active Directory

Limitations

• No automatic node updates

• Inability to update availability zone settings after a cluster is created

Comparison Criteria

Google, Amazon, and Microsoft Azure all offer Managed Kubernetes solutions, but you’re at liberty to select the provider whose offerings meet your organization’s needs and requirements. If you don’t want to use EKS, AKS, and GKE, the three cloud providers do have other alternatives that you can use to run container workloads.

Alternative to Kubernetes for Running Container Workloads

There are two ways to run container workloads on the cloud: using managed Kubernetes services like AKS, GKE, EKS or by leveraging the serverless container services provided by the cloud providers. If you don’t plan to run on the cloud, you can also adopt a self-hosted solution like OpenShift to run container workloads.

By implementing serverless container services, you can manage the life cycle and availability of your container workloads. Examples of serverless container services include Google Cloud Run, AWS Fargate, and Azure Container Services.

AWS Fargate

AWS Fargate is a serverless compute engine that lets you run containers without the need to provision, configure, or scale clusters of your virtual machines. With Fargate, you don’t have to choose server types, decide when to optimize cluster packing, or scale your cluster. Development teams can thus focus on building and operating applications while Fargate manages all the infrastructure and scaling needed to run the app with high availability.

Google Cloud Run

Cloud Run is a serverless compute platform that allows you to build and deploy highly scalable applications without the need for Kubernetes-based or VM deployments. With Cloud Run, you can build applications with your favorite tools/dependencies and in your favorite language and deploy the apps in seconds.

Cloud Run offers several benefits, including auto-scaling, no up-front provisioning, and zero server management. It’s ideal for use cases like mobile backends, web applications, stateless HTTP applications, streaming, batch data processing, etc.

Azure Container Instances

Azure Container Instances (ACI) is a serverless compute platform that enables you to easily run containers without having to handle the management of the virtual servers. ACI lets you deploy containers in the cloud with remarkable speed and simplicity. It also enables you to secure applications with hypervisor isolation.

By running your application in Azure Container Instances, you can focus on developing applications without worrying about managing the infrastructure that runs the app. ACI is suitable for data-processing jobs, building event-driven applications, and elastic bursting.

Wrapping Up

Containers solve the problem associated with the multiple, different environments that development teams must deal with today, letting them build and deploy the same application for development, testing, and production environments. And Kubernetes allows you to orchestrate these containers.

Although there are several cloud service providers for your container workload orchestration, you need to compare what each provider brings to the table. By understanding the strengths and limitations of the various managed Kubernetes services, you’ll be well positioned to select the right provider that perfectly suits your organization’s needs.

Editorial note: This post was initially published on Iamondemand’s blog. You can check out the original here, at their site.

When your systems are distributed, various things can go wrong. Even if you’ve written the perfect code, a node may fail, a connection may timeout, or participant servers may act arbitrarily. The bottom line is that things will break. And when they do, you want to be able to identify and fix the problem as soon as possible before it alters the entire system’s performance, or affects customers or your organization’s reputation. For this reason, we need observability to run today’s services and infrastructure.

People have varying knowledge of what observability means. For some engineers, it’s the old wine of monitoring in a pristine bottle. For others, it’s an umbrella concept that includes log analysis, trace analysis for distributed systems, visualization, and alerts management.

Honeycomb, in its Guide to Achieving Observability, also defines observability as the ability to ask arbitrary questions about your production environment without having to know beforehand what you wanted to ask. Despite the variability in these definitions, they both explain the overarching goal of observability, which is to achieve better, unprecedented visibility into systems.

Observability is a system that enables you to understand what’s really happening in your software, from the outside. An observable system provides all the information you need in real time to address the day-to-day questions you might have about a system. It also enables you to navigate from effect to cause whenever the system develops a fault.

An effective observability solution may address questions like:

• Why is “y” broken?
• What went wrong during the release of feature “x”?
• Why has system performance degraded over the past few months?
• What did my service look like at point “y”?
• Is this system issue affecting specific users or all of them?

You may ask why you should put in the effort to make your systems observable. I discuss the reasons in the following sections.

The Unpredictability of Distributed Systems

A distributed system comprises multiple components located on different networked computers, communicating and coordinating their actions by passing messages from one to another. The inherent integrations and nature of distributed systems lead to “layers of distinct ownership,” which are sometimes challenging to manage. By implementing observability across a development environment, you’re able to understand your system’s failure modes and trace issues to their root cause.

The Need to Always Maintain Software

After an application is designed, implemented, and launched, it also has to be maintained. Development teams need to continuously adapt to changing customer behavior and ensure that the application runs at peak performance levels and works as expected. An observable system makes application maintenance easier. You’re able to easily identify errors, fix bugs that might arise, customize the application to users’ needs, and eventually improve the app’s performance.

The Need to Speed Up Code Development and Deployment

DevOps is about speed: faster application development and shipments, more immediate updates, and continuous development—all of which lead to a shortened development lifecycle and provide faster and continuous delivery with high software quality. If your development teams cannot identify and address issues before they occur, or they cannot react quickly to changes, it may be difficult to speed up the time to market. By leveraging a strong observability platform, software development teams can increase speed and effectiveness in deploying code, updating, and tracking changes.

In a nutshell, observability offers many benefits to managers and engineers. It also plays a vital role in an organization—especially in maintaining and enhancing a system’s overall architecture and performance. As distributed systems become more complex, engineers can infer the system’s internal states through externally available knowledge, including tracing data, log data, monitoring data, and so on.

Pillars of Observability

Observability is about instrumenting systems to gather useful data that tells you when and why systems are behaving in a certain way. Whenever a system develops a fault, you should first examine telemetry data to better understand why the error occurred.

The following telemetry data—commonly referred to as the pillars of observability—are key to achieving observability in distributed systems:

• Logs
• Metrics
• Tracing

Figure 1: Three pillars of observability

Logs

Logs are structured and unstructured lines of text a system produces when certain codes run. In general, you can think of a log as a record of an event that happened within an application. Logs help uncover unpredictable and emergent behaviors exhibited by components of microservices architecture.

They’re easy to generate and instrument. And most application frameworks, libraries, and languages come with support for logging. Virtually every component of a distributed system generates logs of actions and events at any point.

Log files provide comprehensive system details, such as a fault, and the specific time when the fault occurred. By analyzing the logs, you can troubleshoot your code and identify where and why the error occurred. Logs are also useful for troubleshooting security incidents in load balancers, caches, and databases.

Metrics

Metrics are a numerical representation of data that can be used to determine a service or component’s overall behavior over time. Metrics comprise a set of attributes (e.g., name, value, label, and timestamp) that convey information about SLAs, SLOs, and SLIs.

Unlike an event log, which records specific events, metrics are a measured value derived from system performance. Metrics are real time-savers because you can easily correlate them across infrastructure components to get a holistic view of system health and performance. They also enable easier querying and longer retention of data.

You can gather metrics on system uptime, response time, the number of requests per second, and how much processing power or memory an application is using, for example. Typically, SREs and ops engineers use metrics to trigger alerts whenever a system value goes above a specified threshold.

For example, let’s say you want to monitor the requests per second in an HTTP service. You notice an abrupt spike in traffic, and you want to know what’s happening in your system. Metrics provide deeper visibility and insight that help you understand the cause of the spike. The spike could be due to an incorrect service configuration, malicious behavior, or issues with other parts of your system. In addition to providing visibility, you can also use the information to detect and determine the severity of issues.

Trace

Although logs and metrics might be adequate for understanding individual system behavior and performance, they rarely provide helpful information for understanding the lifetime of a request in a distributed system. To view and understand the entire lifecycle of a request or action across several systems, you need another observability technique called tracing.

A trace represents the entire journey of a request or action as it moves through all the nodes of a distributed system. Traces allow you to profile and observe systems, especially containerized applications, serverless architectures, or microservices architecture. By analyzing trace data, you and your team can measure overall system health, pinpoint bottlenecks, identify and resolve issues faster, and prioritize high-value areas for optimization and improvements.

Traces are an essential pillar of observability because they provide context for the other components of observability. For instance, you can analyze a trace to identify the most valuable metrics based on what you’re trying to accomplish, or the logs relevant to the issue you’re trying to troubleshoot.

Tracing is better suited for debugging and monitoring complex applications that contend for resources (e.g., a mutex, disk, or network) in a nontrivial manner. Tracing provides quick answers to the following questions in distributed software environments:

• Which services have inefficient or problematic code that should be prioritized for optimization?
• How is the health and performance of services that make up a distributed architecture?
• What are the performance bottlenecks that could affect the overall end-user experience?

Although logs, traces, and metrics each serve their own unique purpose, they all work together to help you better understand the performance and behavior of distributed systems. If your organization already uses microservice or serverless architecture, or if they plan to adopt containers and microservices, a combination of all the telemetry data will provide the detailed information needed to understand and debug your system.

Implementing Observability

The fundamental process of implementing an observability-centric culture across your development environment relies on adopting tools that enable you to support and derive insights from the pillars of observability outlined above.

First, you need to select a modern observability platform that perfectly fits your specific needs. A full-stack platform that condenses all the telemetry data from metrics, logs, and traces into a visual, intuitive, and easy-to-understand dashboard can be a good starting point. (Just be careful not to choose a platform that locks you into a specific cloud platform.)

Then, you need to identify and monitor metrics related to issues you’ve already experienced and those you could likely encounter in the future.

After integrating observability into your incident management process, the next step is to establish an observability-centric culture across your organization. Absent of this, you won’t be able to get the best out of tracing, metrics, and log management. No amount of observability tooling can serve as a substitute for sound engineering instincts and intuition.

By selecting the right observability platform, getting a connected view of your system’s performance data in a singular view, and establishing an observability culture, you’ll be able to identify issues faster, understand what caused them, and eventually, build customer-focused products at greater speeds.

Conclusion

As the adoption of microservices and containers increases, engineers and managers need to develop a culture of observability. Doing so will help your teams get the best out of your investments in the cloud. It will help you form a continuous innovation culture that delivers high-end software to customers.

It doesn’t matter if you’re a seasoned DevOps pro or just getting started with observability. Understanding the three pillars of observability is an essential part of developing an observability-centric culture. Practice what you’ve learned in this post, and you’ll be on your way to achieving observability in distributed systems.

Editorial note: This post was initially published on Iamondemand’s blog. You can check out the original here, at their site.

Machine Learning in the Cloud

Cloud-based machine learning enables organizations to access high-performance resources and infrastructures that they couldn’t use (or even afford) on their own. Today, cloud use involves networking, storage, and computing, but when cloud computing is combined with ML to form the “intelligent cloud,” its capabilities increase and it is much easier to protect, scale, and handle. With the great strides taking place in cloud and machine learning development, their future seems ever-more tied together.

Although cloud computing is not new, some organizations have yet to embrace it. The question that comes to mind is: What do you gain by running ML workloads in the cloud? Below are some answers.

Reduced Costs

Hosting your learning workloads on the cloud will save you a considerable amount in capital costs because you won’t need any physical hardware investments.

Also, you won’t need to hire professional personnel to maintain the hardware, since cloud service providers buy and manage the hardware equipment themselves.

Availability

In an on-premise work environment, anything can happen. Things break apart: processors burn, hard drives get corrupted, and networks get interrupted.

The cloud aims to deliver abstract computing space, memory, and power, making you forget about the challenges of the physical world. Fortunately, most cloud service providers are reliable and maintain 99.9% uptime, so your Machine Learning team or engineers can always get their work done on the cloud. And of course, you can access ML services on the cloud from anywhere at any time.

Elasticity

Often, businesses don’t predict their future needs correctly and purchase more hardware than they need or else don’t have enough servers to handle computing tasks. Sadly, as they try to correct their errors, the process becomes more and more expensive.

The cloud enables businesses to experiment with different ML technologies and scale up or down as and when needed. With nominal monthly fees, they can expand capacity.

Of course, the resources of every cloud service provider are limited, but they’re still far beyond what 99% of businesses will need. A good cloud computing system has many different tools, bandwidths, and storage space options.

On-Demand Self-Service

When machine learning workloads are hosted in the cloud, you can access them at any time without administrator approval, as you would need with on-premise data centers. Cloud computing achieves this through automation and self-service.

On-demand self-service also helps businesses to plan. For instance, you can request new virtual machines or storage capacities at any time and expect to have them within seconds.

Broad Network Access

With machine learning workloads hosted in the cloud, you’ll have unlimited data access privileges. Machine learning uses data as its input for smarter decision-making and improved performance, and cloud environments can hold massive volumes of data. That means that using ML in the cloud will help you overcome the delay in traditional silos and accessibility. ML and AI are also great instruments for transferring data between the on-premise infrastructure and cloud environment.

Amazon SageMaker Cloud ML Service

Amazon SageMaker is a fully-managed service that helps data scientists, analysts, and software developers to build, train, and deploy machine learning models on the cloud.

SageMaker provides an integrated Jupyter notebook environment for easy, seamless access to your data sources for analysis and exploration, as well as optimized ML algorithms that can run effectively against a huge amount of data in a distributed environment. SageMaker helps to remove any obstacle that might prevent you from building ML solutions or cause a delay.

How SageMaker Works

SageMaker has a three-step process that simplifies machine learning modelling. Let’s take a look:

Figure 1: AWS SageMaker: Build, train, deploy

Build

SageMaker has a completely integrated development environment you can leverage to build production-ready ML models. With its hosted and easy-to-use Jupyter notebooks, you can visualize and explore data stored in Amazon S3. Besides, SageMaker also comes installed with tons of optimized ML algorithms.

Train

You can use SageMaker’s Experiments to organize, track, assess, and compare every iteration to ML models. You can train your ML model on a managed infrastructure by making a simple API call or via a single click. With the help of its automatic model tuning capabilities, you can select the best set of hyperparameters from your preferred algorithm.

Deploy

After model training and hyperparameter tuning, your next step is to deploy the ML model to generate predictions for batch data or new data. Once the model is deployed, it will be hosted on auto-scaling Amazon ML instances across several availability zones for high availability and performance.

Amazon SageMaker Features

SageMaker has many useful features that make building, training, and deployment of ML models easy for users. These include the following:

Inbuilt Machine Learning Algorithms

Amazon SageMaker comes installed with many ML algorithms that you can use to train big datasets. These algorithms are always available and optimized for accuracy, scale, and speed.

SageMaker includes supervised ML algorithms such as logistic/linear regression and XGBoost. These algorithms are used to solve time series prediction or recommendation problems. SageMaker also includes support for unsupervised learning, such as principal component analysis and k-means. These unsupervised algorithms are used to address problems like clustering and customer segmentation.

End-to-End Machine Learning Platform

An end-to-end machine learning platform is a platform designed to speed up modeling and deployment and ensure scalability and reliability in production. Amazon SageMaker supports the end-to-end lifecycle of ML applications, from data collection to model building, training, deployment, and scaling.

Zero Setup

Research shows that there’s an increasing demand for ML, and the future of ML is promising. Yet according to a 2018 US Census Bureau survey of nearly 600,000 businesses, a mere 2.8% were using machine learning, and only 8.9% were using any form of AI. This shows that many enterprises are facing a crucial knowledge gap—but the cloud offers a solution.

Amazon SageMaker lets you implement ML features without some of the operational overheads experienced with on-premise setup. SageMaker also provides the APIs and SDKs, freeing you from responsibility for setup, and you can embed ML functionalities on the go.

Pay for What You Use

Using the cloud gives you cost-effective resourcing and increased agility. With AWS SageMaker, you pay only for resources that you use. When building, training, and deploying your ML models on SageMaker, you’ll be billed by the second, with no upfront commitments and no minimum fees. Pricing is broken down by on-demand ML storage, ML instances, and data processing.

The pay-as-you-use model is perfect for bursting ML workloads and helps you leverage the power and speed of GPUs for training without incurring the costs of hardware investment.

Flexible Model Training

ML model development can be an iterative, expensive, and complicated process. Also, model training jobs can take anywhere from minutes to hours or even days. With SageMaker, you can efficiently train and build ML models at scale. This means you can experiment with ML capabilities and services and automatically scale based on resource demands.

Training Your First Model on Sagemaker

On a cool Monday morning, you get an email from your boss saying, “Our focus for the next quarter is building an application that predicts whether a customer will sign up for a Certificate of Deposit (CD).” Our target customers are people who are looking to invest their capital. Your job is to build a machine learning model capable of predicting if a customer will enroll, given the features and the data from the marketing team. This is a typical end-to-end ML project that you can efficiently build and deploy on SageMaker. In this section, we’ll learn how to get started with SageMaker and to train and deploy ML models on it.

Note: To use the service, you’ll need to create an AWS account.

Now let’s dive right in.

Step 1: Create a Notebook Instance

1. Log into the AWS Management console, type “SageMaker” in the search bar and select it.

2. In the SageMaker console, click “Notebook instances,” then “Create notebook instance.”

1. Fill in the required information (like Notebook instance name or Notebook instance type), then click “Create notebook instance.”

1. When the server status changes from “Pending” to “Inservice,” click “Open Jupyter.”

Step 2: Load, Visualize, and Prepare Data

1. Once Jupyter opens, click “New” and select “Conda Python 3.”

1. Import the relevant libraries, define some environment variables in the notebook environment (as shown below), and run the cell.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import boto3, re, sys, math, json, os, sagemaker, urllib.request
from sagemaker import get_execution_role
from IPython.display import Image
from IPython.display import display
from time import gmtime, strftime
from sagemaker.predictor import csv_serializer

# Define the IAM role 
role = get_execution_role()
prefix = "sagemaker/DEMO-xgboost-dm"
containers = {"eu-west-1": "685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:latest",
              "us-west-2": "433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest",
              "us-east-1": "811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest",
              "us-east-2": "825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest"
              }
# every region has its XGBoost container 
my_region = boto3.session.Session().region_name
# set the region of the instance
print("Success - the MySageMakerInstance is in the " + my_region + " region. You will use the " + containers[
    my_region] + " container for your SageMaker endpoint.")

1. Create an S3 bucket. The training data and model artifacts will be saved in the bucket. (In the screen capture below, the bucket name is “awsexperimentbucket1000.”)

If the S3 bucket is created successfully, your code will run without any errors (as shown below).

bucket_name = 'awsexperimentbucket1000' # you can always use any name as your bucket name
s3 = boto3.resource('s3')
try: 
    if my_region == 'us-east-1':  
    s3.create_bucket(Bucket=bucket_name) 
    else:  
    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={ 'LocationConstraint': my_region }) print('S3 bucket created successfully')

except Exception as e: print('S3 error: ',e)

1. Download the data, then load it into a dataframe.

   If everything is successful, your code will run without any errors (as shown below).

'''this is the link to the datasethttps://d1.awsstatic.com/tmt/build-train-deploy-machine-learning-model-sagemaker/bank_clean.27f01fbbdf43271788427f3682996ae29ceca05d.csv", "bank_clean.csv"'''

try:
    urllib.request.urlretrieve ("https://d1.awsstatic.com/tmt/build-train-deploy-machine-learning-model-sagemaker/bank_clean.27f01fbbdf43271788427f3682996ae29ceca05d.csv", "bank_clean.csv")
print('Data downloaded successfully')
except Exception as e:print('Data load error: ',e)

try:
    df = pd.read_csv('./bank_clean.csv',index_col=0)print('Data loaded into dataframe successfully')
except Exception as e: print('Data load error: ',e)

You can use the Pandas head, shape, or columns function to explore the dataset.

1. Divide the dataset into a train set and a test set.

The train set (75% of the data) will be used to build the model, while the test set (25% of the data) will be used to evaluate the model’s performance.

### the np.split() function splits the dataset into train and test 

settrain_df, test_df = np.split(df.sample(frac=1, random_state=42), [int(0.75 * len(df))])
print(train_df.shape, test_df.shape)

Step 3: Train the Model

In this step, you’ll train your ML model with the train_df dataset.

1. To use SageMaker’s prebuilt XGBoost model, reformat the dataset structure and load the data from the AWS S3 bucket. See code below:

pd.concat([train_df['y_yes'], train_df.drop(['y_no', 'y_yes'], axis=1)],
          axis=1).to_csv('train.csv', index=False, header=False)
boto3.Session().resource('s3').Bucket(bucket_name).Object(os.path.join(prefix,
                                                                       'train/train.csv')).upload_file('train.csv')
s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket_name, prefix),
                                    content_type='csv')

1. Set up an Amazon SageMaker session, instantiate the estimator (XGBoost model), then define its parameters. See code below:

xgbsession = sagemaker.Session()
xgboost_model = sagemaker.estimator.Estimator(containers[my_region], role, train_instance_count=1,
                                              train_instance_type='ml.m4.xlarge',
                                              output_path='s3://{}/{}/output'.format(bucket_name, prefix),
                                              sagemaker_session=session)
xgboost_model.set_hyperparameters(max_depth=5, eta=0.2, gamma=4, min_child_weight=6, subsample=0.8, silent=0,
                                  objective='binary:logistic', num_round=100)
_predictor = xgboost_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

1. Now that the dataset has been loaded and you’ve set up the estimator, use gradient optimization to train the model on the “ml.m4.xlarge” instance. See code below:

xgboost_model.fit({'train': s3_input_train})

Violà! The model training is successful.

Step 4: Deploy the Model

Deploy the model built to an endpoint. See code below:

xgb_predictor = xgboost_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

Once the deployment is successful, the code above will run without any errors.

Now that you’ve deployed the model, you can use the test set to generate a set of predictions. See code below.

  # load the data into an array
test_df_array = test_df.drop(['y_no', 'y_yes'],axis=1).values 
# set the data type for an inference
xgb_predictor.content_type = 'text/csv'
# set the serializer type
xgb_predictor.serializer = csv_serializer
predictions = xgb_predictor.predict(test_df_array).decode('utf-8')
# predict!
predictions_array = np.fromstring(predictions[1:], sep=',')
# and turn the prediction into an array
pd.DataFrame(predictions_array).rename(columns={0: "predicted_values"})

Conclusion

Cloud computing and ML technologies are transforming the world we live in. Amazon SageMaker minimizes the need for maintenance and streamlines the machine learning pipelines while cutting costs. It’s a valuable tool for building end-to-end ML solutions at scale. I hope this article helps you get started.

But this post is not about buzzwords but rather a word that many people misunderstand: DevOps.

I get emails from recruiters and companies looking for DevOps engineers all the time. This makes me wonder how different are people’s ideas of DevOps. Needless to say, the countless number of posts that had been written on technology to master to become a DevOps guru. If you’re looking to transition from a backend engineer to a DevOps engineer and vice versa, the internet got the answer for you. Today, no fewer than 7 million results are available on Google when you search the word “DevOps Engineer position.” Companies are waiting for that candidate to be their DevOps god.

At a time in my career, I thought DevOps was another tech role. Five years ago, I remember sitting at an interview and the engineer at the other side of the table popped the question. He said, “What do you understand by DevOps”? Guess what? I answered it all wrong. I thought it was a role. Retrospectively, DevOps is still a word not many people comprehend.

You might ask, what is DevOps? To understand what DevOps is, we first need to understand what DevOps is not.

It’s not a role or a title.

DevOps is not a role you assume after graduating with a CS degree from a top university. Neither is a role you’re entitled to after building test automation infrastructure for a company’s product. DevOps is much more than a title conferred on one person.

It’s not about tools.

While many tools can augment your DevOps initiatives, DevOps is not about tools. You don’t practice DevOps by installing or learning tools. DevOps is broader than the tools you use. If your organization does not fully grasp DevOps’ main concept, you will have a hard time benefiting from all the concepts DevOps has to offer.

It’s not all about bringing Devs and Ops together in one team.

DevOps is much more than combining Devs and Ops in the same team. While having Devs and Ops in the same team with the purpose of improving collaboration is one of the DevOps practices, it’s easy to fall into the trap that you’re practicing DevOps by merely bringing Devs and Ops together in one team.

It’s not a team

It’s common to see businesses wanting to adopt a DevOps mindset assume they practice DevOps by creating a team tagged “DevOps team” mainly composed of Ops engineers in charge of operations related tasks. Having a DevOps team does not mean you have adopted DevOps.

DevOps is a culture, a way of doing things. It transcends a single team.

It must be embraced by the entire organization. You may have heroes who understand DevOps and have shown the willingness and strong leadership skills able to help drive DevOps transformation across the board in your organization. However, a dedicated team of Ops engineers is not the same thing as practicing DevOps.

What does DevOps mean?

The word DevOps came to be in 2009 after Patrick Debois coined it. It’s a word formed by combining “development” and “operations.” To understand the reasoning behind the word and how it came to be, you first need to understand the problem it tries to solve.

Historically, there is friction between development teams and operation teams, which creates a myriad of problems such as longer time to market, frequent outages, reduced quality, and continuous technical debt mounting. This friction is a result of two opposing goals that both teams must pursue simultaneously.

The development teams are tasked with shipping more features, making frequent changes, and responding to shifting and changing market demands. On the other hand, IT Operations teams are tasked with making service reliable, stable, and secure.

When you make frequent changes, the chances of disrupting service stability or breaking something are high. Since the IT Ops team has a goal of service reliability and stability, it often responds by putting measures that make it difficult to disrupt production services. It hinders the development team from responding quickly to changing markets. These measures often include bureaucratic approval policy, outright rejection, or long waiting time from my personal experience.

In the real world, apps that generate the most revenue are the ones that need frequent changes, releases, and need to be the most reliable and stable. These opposing goals make developer-operations conflict unavoidable, and hence something has to be done. Hence, the word DevOps was born.

What does it mean to practice DevOps?

DevOps is about people as well culture with one goal of continuously delivering business value.

It’s a set of practices and patterns that turn human capital into a high-performance organization capital.

It’s a way of working. It’s about building practices around Culture, Automation, Lean, Measurement, and Sharing. It’s hinged on a collaborative working relationship with IT Ops and Developers.

Image from https://itnext.io/do-not-put-devops-in-a-cage-3604a83821e1

Rather than having developments and IT Ops in separate teams, they are brought together in the same team for a fast feedback loop. The cross-functional team implements feature validation and ensures quality. It’s like my all-time favorite quote from the CTO of Amazon: you build it; you run it.

The cross-functional team builds it; they are responsible for running it.

The team that practices DevOps works in a production-like environment where features are continuously rolled out to production. Instead of a one big bang deployment that happens once a month or year, they deploy several times a day.

Rather than working on a huge feature that takes months to deliver, features are broken into smaller deliverable chunks that provide business value. Short-lived PRs are opened and merged quickly once the feature is done, creating a fast feedback loop. With the fast feedback loop, they can see how their changes and actions negatively or positively impact the business outcome. This team has efficient and fast automated tests that run whenever changes are committed into version control, ensuring stability and security.

Rather than a culture that frowns at mistakes, mistakes are embraced and become a valuable learning tool. The team members are encouraged to take risks.

Rather than waiting for someone to tell them what to do, they take initiative and innovate.

Instead of finger-pointing when a problem occurs, they openly discuss it and learn from it. The organization builds a culture where people are rewarded for taking risks. With the fear of making mistakes gone, they frequently experiment, thus fostering more innovation.

Some features are in production for weeks yet remain invisible to the end-users. They are turned on for the internal or some selected users allowing teams to test and verify that they meet business outcomes before turning on all users.

They conduct several A/B tests to learn user behavior and understand how each feature directly affects the business outcome.

Because the organization hired smart minds, micro-management is non-existent. Each team is autonomous and makes decisions in the business’s interest. Instead of assigning tasks to teams, problems and goals are given. The business trusts the smart minds it hired to develop the best solution to solve the problem.

Because the business cares about achieving business goals, they create long-term teams around each goal responsible for meeting them. Developers are no longer reshuffled or re-assigned to new projects. Each team takes full responsibility and works independently to achieve its assigned goal.

Rather than thinking in terms of output, they think in terms of outcome. They don’t celebrate based on the number of tasks completed but rather on outcomes or business outcomes. Because the team cares about quality, resilience, and reliability, they deliberately inject failures into their production environment to uncover weaknesses and learn how their systems fail and remediate them.

Wrapping up

We could go on and on what DevOps is; the bottom line is DevOps is not a title, a role, or a job function, and you can’t hire it. It’s more of a culture, and your entire organization needs to embrace DevOps for it to work. Beautifully sums up by Irma Harlann in his post, *“The whole company needs to be doing DevOps for it to work”

History is filled with people trying to predict the future by looking at trends and patterns. We don’t have the power to change the past, but we can control future outcomes if we know the future.

For businesses, the ability to predict the future and make informed decisions is critical to their survival.

The traditional method of generating forecasts from time series data often struggles to generate accurate predictions, especially when dealing with extensive data with irregular trends.

The ability to predict demand accurately is a critical need for retailers. They need to know how many inventory store units to have at hand to be at full stock for each product at a given time.

A low inventory level increases the risk of having a stock out, and a too-high inventory level increases the cost related to handling inventory.

Recent research shows that 43% of retailers admitted that they have inventory problems. Retailers have constant pressure to meet market demand. This pressure is exacerbated by the fact that consumers have many options. A customer with unmet needs will not likely return to the same store. It’s estimated that online out-of-stocks problems cost retailers $22B in sales.

The question is, can recent advances in AI help retailers forecast demand and keep customers happy at all times?

The rapid democratization of machine learning solutions (AutoML) is empowering individuals and businesses with little to no machine learning skills to solve complex problems that previously required strong expertise in AI. With AutoML, businesses can apply machine learning to complex problems without getting buried in the complexity associated with building, training, and serving ML models at scale.

One of such offerings is Amazon Forecast.

Amazon Forecast

Amazon forecast is a managed service that uses ML to deliver highly accurate predictions. It looks at historical data (time series data) to build forecasts. Amazon Forecast abstracts the complexity of training, building, or deploying a machine learning model. It automatically examines your data, engineers features, and generates a forecasting model.

Amazon forecast is based on the same technology used at Amazon.com. It has successful use cases from companies like More Retail, Anaplan, AxiomTelecom, Omnys, etc.

In this post, accompanied by a Jupyter Notebook, you’ll learn how to predict sales demand using Amazon Forecast. The workflow demonstrated in the notebook, from data preprocessing to training a predictor, can be fully automated.

Required Steps:

• Download Jupyter Notebook
• Pre-processed dataset and upload to an s3
• Import training data
• Create a predictor
• Create a forecast
• Retrieve forecast

Download Jupyter Notebook

To follow this tutorial, you’ll need to download my Jupyter Notebook here which you can run on Amazon Sagemaker Notebook Instance. Please ensure that your notebook instance has an IAM role with AmazonForecastFullAccess policy, AmazonSageMakerFullAccess policy, and S3 Policy with a read and write access to an S3 bucket that contains the preprocessed dataset.

Pre-processed Dataset

For this demo, I used the public Olist public e-commerce dataset to train a predictor. The dataset can be found in this repository.

It’s important to note that you can’t just feed any datasets into AWS Forecast. A dataset domain must be specified, and the dataset must conform to the structure of the domain. You can think of a dataset domain as a predefined dataset schema or format for a use case. For this example, the RETAIL Domain is a perfect choice.

The RETAIL Domain defines three required fields, the item_id (string), timestamp (timestamp) and the demand (float). This means we have to do some data pre-processing. The following code already.

• item_id: (string) — A unique identifier for the item or product you want to predict the demand.
• timestamp: (timestamp)
• demand: (float) — The number of sales for that item at the timestamp. This is also the target field for which Amazon Forecast generates a forecast.

The dataset has to be pre-processed to conform to our domain: RETAIL domain.

Import Training Dataset

To import the training dataset, we first need to create a forecast dataset. This is where we specify information about the dataset so that AWS Forecast understands how to consume the dataset. We can use Forecast CreateDataset API to achieve this as follows:

Once the import job succeeded, the next step is to create a forecast using the CreateForecast API.

Create a Predictor

To create a predictor, we specify the predictor’s name, the dataset group and some other parameters. We set PerformAutoML to true which means Amazon Forecast will evaluate each algorithm and choose the one that minimizes the objective function.

Note: The predictor will not be available for use until it’s inactive. You can check for this status using forecast DescribePredictor API.

Create Forecasts

Once the predictor is active, we can create a forecast for each item used to train a predictor. In this example, I specified 3 quantiles (0.1, 0.5 and 0.9) per forecast.

Retrieve Forecasts

You can either retrieve a forecast through the console or by using the CreateForecastExportJob API to export the complete forecast into an S3 bucket.

Demand forecast for product moveis_decoracao from September 1st to September 17th

Demand forecast for product pet_shop from September 1st to September 17th

Conclusion

We have seen one use-case of Amazon Forecast in retail and how businesses can apply it to save costs derived from keeping too high or too low inventory. I hope this sparks new ideas as you embark on building your own solution to solve demand problems in Retail.

There’re rooms for improvement, and if you’re looking to develop a fully automated and large scale forecasting solution, I encourage you to look at the Forecast process by applying MLOps

Jupyter Notebook

2020 turned out to be radically different from what everyone had expected. COVID-19 has impacted our lives in many ways. As the pandemic spread, one question that baffled us was: what can we do with technology to combat the virus?.

A few days later, we got the answer.

In a phone conversation in March 2020, Olalekan Elesin informed me of an idea he thought about while doing his regular grocery shopping at DM Drogrie. The idea is centered on a bracelet like your typical watch that dispenses Sanitizer. Before this time, Sanitizer comes packaged in bottles and cans of different sizes and kinds. Some are mounted on walls, and some are portable enough to be carried around in handbags. But not portable enough to be worn on the wrist.

Having spent years building scalable applications and data applications at various startups, we understood that every idea must be validated. We have to know it’s an idea that people want and are willing to pay for. As for us, this means doing some market research.

Problem Discovery and Customer Validation

We started out with one goal split into two hypotheses: Is this a big enough problem, and are customers willing to part with cash in exchange for the product. Validating such assumptions with software/digital products is relatively straightforward. One could build a simple landing page, explain the idea, and add a form — the lean startup way. But for physical products, this is quite different. Developing MVP could mean designing and 3D-printing hand bracelets.

After several iterations in coming up with the leanest and cheapest way to test, we arrived at one: blogpost as MVP.

In the blogpost, we embedded a pre-order Google form to collect some information about potential customers. The form included the color, price, payment method and quantity fields.

While doing market research, it’s essential to collect potential customer contact details like email addresses, etc. If your product resonates with people, potential customers will not hesitate to give their contact details. Customers who provide contact details at the pre-launch phase are usually the first to be converted when you go live.

Before our product hit the market, we recorded over $100,000 in pre-order bookings from matured markets such as the USA, Germany, Italy, France, and the United Kingdom through the medium post with zero ad budget. More than 85% of customers were willing to pay online and have their SanitizerWristbands delivered to them. We validated our problem and customer hypotheses with these and other leading indicators: big enough pain, and customer willingness to pay. This informed our next decision to develop the first physical MVP.

Manufacturing: An Uncharted Territory

Physical goods manufacturing was uncharted territory for us. It was not long before we realized it’s different from everyday app development. In manufacturing, you design and create a product and then replicate it repeatedly. In software, the product design is the product. This is not to say one can not apply certain software development principles to manufacturing. The concept of lean, which is popular in tech, originated from the Toyota production system.

With our first product design ready, we spoke to manufacturing companies to build the prototype. We learned quickly that our design was too cumbersome and would cost a substantial amount to produce.

Figure 1: An Earlier Iteration

We arrived at the simplest possible design that works through customer feedback and trying to cut down the production cost to match the amount customers are willing to pay for.

We were ready and it was time we put our store online.

Scalable eCommerce Website With 0$ Commitment

There are many e-commerce platforms for online retailers that don’t require technical expertise. Shopify and Square are popular choices. We wanted a platform that would allow us faster access to the market with little to zero cost commitment. Shopify and other popular platforms didn’t meet our cost requirements, so we built a custom solution in 2 days with 0$ commitment. It’s a static website hosted on s3 with a serverless cart and inventory capabilities provided by Snipcart.

The high-level architecture looks like this:

Figure 2: E-commerce Website Architecture

Right from the start, our goal was to automatically deploy changes made to our website whenever a PR is merged. We leveraged AWS CodeBuild and CodePipeline to achieve this and automatically deploy new changes to an S3 bucket.

Figure 3: CI/CD with AWS

We sent a campaign email to customers who had pre-ordered before we launched.

Automating Orders Fulfilment

With orders coming in, there was yet one more problem we had to solve: automating orders fulfilment.

Our warehouse and logistics partners are in China. This implies we must find a way of notifying them every time an order is placed. For a few orders per day, sending an email with an attachment to our logistic partners suffices. As the number of orders per day grows, this becomes really boring and automating the process became a necessity. Could we automate it?

To automate fulfilment of orders, we created a serverless “order fulfilment addon” whose architecture looks like this:

Figure 4: Automating Order Fulfilment (WIP)

Making Informed Decisions with User Behaviour

Our conversion goal was to get potential customers to buy our product. To reach this goal, we needed to learn about the things that interrupt user flow from right from landing on the website to actually placing an order. For us, this means conducting several A/B, Multivariate tests weekly and implementing what gets us closer to our goal.

Lessons Learned

While we’re yet to achieve a massive scale like the popular e-commerce platforms, we’ve learned some valuable lessons that could spark new ideas for people looking to start online businesses.

• When it comes to manufacturing, you’ll save a sizeable amount of money by manufacturing from developing countries (like India, etc.) than from developed countries like the US, Germany, etc.
• If you’re in Germany, registering a business can take a long time. If you plan to get to market fast, ensure you factor in the time, it will take you to navigate the bureaucracy.
• Patent your ideas before doing market research. We started with market research. After we saw that the idea was valuable, we wanted to move on with patenting; unfortunately, it was too late. Someone already filed a patent
• China makes manufacturing look like a piece of cake.
• Data is king. “In God we trust, all others bring data” — Edwards Deming. It’s important to start with a data-driven mindset, no matter how small. Think of how to decompose big ideas into testable components and figure out the data required to validate along the way.
• Get your analytics right early! This is not referring to Google Analytics. Consider the metrics that are indicative of your success and instrument them from the get-go. Great resource — $9 Marketing Stack: A Step-by-Step Guide.
• Experiment like no one cares. For us, we were always experimenting, we have a minimum of 5 tests running weekly. A/B tests, multivariate tests, Redirect tests, etc. You can do this for free with Google Optimize.

According to a report, there are 48 million bots on Twitter — and hubofml happened to be one of these bots. Hubofml started as a simple bot written in Node.Js (on one Sunday evening) running on a free Heroku dyno that tracks specific hashtags like machinelearning, computervision, and retweet tweets containing those hashtags.

My goal was to use the bot to collate information on machine learning and re-broadcast to people interested in them — after all, some of the best posts I read on machine learning came from links shared on Twitter by the community.

I thought it would be cool to have a bot that tracks hashtags related to machine learning that I follow to stay informed.

Days after the bot was deployed, I started noticing some forms of abuse and spam like this:

People would write tweets unrelated to machine learning and hashtag “machinelearning,” and the bot would retweet it.

When we think of spam, it’s easy to see them as unsolicited emails one receives from an unknown person.

As odd as it may seem, spam is not limited to emails alone these days. Spammers now target everything you can think of. From your inbox to comments on social media posts, you’ll find spam lurking around.

Personally, I think spam is destructive to communication and undermines the goal of social media.

For the rest of the post, I will focus on how I used text categorization to combat spam on Twitter.

Text categorization is the process of automatically assigning one or more predefined categories to a text document. It has a wide range of use cases like articles categorization, spam detection, intent detection, etc.

As I proceed, if you like to take a look at the Jupyter notebook I used for this task, you will find it on Google colab here or in this repository.

Getting Twitter Datasets

There are four ways to obtain Twitter public data. Justin Littman wrote an impressive article on the four approaches.

For this task, I used the free (Twitter Standard) APIs. I downloaded about 110k of tweets datasets from Twitter using a python library called Tweepy.

I annotated the datasets using two labels: “yes” and “no.” All spam tweets were labeled “yes” and non-spam tweets were labeled “no”.

Spam tweets (in this context) are tweets with no context to the field of machine learning. They are usually about politics, crime, religion, trolling, etc. I retrieved spam tweets based on randomly selected keywords from Wordnet as hashtags.

Non-spam (Ham) tweets are tweets related to machine learning, computer vision, NLP, etc. I collected non-spam tweets by tracking hashtags like machinelearning, computervision, NLP on twitter using the Tweepy library.

Data Exploration

My next step was to accrue some knowledge about the data and its nature after downloading the datasets. The datasets contained 99,989 tweets labeled as spam and 10,538 tweets labeled as non-spam.

Spam vs Non-spam

The data distribution between the two labels shows that the tweet datasets are highly unbalanced. Unbalanced datasets often have substantial effects on how machine learning models generalize. The model could learn that spam tweets are more predominant, making it natural to lean toward the predominant class during generalization.

I downsampled the majority class to arrive at an equal number of spam and non-spam (ham) tweets, 10,000 tweets in each class, accounting for a total of 20k tweets in the downsampled dataset.

Spam vs Non-spam

To get further insight into the data, I plotted the top-10 most common words in the two classes. For the non-spam tweets, ai, machinelearning, artificialintellgience, data, data science, biodata, python, and deep learning were found to be most common, as shown in the figure below.

On the other hand, words like good, lol, like, thanks, day, know, and sorry were found to be most common in spam tweets.

Data Preprocessing

Like text documents, tweets are not exempted from noises. This is, as a result, no formal way of representing tweets. Tweets contain certain special characteristics, such as usernames and retweets, signified by “RT,” links, emoticons, and unimaginable things.

It’s essential to clean them up before fitting a model. I applied several preprocessing steps like lowercase conversion, URLs removal, usernames removal, emoticons removal, tokenization, and stemming.

In addition, tweets also contain common words that are of little value to the context of the text. These words are known as stop words. Stop words are a set of commonly used words in any language. Hence the removal of stop words from tweets was crucial to focus on the important words in the tweet.

Feature Selections

Vocabulary list

A vocabulary list is a dictionary of words having each word in the dataset as a key and the number of times they occurred as value.

A vocabulary list could be:

{"machinelearn":100, "Trump":10, ""}

I built a vocabulary list of all words, excluding stopwords and words with less than 2 occurrences in the datasets.

Tweet Vectorization & Padding

Machine learning models take vectors as input. In order to perform machine learning on text documents, we need to transform text documents into vector representations. This is known as text vectorization.

In my approach, I assigned a unique number to each word the vocabularies. Each tweet is encoded using the unique number assigned to the word. If a word could not be found in the dictionary, it’s automatically assigned a value of 1 — a value reserved for words that were not found in the vocabulary list.

Given the following vocabulary list:

{ "The": 5, "cat":1, "sat":3, "on":4, "mat":2, "in":6, "unk":0}

A sentence like “The cat sat on the mat in the morning” will be encoded as [5,1,3,4,5,2,6,5,0 ].

Finally, I used pad sequence to convert the variable-length sequence to the same size. This is crucial for the network to take in a batch of variable-length sequences.

Dataset Splitting

I split the datasets into training and test set. A training set is used in learning (fitting the model) and a test set for testing how the model generalizes on unseen data. 60% of the dataset was used for training, and 40% was used for testing.

Model Architecture

In the past few years, there have been many groundbreaking successes from applying RNNs to a variety of problems: speech recognition, language modeling, translation, image captioning, etc.

RNNs are great, but they suffer from short-term memory. If a tweet is too long, an RNN might have a hard time carrying information from earlier time steps to the current time step.

I used an LSTM network, a variant of RNN. LSTM avoids the long-term dependency problem by remembering information for long periods. They have internal features called gates that regulate the flow of information in and out of the cell state. There’s an interesting article on LSTM if you want to know more about how it works.

The network architecture consists of three layers, and it’s bidirectional. A bidirectional network allows inputs to be processed from the first to the last and from the last to the first. This ensures that the network is able to preserve information from both the past and future.

The first layer is the embedding layer, which transforms the input vectors into dense embedding vectors.

The second layer is the hidden layer, which takes in the dense vector and the previous hidden state to calculate the next hidden state. The final layer takes the final hidden states and feeds them through a fully connected layer, transforming it to the correct output dimension.

You can find more on the architecture in the Jupyter notebook.

Deployment & Inferencing

After training the model, I deployed the model artifacts using Flask to Heroku free dyno for real-time inferencing.

From tracking tweets to retweeting or liking, the communication process looks like this:

• Bot finds a new tweet on machine learning
• Bot calls the Heroku endpoint to predict the class of tweets.
• Non-spam tweets are retweeted/liked while spam tweets are ignored

Conclusion

While the model performed beyond my expectation, however, it’s far from being accurate. Occasionally low-quality tweets still manage to escape the spam filter. I believe that could be improved by using quality datasets.

The tweet datasets I used for this task were in tens of thousands; there were tweets with wrong labels. This can be improved by crowdsourcing the labeling task using services like AWS Mechanical Turks.

This is my first attempt at combating spam on social media, and I do hope to take this work further in the future.

Every month, I send out a newsletter containing lots of exciting stuff on data science, software engineering, and machine learning. Expect quick tips, links to interesting tutorials, opinions, and libraries. Subscribe here