Management in AI

The Ultimate Guide to OKRs and KPIs in Machine Learning

November 3, 2022
Back to blog

In this guide, we’ll present a tool called OKRs (Objectives & Key Results) which provides company direction and accountability under times of uncertainty. We’ll discover how to apply OKRs to machine learning projects, because there are key differences with non-ML projects. We’ll also describe critical mistakes to avoid when applying OKRs. In the final section, we’ll consider ethics and safety problems when using these powerful tools.

1. A Brief Introduction to OKRs and KPIs

One critical task for machine learning leaders is to align project outcomes with the business’ goals. This is already challenging for traditional projects, but even harder for ML since their outcome is often unpredictable. So how can you align ML project milestones with business goals? How can you ensure different teams are all going in the same direction?

What are OKRs and KPIs?

Let’s define OKRs as favorable outcomes or goals, and KPIs as interim measurements towards that goal. Each objective should be bold and inspiring, but also have a few measurable results attached. This allows teams to dream big, while also encouraging progress and holding contributors accountable. Here’s an example:

Objective:  Build a superior manufacturing production line.

Key Result #1:  Achieve below 1% production failure rate.

Key Result #2:  Produce 1.5x more than current capacity.

Key Result #3:  Deliver the first 5.000 products to customers.

Who should dictate Objectives and Key Results?

Ideally, objectives are defined top-down and key results are defined bottom-up. This allows a small group of executives to choose the overall direction of a company, while allowing teams to create key results that contribute to this direction. Brainstorming new objectives can happen in dialogue with employees, but leadership must ultimately converge on the most impactful ones.

Conversely, teams working closely on the problem are in the best position to define key results. They should collaborate with their team managers to choose the results which move the needle on projects. Managers should be prepared to explain these key results to leadership and stakeholders, but leadership must ultimately trust them that these are the most critical results.

How to balance OKRs with the right KPIs?

Some of the most complex work in modern companies is to find the right KPIs that support moving towards an OKR. Whereas OKRs are qualitative goals, KPIs are quantitative metrics. If not done carefully, pairing objectives with the wrong key results can lead to adverse outcomes.

Imagine a firm that wants to improve their documentation process of ML projects. They want to capture all assets and document them for future reuse. They could reassign engineers to handle the tasks of tracking, annotating and unifying all important assets. An OKR could look like this:

Objective: Capture all ML assets and make them interpretable 

KR1: Track and store all assets generated by ML team

KR2: Annotate documents and track important metrics

KR3: Create weekly status reports for review meetings

2. How to Use OKRs and KPIs for Machine Learning

One critical difference between traditional projects and ML projects is that machine learning is research-based. Outcomes are not rigidly defined and projects might fail. Some ML managers won’t use OKRs on the grounds that machine learning is iterative and speculative. They argue that ML projects are an exploratory process and cannot be planned with deadlines. 

But OKRs are great tools to manage ML projects, since it allows us to set ambitious goals while remaining flexible with quarterly reviews and variable KPIs. They create a system of progress and accountability under conditions of uncertainty. Without goals, ML teams are just trying things and hoping for the best. 

How can I set Objectives for ML projects?

Leadership should understand that machine learning isn’t a magical solution that will solve your problems automatically. Rather, ML algorithms can be used to optimize or automate current procedures. Ask yourself what breakthrough would set your company apart from competitors. What part of your business would benefit the most from automation or operational efficiencies? What should your team work on if they had 10% more time or resources?

Objectives can be defined locally (for teams) or globally (for organizations). An example of a local objective is improving a recommendation system relevance score to 90%. While success is not guaranteed, defining the most impactful outcomes sets teams on the right path to accomplish them. Working on these projects requires a combination of metrics-informed and intuition-based decision-making.

3. Common Mistakes when using OKRs in ML

Misaligning Talented Employees

Misalignment happens when teams or employees are not working on projects that move the needle on OKRs. Highly trained professionals spend years developing technical skills and knowledge. They don’t want to complete simple requests for leadership which are “somewhat” related to their field of data science. This happens surprisingly often and it’s one of the top reasons why data scientists are leaving organizations.

Ideally, companies want data scientists or machine learning engineers working on solving a core set of data problems. Analysts should perform technical work in an engagement model to support customers directly. You don’t want a specialized expert providing answers without context. You want to ensure that their time is spent on core tasks that contribute to the KPIs.

Setting Unreasonable Objectives

Leadership should be careful to set ambitious, but reasonable objectives. This is particularly true with cutting-edge technology like “machine learning”. Leadership should aim to have a reasonable understanding of the capabilities and limitations of machine learning. Rather than focusing on “having ML in their organization”, they should ask themselves which aspect of their product or company they want to optimize. They should define objectives with ML managers, who then define metrics with key contributors. 

Doing “Business As Usual”

OKRs should not be used for doing “business as usual” either. This includes maintenance and recurring work (risk mitigation, administrative tasks, hiring procedures, server monitoring). While you can still use KPIs to stay above or below desired performance levels, you should only use OKRs when aiming to implement big changes or achieve bold objectives.

The good news: machine learning is one of the few tools that can redefine “business as usual”. ML models can find patterns in large amounts of data and automate decision-making. When done right, it can have a profound impact on KPIs. This forces leadership to think about the most impactful areas of the business.

4. OKRs for Machine Learning: Ethics, Morals and Safety

Machine Learning Ethics

Modern technology has a global reach with massive network effects which can amplify negative outcomes. As such, companies releasing products or services are held responsible by their governments to protect consumers. While many industries have been around for decades and already have regulations in place, machine learning is still a new field that’s under development. This makes it extremely important to be prudent when releasing ML products on the market.

Companies might get too caught up with their goals that they stop thinking about negative outcomes. Considering all the hype surrounding machine learning, many leaders might want to implement ML into their business as soon as possible. However, leadership should understand that ML is different from traditional software. ML models have a hidden layer that makes the outcome unpredictable. Some of these outcomes might cause harm or discrimination.

The Black Box Problem

One problem with machine learning is that the models act as a “black box”. Models are fed with data, hidden parameters get updated (“learn”) and an output is provided. While engineers can curate input data and evaluate output data, they cannot easily evaluate what’s going on inside the model. As such, they cannot predict every outcome the model will provide. Some of these outcomes might cause harm, and it is the company’s duty to protect consumers from these outcomes and update their models accordingly.

Bias and Misrepresentation

Another problem with machine learning is that output is based on sample input data, which can reflect human biases or misrepresent the population. In other words, racism or discrimination can be hidden in data sets because the data was created by humans with such tendencies in the first place. 

Since machine learning is such a new field, companies are still struggling to create techniques that counter biased data. Be aware that your organization holds the responsibility for a model’s output and you will need to invest resources in correcting and compensating for harmful outcomes. 

How to Avoid Negative Outcomes in ML

Biased outcomes can be prevented and corrected with better data processing and continuous monitoring after deployment. Data sets can be tested and corrected for bias before using it as training data. And ML tools post-deployment should have a short feedback loop where users can quickly report unethical or discriminatory results.

Machine learning models should undergo a rigorous selection process before being deployed in the real world. If the model performance does not comply with standards, it should be discarded. That’s one of the reasons why most ML models never see production. While this may seem like an expensive trade-off, the negative consequences of harmful output can be far costlier.

If you want to learn more in detail, here are 4 detailed articles that include examples.

- A Brief Introduction to OKRs and KPIs

- How to Use OKRs and KPIs for Machine Learning

- Common Mistakes when using OKRs in ML

- OKRs for Machine Learning: Ethics, Morals and Safety

About Vectice

Vectice, the Data Science Management (DSM) Software, empowers enterprises and their teams to establish best-in-class AI practices to accelerate the impact of AI on their business.

Vectice offers the first solution that embraces the iterative nature of data science workflows, centralizes AI assets into a searchable catalog, and is compatible with the tools data scientists already use.

Vectice enables data scientists to capture milestones with associated assets and report key findings to colleagues and stakeholders.

For managers, Vectice is the solution of choice for defining and reviewing initiatives, securing team knowledge, and showcasing projects' impact to stakeholders.

Get in touch

Get updates, early access and provide input while we are designing Vectice for you.
Contact Us