So you’ve started to scale your business you’ve secured that latest round of funding and now you need to move from MVP product to enterprise scale…
This is where things get well quite formal, and skillsets start to change.
How do you make sure you are hitting the non-functional performance requirements your big corporate enterprise customers have? Or your impending millions of users?
How do you demonstrate to your investors your solution is performing well against customer demands, and that your team is efficient and has the right tools and skills for the job?
How do you incentivise your staff to improve performance and add some real discipline to delivery?
How will your leadership team make key decisions?
Well, the obvious answer is data and that also means setting up the right KPIs so you have meaningful measures against which you can all from the lowest levels of your day-to-day engineering capabilities to your C-Suite leadership team track that progress.
KPIs for Solution Performance
Your solution should have parameters set for performance. For example how fast a query should return a data-driven result if it's an accounting application. How many concurrent users your solution should support — including peak periods? Imagine running that digital retail channel and not having enough server capacity for the number of users all buying online AT THE SAME TIME. Losing sales as your servers hit capacity isn’t a great look when autoscaling instances is so easy to set up…
Defining your non-functional requirements is key for defining scalable solution architecture, it's also important to demonstrate metrics in order to measure this against customer feedback and perception. Poor solution performance can lead to customer churn. Not anticipating growing requirements leads to technical debt.
Each release should deliver improved performance, each release test should be re-run to demonstrate that improvement. That performance then needs to be baselined.
KPIs for DevOps and Infrastructure
There are some standard SaaS KPIs for DevOps and that is because so much of this discipline sits around releases and improvements to those deliverables.
See link to Atlassian article here and edited summary below.
https://www.atlassian.com/devops/frameworks/devops-metrics
1. Lead time for changes
Lead time for changes is the length of time between when a code change is committed to the trunk branch and when it is in a deployable state. For example, when code passes all necessary pre-release tests.
2. Change failure rate
The change failure rate is the percentage of code changes that require hot fixes or other remediation after production. This does not measure failures caught by testing and fixed before code is deployed.
3. Deployment frequency
Understanding the frequency of how often new code is deployed into production is critical to understanding DevOps success.
4. Mean time to recovery
Mean time to recovery (MTTR) measures how long it takes to recover from a partial service interruption or total failure.
Release Metrics
If you are new to release metrics, I strongly suggest starting with two, one for efficiency and one for quality. My favs are time to release — as efficient releases shouldn’t take hours, and change failure rate. Rates of error, once you have deployed into production, should be very low, or if you have inconsistency in your environment, testing etc it is critical that changes deployed are an improvement not another source for customer complaints.
KPIs for Delivery
Delivery KPIs sit around accurate forecasting and delivering on that forecast. Forecast against actuals. My consistent view over many years is that someone on the team needs to be responsible for the project management of all deliverables. Ensuring things are accurately forecast and delivered within customer expectations. Owning that sprint reporting in Jira, making sure all staff hours are tracked and budgets managed consistently.
- Sprint deliverables
- Customer deliverables
- Forecast and actuals
- Hours and story points
Investors will want to see that customer deliverables are met, product and technical roadmap items are delivered and that releases are efficiently managed. They will want to see efficient use of capital. You will need enough cash flow to get you to your next financing round. Someone needs to make sure that is managed on a detailed team level day to day.
KPIs for customer value
Regular customer reviews are essential, measures around service levels, and ability to deliver to commitments are key. All software firms have a level of bug fixing and customer-specific requests for teams to respond to. Making sure those service levels are consistent. Customer feedback sessions can give you some insight on perceived versus real metrics. Often customers perceive poor performance long after an issue is resolved. Make sure you have the data to back it up.
KPIs for business value
Customer churn and LTV are obvious sales metrics you should be connecting over time to levels of technical debt and the ability to respond to customer support and service-level issues. These are really critical, high churn businesses are not sustainable ones.
KPIs for Product
Customer engagement with the solution is an obvious one, as is making sure that you test your customer journey. If they are struggling to identify basic functions in your solution it is not an efficient UI /UX experience. Likewise, if they are barely engaging with the solution you may not see that subscription or license renewal.
Ratio of engagement also to product pricing, is really critical, once you have a good enthusiastic customer you need to capitalise on that. Customer feedback and interviews will not just help assess customer satisfaction but can also help uncover that next product development opportunity.
Product Managers should be responsible for the profitability and commercial success of their products. They will not deliver if they don’t have this built into their KPIs. No amount of nice fluffy UX talk will deliver a product that makes money. Commercial focus is key. Make sure you outline targets for growth and that they can offer an effective strategy and KPIs they share with sales teams to deliver on that.
Tying it all together.
These KPIs should be part of a consultation process with your team members, they have to be sure that their KPIs are tangible and realistic for them. Individuals who will not wear KPIs against their roles are unlikely to deliver value and fear the scenario they are in. This is often indicative of poor skills for the role or a chaotic scenario where the team and solution are simply not set up to deliver the performance required. If you can’t achieve or agree on KPIs then you need a plan of improvements in order to get there. This can mean training and hiring new skills, paying down technical debt, and putting procedures in place.
Having an HR solution in place to ensure these KPIs are managed and set is key and annual reviews for staff performance against these KPIs is essential. For engineering staff, you may want to support skills with technical interview reviews, training and external certifications.
Understanding service provider KPIs
Cloud providers for which most of the business world is dependent on some level these days have their own KPIs and best practice use of their tools. Its critical these are understood by both engineering and customer-facing staff and that line of demarcation where their services start and stop is understood. If your business runs on infrastructure provided by a 3rd party not understanding the parameters of that service provision can be very expensive indeed. Poor service levels are often a result of this. Leadership needs to anticipate changes months in advance as these services require anticipation and assume a level of professional planning and adherence. Check appropriate training and certification levels or consider this in your hiring requirements against your service levels and solution requirements.
Conclusion
In general, start simple, and ensure reports can be aggregated to demonstrate summarised progress and tracking for investors, where you make progress expose that for your customers and make sure its tangible for your staff. Many managers and businesses set unrealistic expectations around metrics this will not help you deliver.
Operational efficiency is about tactical improvements over time and ownership of those improvements by your teams. KPIs are essential for trust, transparency and reporting. Make sure you are realistic and balance uncertainty with pragmatic solutions.
Building a business from scratch is hard. Not that many people are cut out for early-stage scaling organisations or have the skills to really deliver and follow through. For those that do well, I wish you well and good luck.