That’s good and necessary, but it’s also hard. Whether it’s access to the nearest national park or access to health care, subject matter and technology experts inside government and those supporting them, like me, share a responsibility to create products that build trust in government.
As more government agencies adopt a product thinking approach to their digital services, they often face a problem — how do we scale success on individual product teams into success for our entire product organization? Product organizations in government navigate the typical challenges of a private sector product organization (will people use this product, can they use it, can we even build it) alongside added challenges in an ecosystem of bureaucratic and policy uncertainties.
It’s in this environment that we’ve found that product operations can help scale impact by putting objective indicators at the center of product decision-making.
We’ve seen success in supporting product thinking at agencies like the Department of Veterans Affairs (VA), where they made it easier for Veterans to access employment and education assistance and for caregivers to receive needed support.
They’re seeing success because they changed the way they build digital services:
- They understand the difference between — and respective roles of — product and project thinking.
- They pursue opportunities with cross-functional teams made up of product management, design, research, engineering, and other experts.
- These cross-functional teams are more empowered to discover and deliver products than centralized command-and-control operations that supply prescriptive solutions.
Because of their success, they are challenged to achieve even more. This could mean they improve their reach to underrepresented groups, reduce the time it takes for a Veteran to receive benefits, or experiment with technology like mobile devices that are ubiquitous outside government tech.
But we know it’s not that easy to simply achieve “more.” These teams figured out how to accomplish outcomes at a certain, likely smaller, scale; that original recipe may not be easily multiplied. Product operations offers a path forward to scaling this success.
What is product operations
Product operations helps product organizations within agencies strengthen and scale their ability to achieve outcomes for the people they serve.
Blake Samic, Stripe’s head of Product Operations, visualizes product operations as the “connective tissue” within an organization, and Melissa Perri, author of Escaping the Build Trap, shares a helpful definition that breaks product operations down into internal, external, and process-driven components.
To me, product operations is how a product organization assesses and improves the quality of its practices as it builds valuable and usable products in pursuit of its vision. This model assumes two prerequisites: strong practice leadership and a compelling and relevant organizational strategy. It’s leadership’s role to ensure folks from the various disciplines are supported as they execute the organization’s strategy — an assessment of the future they want to create and how they intend to overcome obstacles in their way. It’s across this map that the product organization navigates.
At the heart of a strong product operations model are three categories of indicators that help teams gauge and track progress:
- What people find valuable
- The state of internal disciplines
- Team health
These aren’t all the indicators a product operations team will need, but they are the minimum metrics that product operations teams should track as they begin scaling product thinking across their organization.
With each indicator comes:
- Artifacts to elevate and track the organization’s progress against a specific indicator
- Data that teams gather and analyze
- Insights teams generate out of the data that help them make educated product decisions.
Let’s go more in depth into each of these minimum indicators that organizations can use to gauge the health of their product organization.
What people find valuable
This category helps teams understand the impact, or value, people feel as a result of a team’s decisions. It requires knowledge of their market, or in the case of government digital services, the journey a Veteran, a senior, or a Medicare patient is on.
A common source of this data is analytics, which teams capture to measure and track product key performance indicators (KPIs). Teams develop conjectures for how best to improve a product and then validate them with data to generate insights.
Qualitative studies are another source of data that teams can use to generate insights. Teams conduct studies using prototypes of varying levels of fidelity to understand how usable an experience might be.
Minimum indicators
- Outcome metrics are the most intuitive of the indicators and measures how well a product delivers a service. It doesn’t focus on granular interactions like page views or clicks (though they might be helpful signals). Instead, it focuses on the value delivered such as the volume of successfully submitted applications for health care last week.
- Iteration rate focuses on how well teams learn, build, and measure. We capture this information to assess how quickly we can incorporate new insights into subsequent iterations.
Sample artifacts
- KPI dashboards consolidate and present the most important product metrics.
Sample insights
- People are more likely to apply for education opportunities if they learn about the benefit a certain amount of time prior to discharge from the military.
The state of internal disciplines
This category focuses on the quality of a team’s product, engineering, research, design, and other practices. They help teams understand whether they maintain a high standard of work within the respective disciplines.
Successful product organizations leverage the combined expertise of individual practitioners to build the systems that power service delivery; these practitioners conduct discovery and delivery as a cohesive unit and avoid hand-offs between groups. That requires that they understand how the components of those teams fare.
Minimum indicators
- Product
- Opportunity-oriented documentation assesses a team’s focus on the problem to be solved, its alignment to the organization’s strategy, and its prospective impact on people and the organization.
- Outcome-focused prioritization centers on interactions that create value for the user and organization. We ask whether teams are proactively defining metrics during prioritization (vs. prioritizing initiatives without a clear articulation of the measurable impact they intend).
- A transparent and frequent feedback loop evaluates a team’s habit of reflection on product bets they make.
- Engineering
- Test coverage determines whether teams have adequate checks in place for the software they write.
- Telemetry gauges teams’ visibility into product and platform performance.
- Research
- Referenceable research checks for a healthy collection of direct qualitative feedback that existing and new teams can use.
- Inclusive research practices measure “upstream” representation of populations who use the products and their accommodations.
- Design
- Design system contributions measure the team’s culture of reusability.
- Usability studies with assistive technology center a user’s perspective in validating assumptions around design and interactions, including those with assistive technology.
Sample artifacts
- An impact review is an artifact that highlights teams’ transparent and frequent feedback loops. In these reviews, teams reflect on the problem they pursued, their chosen hypothesis (and alternatives they considered), impact to outcome metrics, and their recommendations for how to make further progress.
- A product requirements document (PRD) is a common product management artifact. We should be mindful that these documents don’t focus too much on the solution itself and instead on the problem the team pursues, which is the intent of the opportunity-oriented documentation indicator.
Sample insights
- We are diligent about inclusive research practices for certain benefit types and demographics.
- Accessibility checks are not a natural part of our engineering process.
- A particular user interface pattern is useful in some situations but harmful in others, especially if the person uses certain assistive technologies.
Team health
This category focuses on how folks are doing within their teams and in their collaborations with other teams. After all, product organizations (and their products) are only as sustainable and healthy as the people who build them. And insights into team health help leadership understand how best to support them.
Minimum indicators
- Team cohesiveness gauges internal team alignment and trust.
- Team composition evaluates the balance of skills and roles across teams.
- Team decision-making assesses team members’ willingness to share concerns and have a say in iterating its own processes.
- The cross-organization relationship considers the health of collaboration between teams, including government and vendor folks.
Sample artifacts
- Team health surveys devised by practice leadership gather data on how teams work and where to focus their energies.
Sample insights
- Teams feel confident in raising and correcting issues on their own.
- Teams do not have the right number of people and mix of skills.
Creating your own indicators
Above, I shared the minimum indicators a product organization should track. If your organization wants to create and track additional indicators across your teams, I recommend that they exhibit these characteristics:
- Relevant: The indicator is conducive to product decision-making. You don’t want teams wasting their time and energy measuring interactions and information that isn’t helpful to generate insights that aren’t pertinent. For example, we’re not interested in the number of tickets teams opened last month. We are interested in whether those teams are stuck behind immovable blockers and dependencies.
- Organic: The indicator is a part of the product development process and there’s a natural discipline about its use that doesn’t feel forced. For example, we’re interested in how often members of an organization take time to rest and recuperate, which we can easily learn more about through artifacts like internal systems or 1:1s.
- Measurable: The organization must have a way of measuring and tracking progress against the indicator. If we can’t measure it, it’s not useful. That might mean we use surveys and other qualitative tools to assess our progress.
John Cutler, Amplitude’s Head of Product Education, has a helpful heuristic for any indicator, artifact, or process an organization might use or develop – ‘what job are we hiring it to do?’
Why we, and the agencies we support, should care
The job of a product organization within an agency is to support their teams as they build digital service products that are usable and valuable. What drove their initial success — their nimble nature and ability to quickly learn and then iterate their products — becomes more difficult as they grow.
Data and insights can be isolated to unseen or unfamiliar pockets of a growing organization. That isolation increases as organization size and complexity — such as within a government agency — increases. This isolation is further exacerbated if organizational incentives aren’t aligned. That misalignment makes it harder, for example, to share actionable information about our products and people. Judgment is still critical. Teams within successful product organizations figure out how to combine strong judgment with repeatable processes to achieve their desired outcomes.
Product operations centers the data a product organization needs to scale its impact. This is data about the value people receive from our products, the state and quality of our internal practices, and the health of our teams. The role of a product operations indicator is to measure our progress in those areas so we can improve our ability to discover and deliver value to our users. Product operations puts a name and framework around cultural aspects of successful product teams so their organizations can foster and scale that success.
