Cross-sector Collaboration during COVID-19: Lessons Learned from Colorado Exposure Notifications Metrics
Authored by Michelle Park
In October 2020, the Colorado Digital Service (CDS), in partnership with the Colorado Department of Public Health and Environment (CDPHE), launched CO Exposure Notifications Express (ENx) to augment Colorado’s COVID-19 disease control strategies. Exposure Notifications Express is a service on an iPhone or Android device that alerts users who have been in close proximity with someone who tested positive for COVID-19. Developed by Apple and Google, in collaboration with the Association of Public Health Laboratories, the service allowed for custom configurations to be individually deployed by state governments.
In December 2020, I joined Colorado’s ENx team as a Product Manager with CDS as part of my rotation through the Schmidt Futures Associate Product Manager program. For the following five months, I walked through the unique challenges of delivering software in government during a pandemic. This piece details a snapshot of our work and what I learned along the way.
Some of my first sketches to understand the user journey through ENx
Automating Exposure Notifications Metrics
From the onset, we decided to collect minimal information to protect Coloradans’ privacy.
Because ENx was a collaboration across several different organizations, there were limited anonymous and aggregate data sources scattered across dozens of platforms that took substantial effort to collect and analyze. One engineer on our team would spend several hours manually finding, cleaning and copy-pasting this anonymous data from each source into a spreadsheet. These numbers were then copied into multiple other spreadsheets by different people who needed to aggregate the numbers in a certain way. This led to a large array of places to check for the most up-to-date numbers for the service in order to make product strategy decisions.
To address this challenge, my first project at CDS was to lead the State’s efforts to automate its metrics pipeline for ENx. Our goal was to automatically collect key performance metrics for the system into a single database and update relevant numbers daily into a single spreadsheet.
Diagram of the goal ENx metrics pipeline (ongoing work)
To augment our engineering capacity for this effort, we onboarded three volunteer developers from the US Digital Service and US Digital Response (you can read more about this collaboration here). After three months of research, design and development, we successfully launched a general-purpose framework that collects data across several platforms into an aggregated database and updates this daily into a spreadsheet viewable by Colorado’s epidemiologists, Governor’s office and our partners.
Cross-state collaboration on metrics
As we worked on metrics, we quickly noticed that this challenge in aggregating data was not a problem just for Colorado. Several other states and government agencies were separately spending resources to aggregate ENx metrics from the same set of data sources for their own jurisdiction. In an effort to reduce this repeated work, we set an early team goal to prioritize reusability in the design of our metrics pipeline and build with the intent to open source. This empowered our developers to lead discussions on generalizability, even when other urgent issues arose for the system.
In addition to doing the work, we prioritized showing up and being transparent with other states about our challenges. For example, we spoke frequently with California, Washington, Oregon, Hawaii, Massachusetts, and several other government entities as part of a collaborative cross-state group focused specifically on ENx. These conversations showed me that cross-state relationships can become force multipliers for separate instances of similar government work, especially when focused on actionable ways to address a shared problem. Here are some of my key takeaways from our work.
Key Lessons Learned
1. Relationships are critical, so invest in others early and often.
Building anything in the government starts with strong relationships, both with those who know more than you and those working alongside you. State technology infrastructure is extremely complex, highly regulated, and contains decades of historical decisions that are impossible to navigate without help. In environments with high ambiguity and complexity, knowledge becomes a form of power — and without structural incentives to share that knowledge, relationships become the primary way to access the knowledge of others.
2. Rapid iteration in government is challenging, but critical and possible.
Rapid iteration is a critical strategy to build good products, and this is especially true in government due to the magnitude of the product’s impact. However, rapid iteration is also significantly harder to do in the government. For instance, how should teams fail fast in the public sector when it can take upwards of a month to get clearance for a single data source? The constraints and opportunities in government are very different from those in the private sector, but traditional Agile methods can be adapted with this in mind. For example, to iterate as quickly as possible while waiting for approval to access relevant state systems, our volunteers began developing a general-purpose framework in a separate and secure repository while maintaining a close loop with the state developer on our team. Although their eventual deployment into Colorado’s infrastructure took a long time, we were able to iterate while access control and permanent environments were underway. Looking back, our team recognized opportunities to adapt even better going forward, such as by pairing up with a state developer to deploy smaller chunks of code earlier while waiting for access. Being agile is significantly more challenging in the public sector, but it is both possible and worthwhile to modify iterative product practices to work in government.
3. Keep a pulse on your team and their ecosystem, and stay flexible.
The CO ENx team included people working part time, full time, or as volunteers across many different organizations and industries. This diverse team makeup posed an interesting challenge in creating a strong, cohesive team culture in a remote work environment. To make this work, I kept a close feedback loop with the team and tried several ways of working until I found what clicked. For example, I moved away from daily standups and product requirement documents that would create too much structure. Instead, I focused on driving asynchronous communication channels and running one efficient weekly meeting. Because our engineering team consisted primarily of volunteer developers, I also kept a close pulse on their time constraints, estimated rolloff date, and project interests.
Importantly, I ran into several hurdles along the way. For example, I underestimated how long it would take for volunteers to go through proper security protocols to gain access to state systems. I also later learned that our volunteers were missing an asynchronous way to discuss implementation that was more visible to everyone on the team. To surface these roadblocks as quickly as possible, I prioritized establishing a strong sense of trust in our team. This meant checking in, actively listening, encouraging honest feedback, and making space for laughter on the team (credits to this random icebreaker generator and the dancing Doge emoji for helping me with this). Responding to COVID-19 required all hands on deck; empowering everyone to contribute — regardless of their organization — was absolutely critical to our delivery.
4. Choosing the right tools matters.
Working on ENx also shifted my understanding of what makes a tool useful. With increased government guardrails on how budgets could be spent, it was a nontrivial task to determine who could access a shared tool and which organization would foot the cost. Furthermore, because our team and stakeholders came from a wide variety of disciplines and sectors (i.e., public health, engineering, private sector, government, nonprofit), each person had a different set of tools they were willing and able to use. This diversity of backgrounds meant that traditionally straightforward tasks, such as setting up a single product backlog for ENx, became much more challenging, with different vendors and partners incentivized to use their own tools to track their portion of the project, and choosing tools that not everyone could access created information silos that slowed down the team. Therefore, I learned to keep a closer eye on finding the tool of least resistance and highest returns for the larger team. Sometimes, a product roadmap on Google Docs that everyone can see is worth much more than a robust product management system only a few can use.
5. If it’s not sustainable, it doesn’t work.
Government systems can last for decades, and this makes product sustainability absolutely critical. However, strong tensions exist between long-term functionality and immediate delivery across all levels of development in the public sector. For example, on a staffing level, many volunteers and Colorado employees serve limited terms on projects in a shared services model, resulting in a substantial amount of knowledge loss and need for short-term delivery as individuals roll off teams. On a logistical level, bureaucratic constraints and access blockers increase the time cost of most tasks in the public sector. When everything takes time and you don’t have a lot of it, it becomes difficult to invest in anything further down the line. Our team still discusses how we can solve for this and adequately embed sustainability into the fabric of government product development, especially when rapid short-term delivery does matter — particularly during a pandemic. The best answer I’ve come up with thus far is to bake the sustainability of my work into my definition of success. I can call my work a success only if the person after me can maintain what I built.
In summary, although the pandemic sped up the urgency of work in the government, it didn’t revolutionize the complex environment we were developing within. I also realized that most things in government — from team culture to tool selection — require far more time and intentionality to do well, but in return, the resulting impact is incredibly high.
Our work is now open sourced!
Snapshot of our open sourced repository on automated metrics
After our launch of automated ENx metrics, we open sourced our repository here for public view. In the spirit of cross-state collaboration, we invite other states and government entities to reuse our work as much as is helpful, and we encourage others to share their work as well.
Thank you to California and Washington, who collaborated closely with us to share and open source this work. Thanks are also due to the entire Western States Learning Collaborative Pact, the Centers for Disease Control and Prevention, the MIT Lincoln Laboratory, the Linux Foundation for Public Health, Apple, Google, our incredible partners, the Association of Public Health Laboratories, and all the states who showed up to collaborate with us on ENx.
Finally, as I wrap up my rotation at CDS, I’m constantly reminded of how many good people are working day and night to deliver services I used to take for granted. Shipping software in the government may be incredibly challenging, but it remains a critical opportunity to improve how the State serves millions of people. The time I spent here was an honor, and I’ll be taking this experience with me to whatever I do next.
Special thanks to the people and organizations that made this work a reality — Janell, Thad, Ginger, Karyn, and Casey from Colorado; Steve from the US Digital Response; Nat and Chris from the US Digital Service; Zach and Austin from Google; Randy and Michael from Apple; Joni from Schmidt Futures; Bryant, Bill, and Daniel from Washington, Mohsen and Eliah from California — and many more!
This article was written by former Colorado Digital Service member and Schmidt Futures Associate Product Manager Michelle Park.
