Navigating the Regulations: UX Research in Government

In the last decade, the delivery of government services has become increasingly reliant on technology as a primary channel for people to engage with services. However, the creation and management of that technology may not always be considered a strategic initiative for a government agency. As agencies rightfully focus first on programmatic development and delivery, they risk seeing technology as supplemental to their work.

The resulting technological system and services then fall into the trap of overly reflecting the business, resulting in Conway’s Law, which states:

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.”

Without an intentional strategy, a technology system will naturally mimic the way the organization interacts, rather than how the public might need or expect to engage with it. When a person needs support for their healthcare, food, housing, and childcare, each of those services are offered by different agencies with unique processes. This puts the burden on the individual to learn how their government is organized and operates, which can prevent them from participating in the services they need (see The Time Tax by Annie Lowrey).

However, there’s a growing movement in government (and beyond) to focus technology and services on the needs of the person that will ultimately engage with them (see Is User-Centered Government Really Attainable?).

Government agencies can improve their technology and services by regularly engaging the people they serve through User Experience Research (UXR). UXR methods can be generative in nature, like site-visits and diary studies (i.e., examining a person’s lived experience), or evaluative, like usability tests or heuristic evaluations (i.e., industry-standard practices to assess the objective quality of a product). While UXR is built upon the scientific method (e.g., hypothesis, observation, evaluation), it does not often follow the same strict protocols.

While conducting UXR, it’s important to understand what regulations govern the collection of data. This guide is meant to give an introduction to how two regulations, Human Subjects Research and the Paperwork Reduction Act, might impact a UXR study.

Stay tuned at the end for best practices while conducting UXR in government.

Human Subjects Research

History

Prompted by human rights abuses in the name of scientific exploration in World War II and during the Cold War, Congress requested the United States Department of Health and Human Services (previously known as Health, Education and Welfare) in 1974 to define rules around research activities that involve human subjects. In 1991, the Federal Policy for the Protection of Human Subjects (known as the “Common Rule”) was codified by 15 federal departments and agencies that conduct research activities. The regulation applies to any study that is occurring under the purview of or sponsored by funds from one of the participating departments or agencies. Each entity is responsible for reviewing research and ensuring compliance through a Human Research Protection Program (HRPP) or an Institutional Review Board (IRB). The regulations are enforced by the Office for Human Research Protections (OHRP) in the Office of Public Health and Science (OPHS).

Definition

To be subject to the regulations of Human Subject Research (HSR), a study must meet the definition of research:

“A systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge.”

In this context, “generalizable knowledge” is information that can be applied to a population or situation beyond the participants in the study. It includes one or more of the following concepts:

1. The information contributes to a theoretical framework or an established body of knowledge.
2. The primary beneficiaries of the study are other researchers, scholars, and practitioners in the field of study.
3. Publication, presentation or other distribution of the results is intended to inform the field of study.
4. The results are intended to be replicated in other settings.

HSR only applies to the study of human subjects, which refers to a living individual who has provided private information or biospecimens to be studied or analyzed. Additionally, “data studies” or “secondary use research,” (studies that use data originally collected for the purpose of another study), are often exempt from the regulation.

If a study meets the definition of research under HSR, it must be approved by an IRB or HRPP before any work on the study may begin. During the study, the researcher is required to adhere to the approved protocol and follow any additional directives from their governing body. Approval for a study can be withdrawn at any time. For example, a pilot study might not initially meet the definition of HSR; however, if its sample size changes, HSR requirements may then be applicable.

Though this regulation exists at the federal level, HSR can still apply to state and local governments if they are receiving funding from a participating department or agency. If a research study is subject to HSR and is found to be operating outside of this process, the entity may be denied federal funding in the future.

Exemptions

The federal regulations governing HSR include exemptions for certain activities, rendering them not subject to the regulation. However, it’s important to note that exemptions do not apply to certain classes of individuals, like prisoners, pregnant persons and children. An exemption does not mean that the study does not qualify as research, rather, it does not have to comply with the regulation. Listed below are a few exemptions that are most applicable to UXR:

A. Methods that include cognitive, diagnostic, aptitude, achievement, survey procedures, interview procedures, or observation of public behavior — including visual or auditory recording — or involve benign behavioral interventions (e.g., brief in duration, harmless, painless, not physically invasive or likely to have a lasting impact) if at least one of the following criteria is met:

1. The information is obtained anonymously.
2. Disclosure of the participant’s responses would not place the person at risk of criminal or civil liability or be damaging to their financial standing, employability, educational advancement or reputation.

B. Secondary research (i.e., methods that use existing data) for which consent is not required as long as one of the following criteria are met:

1. The information is publicly available.
2. The information is collected anonymously.
3. The research is conducted by or on behalf of a federal department or agency using government-generated or government-collected information obtained for non-research activities.

C. Research designed to study, evaluate, improve, or otherwise examine public benefit or service programs.

In most cases, UXR studies do not meet the definition of research because they are not intended to contribute to generalizable knowledge. Even if a UXR study is large enough to contribute to generalizable knowledge, it is often intended to “study, evaluate, improve, or otherwise examine public benefit or service programs,” which is an exemption in the regulation.

When conducting a UXR study, researchers have a responsibility to understand the regulation and the processes of the government entity they are working with. The Office of Extramural Research (OER) under the National Institutes of Health has created a decision tool to help a researcher determine whether or not their study meets HSR definition or exemptions.

Paperwork Reduction Act of 1995

The Paperwork Reduction Act of 1995 is a law governing how federal agencies collect information about the people they serve. The goals for this act are to:

1. Reduce the burden of time for people that are providing information.
2. Ensure that data collected is accurate and high-quality.
3. Limit the collection of private information to only what is necessary. Protect all private information.

If information collection is subject to the Paperwork Reduction Act, it has to go through an approval process that can take anywhere from six to nine months before being approved by the Office of Information and Regulatory Affairs, Office of Management and Budget.

However, the Paperwork Reduction Act also has exemptions, notably:

Direct Observation: Facts or opinions obtained through direct observation by an employee or agent of the sponsoring agency or through non-standardized oral communication in connection with such direct observations.

UXR methods like usability tests and diary studies are considered direct observation and are exempt. Though again, it’s important for a researcher conducting a UXR study to understand how their methods relate to this regulation. Check out this blog post by Erie Meyer where she breaks down the Paperwork Reduction Act and debunks how it can impact the ability to conduct UXR in a government environment.

User Experience Research (UXR)

Conducting UXR upfront and throughout a project lifecycle will increase the likelihood of success as these methods reveal needs, pain points, and constraints of the people who will ultimately be engaged, as well as help to validate assumptions about potential opportunities and solutions. UXR is often used in partnership with modern software development frameworks like Agile to enable fast cycles of learning and iteration.

Conducting UXR in a local, state, or federal government environment is possible and should be prioritized in order to deliver services that are accessible and valuable for the people that need them.

Even if a study is exempt from these regulations, a researcher should still follow best practices in order to protect participant’s rights and circumstances. Listed below are a few recommendations:

Respect the regulations

• Do your research and learn the regulations. Be ready to defend why a study is or is not subject to these rules.
• The term research is often associated with Human Subjects Research. You could use a different term to describe the activity, such as product evaluation.
• When working with populations that have additional protections for purposes of Human Subjects Research, such as pregnant women, prisoners or children, it will be important to ensure any special issues regarding voluntary participation or informed consent are addressed.
• Know your funding source. Any public sector funding source has specific requirements and limitations. Remember that a study funded by a federal source must follow the regulation of Human Subjects Research; otherwise, it can prevent your organization from receiving funding in the future.

Gain consent and prioritize transparency

• Ask permission before you record audio / video or take photos.
• Inform the person that their participation is entirely voluntary and they can withdraw their consent at any time.
• Inform the person why you are collecting data and how it will be used.

Only collect necessary information and always protect data

• Limit the collection of Personally Identifiable Information (PII) and maintain participant anonymity as much as possible.
• Avoid collecting any information that has no clear use.
• Only maintain data in a secure location.
• Limit access to raw or identifiable data to only the people that need to see it.
• Securely delete raw or identifiable data once it is no longer needed.

Avoid biasing participants

• Avoid coercion by only compensating participants for their time and not their answers to the questions (note that it is important to compensate participants).

Be aware of impact

• If possible, send your questions or topics in advance and allow participants to opt-out at their discretion.
• In some cases, research questions might trigger a participant. Consider the impact of the questions and be prepared with resources when appropriate.

Additional Resources

• There are more regulations that could impact a UXR study including the U.S. Privacy Act of 1974. Check out 18F’s guide on privacy and user research.
• To help researchers in the Government of Canada get feedback from the public quicker, the Canadian Digital Service built a tool to create privacy and consent notices. See a faster way to create privacy and consent notices in government.

This article was written by Colorado Digital Service members, Stephanie Cain, Ploy Buraparate, Janell Schafer in partnership with Colorado Department of Public Health and Environment’s Privacy Officer, Joni Koenig.

Interested in following along the work of the Colorado Digital Service? Submit an interest form to stay connected. You can also check out our work at our Colorado Digital Service website, GitHub, Linkedin, or Twitter.