“It doesn’t appear to me that it poses such a huge security risk. I don’t work for the state department and I am not sending sensitive information in an email. So if you want to steal the message about I made blueberry muffins over the weekend then go ahead and steal that.” (A study participant)

As researchers steeped in quantitative methods, during our work studying cybersecurity and trust, we quickly realized that the quantitative tools we were most comfortable utilizing were inadequate to analyze this type of data. We were not really prepared for qualitative data analysis with data like the quote above—how do you even begin to analyze this in a systematic, rigorous way?

This article outlines our experience as a multi-disciplinary team studying user perceptions of and experiences with cybersecurity. We trace our journey from mutual skepticism, to understanding, to acceptance using illustrations from our data. We also discuss our learning along the way—including the many “aha” moments we had. Certainly one of those aha moments was the realization that qualitative methods are an important and valuable tool for data collection and analysis in the cybersecurity space. Having now worked together on multiple qualitative studies, we believe sharing our story can help others who want to explore the use of qualitative methods in this space. We also offer a methodological roadmap for how to design and conduct rigorous qualitative studies that explore and examine the user experience of cybersecurity.

Beginning the Journey: Seeking Enlightenment

Our multi-disciplinary team includes a computer scientist, a cognitive psychologist, a human factors psychologist, and a qualitative researcher/sociologist. As a team, we each bring unique disciplinary and methodological perspectives to the research we do. These differing perspectives provide both opportunities and challenges in our work together, often causing us to experience dissonance in the process as we move in and out of our individual comfort zones.

Our initial team was composed of three researchers who worked at the National Institute of Standards and Technology (NIST), an overwhelmingly quantitative environment. For this study, we set out to examine user mental models in relationship to cybersecurity and privacy, focusing on people’s perceptions about and experiences with cybersecurity. We developed a set of questions and used them to interview 40 people (22 women and 19 men; 27 from the Washington, D.C., metropolitan area and 13 from central Pennsylvania). Initially, we counted how many times and how many participants responded in particular ways as demonstrated in Table 1.

Table 1. Counting coping mechanisms

Coping Mechanism	Number Reporting
It is the bank or online store’s responsibility to protect my personal information	13
Shop at established or trusted online stores	13
Look of an icon, trust mark, or HTTPS	11
Don’t share my personal information (for example, on social media)	10
I don’t store any credit card information online or I use a separate card for online purchases	7
Nothing has happened so nothing will happen/desensitized	6
Change my password or use more complex passwords	5
No one would want my information or I have no money to steal	4
Always log off or turn off my computer	4
Don’t open emails that I don’t recognize	3
Own a Mac	3
Anti-virus protects me	3
Only use protected networks for online transactions (for example, never do any transaction while at the coffee shop)	2
Save files on an external drive	2
IT department keeps my computer safe	2

Simply counting responses resulted in a very quantitative analysis that we knew was incomplete, but we couldn’t define the incompleteness—it just felt wrong. We had a lot of data, but were unsure of how to proceed. In part, it was because we came to a roadblock when confronted with data like the statement at the beginning of this article. While there was only one participant who mentioned blueberry muffins, other participants made statements seemingly unrelated to cybersecurity and we were confounded by how to analyze them.

Our first instinct was to read about qualitative research. As experts in our fields who understand different methodological processes, we believed we should be able to read something, comprehend it, and do it. But after reading and researching the topic we still didn’t understand how to move forward in our analysis. It was time to bring in a qualitative expert, someone who could help us figure out how to decipher the narrative data we had collected—including a statement about a blueberry muffin.

The qualitative expert (Sandra) began by teaching us a three-week course on qualitative methods, using our data as exemplars. We (the initial team) gained a strong theoretical background in and developed hands on skills with qualitative research. However, in many ways, we remained skeptical—especially about the value and rigor of qualitative analysis. We still had many misconceptions; we saw the parts but couldn’t see the worthiness of the process and we continued to often think “this is all made up.”

Sandra, too, remained skeptical about if and how researchers so steeped in a positivist paradigm (that believes knowledge and reality are objective and measurable) could accept an interpretivist (that believes knowledge and reality are subjective and rely on interpretation) approach to data, and she questioned what a qualitative researcher had to offer a measurement laboratory like NIST. In addition, the incorporation of a new discipline and an interpretivist researcher served to disrupt a previously cohesive team of positivist researchers. Below we trace our ongoing journey, some of the “aha” moments we experienced, and what we learned along the way.

Going Back to Move Forward: The Recursive Nature of Qualitative Work

From the beginning, Sandra insisted that all of us go back and identify the problem, purpose, and research questions that were driving the research. We had not done this identification formally before beginning the research. We needed to go back to our hypotheses and see if the data supported them or not. Surprisingly, in this exercise, we discovered that we all had slightly different ideas about what the problem, purpose, and questions were. We had long conversations, often with a fair amount of disagreement, as we defined the foundations of our work. While our hypotheses were similar, some of our assumptions about the research were not.

Clearly define your problem, purpose, and questions and use them to guide each phase of your work

The development of our questions, even after the fact, provided us with the direction to develop coding that best fit the data, our goals, and an analysis process. We were so embedded in hypotheses that it was hard for us to remain open to the data and to not make predictions. This process required a shift in our mental model of what research is and how it is done. Moving back and forth from questions to data is part of the recursive nature of qualitative work, something that was different than what the quantitative part of the team was used to doing.

We also had to go back and define terms, shown in Table 2, since we found early on that we were not always using terminology in the same way.

One aha moment for Sandra was when, in a coding session, she said, “I think this might be significant.” The team responded by asking how many participants had a similar comment and how we could show significance. To our team, significance refers to the number that expresses the probability that the result of an experiment could have occurred by chance. To a qualitative researcher it refers to trends or ideas in the data that are not obvious.

We were operating with different vocabularies, and we found that our use of terms like data reduction, coding, significance, and reliability needed to be revisited often.

Table 2. Research terminology is not the same

Term	Quantitative Definition	Qualitative Definition
Data reduction	The method used to compute the measure of central tendency and to characterize the variation in the data	The act of reducing and/or reconfiguring data in an organized and meaningful way
Coding	To write computer code	To apply a label to sections or chunks of narrative or visual data that captures the essence and/or salient features so that like chunks can be grouped, compared, and/or manipulated
Significance	The number that expresses the probability that the result of an experiment could have occurred by chance	Identification of trends or ideas in the data that are not obvious, but which point toward a new, emerging, and/or interesting understanding of the data
Reliability	The overall consistency of a measure	The dependability of the process and product of the research, which can be provided through an inquiry audit or other means of demonstrating consistency
Validity	The method of power analysis used to detect a relationship	The trustworthiness provided by transparency in study design and process, including the delineation of analytic processes that are systematic and rigorous
Sample size (n)	The number of samples necessary to reach a certain statistical power	The number of participants or cases needed to reach saturation in the data

Make sure you’re speaking the same language

This includes clearly defining (operationalizing) your codes and analytic processes.

Our analysis process began with the creation of our codebook, which included the definition, when to use it, when not to use it, and examples for each code. An example from our codebook demonstrates the level of detail associated with each code.

[bluebox]

Characteristics of Cybersecurity

• Constantly changing. The ways in which approaches/reactions to or understandings of security change quickly based on changes in technology, shifting threats, changing policies/directives, and/or other circumstances.
• Whose responsibility? Questions about and descriptions of who should make decisions about and be responsible for cybersecurity.
• Technical policies. Practices and procedures that have been standardized and have become “official.”
• Control. Participant descriptions of how and when they feel they can exert influence/power over a system.
• Complexity. The intricate and complicated nature of the system(s), infrastructure, or technology that sometimes makes security difficult to see (it is often hidden), implement, or navigate.
• Connectedness. The ways in which so many systems are interrelated and connected through the internet, which has implications for security.
• Bigger than the individual. The ways in which security has the potential for a collective effect that goes beyond one person.
• Myth of security. Shared cultural beliefs about what can be protected, the extent to which it can be protected, and how it is protected that are not necessarily based on evidence.
• Protection. Related to the need for people to guard infrastructure, assets, information, technology in a computing environment.
• Technology. Tools utilized to protect infrastructure, assets, information, etc., in a computing environment.

[/bluebox]

Operationalizing codes highlighted many of the disciplinary strengths and differences of the team. The psychologists defined things very differently than the sociologist, and all three of them saw things differently than the computer scientist, who often focused on the technical. Our most important tactic in this process was to discuss, discuss, discuss until, as a team, we were satisfied that we all understood each code and how to use it. Once we had operationalized a priori codes, we each coded the first four transcripts to determine the convergence and/or divergence in our use of the codes and to see what new codes emerged from the data. Coding each transcript by hand allowed us to express ourselves using colored pencils, highlighters, and even boxes and arrows to identify text with codes shown in Figure 1.

An aha moment for the human factors psychologist came during this early coding process. “Even though our team was multidisciplinary, I was amazed how similar our individually coded data was.”

When we were comfortable with the codes and with our ability to consistently apply them, we assigned each transcript to two members of the team. We met regularly to review our coding, revise the codebook as needed, and talk through initial ideas about the data.

The cognitive psychologist’s first aha moment came during this process when he recognized how a participant’s analogy of a blueberry muffin was connected to cybersecurity and how it represented something larger—what we ended up calling “lack of value”—a code we used to represent when participants talked about the idea of not having anything others would want to steal.

Once we had coded all the transcripts and our memos, we worked through the data again, this time focusing on those codes that had emerged during our first-pass coding (like lack of value). We also began to discuss the relationships we were seeing amongst the codes and within the data, for example the relationship between changes in technology and changes in society.

Eventually, we began to define this relationship as “Brave New World” and we went back to the data to see if and how this theme (a more abstract concept than a code that is just a label for a chunk of text) played out in the data. Another example was when we began to discuss the possible relationship between Loss of Control and Resignation related to cybersecurity, and we recognized how these codes may be related to the larger concept of security fatigue.

Be willing to take a “leap of faith” in your interpretation

Recognize that this leap relies on having faith in the strength of your operationalization and coding processes.

Qualitative data analysis is about the shift from the particular to the general, from the codes that are rooted in the data to a more abstract understanding of how codes connect to each other in a web of meanings—a web constructed by the researcher. This process of abstraction and (re)construction often involves taking a leap of faith—which sounded very risky to the quantitative members of the team. In their disciplines, they were taught not to take risks, not to do anything that you cannot verify and test. They were “risk averse,” according to the human factors psychologist, and here was an “expert” telling them to take a risk. What we [or each of us] slowly began to recognize as we went through the process of analysis, was that in fact we were ready to take a risk—to take a “leap of faith” in the analysis—because we trusted the process we had put in place, we trusted the codes and coding structure we had created, and we trusted each other and the work we had done together.

Our initial exploration of security fatigue as a concept in the data reflected this ability to accept abstraction and take a leap of faith. When Sandra first mentioned that she felt a sense of “fatigue” in participants’ responses, especially when looking at the codes of Resignation and Loss of Control, the rest of the team was not quite sure. Participants did not mention being fatigued, so how could we make that interpretation? However, the more we discussed the concept and the ways in which data coded as Resignation and Loss of Control might contribute to a sense of fatigue, the more we were convinced this was an important finding. It was our ability to take this leap of faith that allowed us to explore and ultimately accept findings like Security Fatigue.

Moving Forward: Finding Our Voices as Qualitative Researchers

While this process sounds neat and tidy when we write about it here, it was definitely messy as we navigated through it. Initially, the quantitative part of the team wondered why we had to keep going back to the data and when the analysis would be complete. In their world, you put your data in, ran your tests, and had your analysis. The notion of saturation was not one they understood. For this process to work, we needed to respect and trust each other—and the process. Each of us needed to be willing to put ourselves in the role of a non-expert and respect the expertise of someone from another discipline with ideas and perspectives very different than their own. The aha moments we experienced were possible because we respected each other’s ideas and began to trust each other—which allowed us to trust the process.

Develop respect and trust; they are key components in this journey

Initially, Sandra did not trust that our team believed in the value and rigor of qualitative research. We acknowledged her as the expert, and we wanted to put our trust in her, but she needed to show us how it could/would work. Faith is trust without evidence, and our team needed the evidence. While we were cautiously trusting in the beginning, the evidence came for us once we had gone through the process, experiencing each phase, and in the end seeing how each piece contributed to the overall rigor of the process.

The computer scientist’s aha moment came at this juncture, when she recognized there was validity, reliability, and rigor in the process. Initially, she saw qualitative analysis as just “hand waving” until she saw the formality and systematic nature of the process. But, according to her, “You must go through the entire process with an expert; otherwise you don’t get an appropriate appreciation for the process and the rigor.” We knew how to define and recognize rigor in our own fields, and trusted it; we did not question it. Now, we needed to learn how to translate that trust into this new space. To do that, we needed an open mind; we needed to move beyond the confines of language and thinking within our own disciplines.

Put yourself in the role of learner

Having an open mind is necessary to let learning happen. Putting oneself in the position of the learner, of the non-expert, is difficult. In this situation, the NIST quantitative team were learners on multiple levels—we were learning about qualitative research from an expert, Sandra, and we were learning from our participants who could also be viewed as experts with respect to their own perceptions and beliefs about cybersecurity. There was a parallel here between keeping an open mind about the methods so we could learn from a seasoned qualitative researcher, and allowing the data to speak for itself so we could learn from our participants. As we became more comfortable in this role of learner, we found that it became easier to recognize and make connections and relationships in and amongst the data and the codes.

As researchers steeped in a positivist paradigm that values numbers over words (and perhaps over each individual), we needed to learn that it was okay to “tell the story” of our data using the participants’ own words and voices rather than the numbers. Reporting how many people said something, how many times a particular code appeared, or how many codes we utilized limited the story we could tell and did a disservice to our participants. The power here is no longer in the numbers, it is in the story(ies). This meant a shift not only in the research process (what we did and how we did it), but also in the way we report on it, the way we tell the story.

This shift is exemplified by the two papers that have resulted from this research effort to date. The first paper, “Basing cybersecurity training on user perceptions” (published in the journal IEEE Security and Privacy) followed a quantitative approach. The second, “Privacy and Security in the Brave New World: The Use of Multiple Mental Models” (published in the journal International Conference on Human Aspects of Information Security, Privacy, and Trust) resulted from a qualitative approach. While the two papers complement each other, the contrasting styles and the differing insights from the same data are striking.

The Path That Brought Us to This Place

An aha moment is a moment of interior discovery, the moment when the picture comes into focus. In a team discussion, the computer scientist articulated what this meant for her. “When you reach an aha moment and the final piece slides into place, I don’t doubt anymore, just like I don’t doubt p values.”

As our aha moments increased, we became believers in the power and necessity of qualitative research in the work we do related to cybersecurity. How can we understand what users know, believe, and do if we do not talk to them? However, this means more than just utilizing qualitative tools for data collection and analysis; it means changing everything, including our mental model of what research is and how it is done.

While having an open mind is a necessary component here, it is perhaps not enough to change our mental model about research, which requires experiencing ongoing dissonance that continues to challenge assumptions that we take for granted. Experiencing something one time is not enough to shift or change a mental model; instead it is a recursive process that involves revisiting ideas regularly.

Be prepared to experience ongoing dissonance

While we do believe that qualitative research is a valuable methodological paradigm for exploring cybersecurity, as researchers, we continue to struggle with this dissonance as we shift our mental model about research. This dissonance appeared even as we wrote this article—as we discussed the best way to refer to ourselves as writers/researchers in the article. Sandra’s sociologist’s tendency was to refer to each of us by name, personalizing the ideas, while the quantitative team was more comfortable using more formal, third-person references. As we discussed the options, we recognized the ways in which this very discussion reflected many of the ideas in the article, and highlighted the difficulties in breaking mental models that we had each spent years building.

In this first study, we learned a lot about each other and about the process that worked best for us. In the end, we all recognized that we have a lot to learn from each other and a lot to contribute to the research process. We became more comfortable with dissonance and with NOT knowing, as we learned to trust ourselves as well as the process. This process included changing how we design a study, collect data, analyze data, and write a paper. While there are certain pieces that we see as germane to this process, we also recognize that they may be different for other teams of researchers. That is perhaps a key piece of learning about qualitative research. However, one constant is the importance of putting in place rigorous, systematic, and transparent research practices. We look forward to our continued work together and the dissonance we will experience as we learn with and from each other and our participants.