This study employed a Delphi methodology to achieve consensus among experts from diverse fields of industry, policy, and academia. The Delphi process is an established iterative research method designed to gather insights and refine expert opinions through a structured sequence of data collection and analysis rounds (Falzarano & Zipp, 2013). In this study, three rounds were conducted: an initial qualitative phase involving semi-structured interviews, followed by two rounds of quantitative web-based surveys. The study was conducted with Institutional Review Board (IRB) approval, and all participants provided informed consent.
The first round consisted of semi-structured interviews aimed at identifying key themes and issues relevant to the research topic. Experts were selected based on their expertise and representation of the fields of industry, policy, and academia. This diverse knowledge base of participants ensured that the study incorporated a broad range of perspectives. Each interview was scheduled for approximately one hour and was conducted in a semi-structured format to allow for both guided discussions and the exploration of emergent ideas. Interviews were transcribed verbatim, and real-time notes were taken to ensure accurate capture of responses. The data gathered during this phase were analyzed thematically, with emergent themes serving as the foundation for the subsequent survey rounds. This process of qualitative synthesis was critical in generating meaningful, contextually relevant questions for the surveys.
The second and third rounds of the Delphi methodology were conducted using web-based surveys administered through Qualtrics, with data analysis performed using R. Themes from the interviews were used as survey questions, with revisions to wording made to ensure participant anonymity and avoid threats of influence of reputation. The same pool of experts from the interview phase was invited to participate in these rounds, ensuring continuity and consistency throughout the iterative process. Participation in each survey round was voluntary, and experts were informed of the study’s goals and processes at every stage.
The second-round survey was designed to quantify the level of agreement with the themes and insights identified in the interviews. Seven-point scales similar to Likert scales of agreement were used as the primary mechanism for gathering quantitative data, enabling participants to rate their level of agreement or priority for various items. To contextualize the quantitative findings, the survey also included open-ended fields where participants could provide justifications or elaborate on their ratings. This qualitative feedback was instrumental in understanding the rationale behind expert opinions and informed the development of the next round.
The third round built directly on the results of the second round. In this round, participants were provided with aggregated data from the previous survey, including the mean Likert scores for each item. This step allowed participants to consider the collective responses of their peers while maintaining anonymity. Additionally, anonymized justifications from the second round were shared to provide context for differing perspectives. This iterative process encouraged reflection and refinement of responses, ultimately fostering a deeper level of consensus. Items that did not achieve consensus in the second round but demonstrated close agreement, as measured by a standard deviation approaching the defined threshold, were included again in the third round for reevaluation.
Consensus was pre-defined as a standard deviation (SD) of 1.0 or less in seven-point scale responses, with consideration of the most recently published best practices for analysis in Delphi studies (Franc et. al., 2023). This metric was chosen to ensure a rigorous and statistically sound measure of agreement among participants while allowing for diverse perspectives to be represented. The use of a standard deviation threshold provided an objective criterion for determining which items were considered areas of consensus and which required further deliberation.
Throughout the study, the research team strived to protect participant identities, to facilitate experts sharing their candid opinions, uninfluenced by their employers or political winds. Participants were informed of their rights and provided with clear explanations of the study’s purpose, methodology, and potential applications. Informed consent was obtained at the outset, ensuring that participation was fully voluntary.
The use of the Delphi methodology in this study was particularly suited to addressing complex, multi-faceted issues that benefit from expert deliberation. By beginning with qualitative interviews, the research team was able to ground the study in the lived experiences and professional insights of a diverse group of experts. This approach ensured that the themes and questions explored in subsequent survey rounds were both relevant and comprehensive. The iterative survey rounds allowed for the refinement of ideas and the identification of areas of strong agreement, providing a clear pathway to actionable conclusions.
The reliance on a consistent pool of experts across all three rounds was a key strength of this study. Maintaining the same group of participants allowed for continuity and movement toward consensus when feedback and rationale were provided. Additionally, the integration of qualitative and quantitative methods at each stage ensured a balanced approach that leveraged the strengths of both types of data.
