Biomedical doctoral students’ research practices when facing dilemmas: two vignette-based randomized control trials – Scientific Reports

We conducted two vignette–based randomized trials embedded in an online survey (NCT04263805 and NCT04263506, registration date 11 February 2020). A vignette can be defined as a brief story or scenario in which participants are asked a set of questions to examine their research practices. In this study, the vignettes were scenarios of research dilemmas that could result in DRPs. We defined DRP as “research practices other than Fabrication, Falsification, and Plagiarism (FFP) that are clearly detrimental to the research process”⁹.

This project was funded as part of the Marie Sklodowska–Curie European Joint Doctoral Training Program on Methods in Research on Research (MiRoR—http://miror-ejd.eu/). This study was approved by the Institutional Review Board of the Inserm (Comité d’Évaluation Éthique, IRB 00003888, Reference Number: 19-587). The study did not collect any identifying information from participants and was conducted in accordance with guidelines and regulations. The protocol is available at https://doi.org/10.5281/zenodo.7387752.

Vignette development

First, we conducted a literature review to identify and classify different types and examples of dilemmas that PhD students may face and could result in DRPs. Two authors (DB, MO) searched PubMed on March 15, 2018, using “Questionable Research Practices[tw]” (including forward citations), yielding 54 articles. An additional 15 were obtained from consultation with experts within the Methods in Research on Research (MiRoR) consortium. Five authors (LB, DB, MO, MG, and MKS) extracted data from the sources using a standardized data extraction form, and 286 DRPs were identified. These were then classified by two researchers (DB and MO) according to the research phase in which they may occur: (i) hypothesis and study design, (ii) conduct (data collection/handling), (iii) analysis, (iv) reporting, (v) publication (e.g. authorship issues), and (vi) others (e.g. policies, ethics, and supervision/training). Disagreements were resolved through discussion with a senior researcher (IB).

To supplement the literature review, thirteen doctoral students from the MiRoR project conducted semi-structured interviews with colleagues after undergoing training in qualitative interviewing (http://miror-ejd.eu/second-training-event/). Doctoral students were located in six countries in Europe: Belgium, Croatia, France, the Netherlands, Spain, and the United Kingdom, and eligible participants (i.e. biomedical PhD students) were identified for interview from professional networks within their current country of residence. Participants engaged in semi-structured interviews either face-to-face, by phone, or via videoconferencing according to the circumstances and preferences of the interviewees. A semi-structured interview guide was developed by two researchers (VN and KG) with guidance from a senior researcher (DH) to support the interview process. Interviews were conducted in English between July and September 2017. All participants provided informed consent prior to the interviews and provided audio recordings. There were no restrictions on age, gender, year of PhD study, or level of research experience. The interview guide covered three main sections (Appendix 1).

1. (i)

   Interviewees’ academic background and level of research experience,

2. (ii)

   Interviewees’ perception of DRPs which is reflected by how they define DRPs,

3. (iii)

   Interviewees’ experiences with DRPs in their research environment and personal experiences of DRPs in their own research.

The interviews were transcribed verbatim and thematically analyzed using NVivo V.11.²⁰, following the method outlined by Braun and Clarke²¹. All transcripts were coded independently by the researchers who conducted interviews. Based on the initial codes, two researchers (VN and KG) developed a coding scheme with themes and subthemes. After consulting senior researchers (IB and DH) who read a random sample of three transcripts to ensure that the developed themes fit the data, the codebook was refined, and a final version was developed.

Through discussions within the authorship team and consultation with experts from the MiRoR Consortium, ten DRPs of interest were selected that were (i) relevant to all biomedical disciplines and (ii) common to PhD students in biomedical research. Three researchers (VN, CS, TVT) collaboratively developed a case vignette for each of the 10 DRPs (Table 1). For each case vignette, the researchers created a dilemma in which the main character was a PhD student. The dilemmas concerned different research phases (e.g. conducting, reporting, and publication), each of which could result in a DRP. Three senior researchers (IB, DH, and AN) reviewed and revised the case vignette. To increase participants’ engagement with the content, the vignette was expressed as a short comic strip summarizing the situation, and the survey was translated into five different languages (English, French, German, Italian, and Spanish) by native speakers of the authorship team.

Trial design

The randomized trials were embedded in an online survey. Each participant answered two vignette-based questions with two different dilemmas in the trial. The first vignette detailed a dilemma without environmental factors, aiming to examine the participants’ baseline research practices. For example, should the PhD student add the head of the department as an author of the PhD student manuscript while he/she was not involved in any part of the article (see Table 2)? In the second vignette, participants were randomized to receive the vignette with or without an additional sentence describing an environmental factor (i.e. experience of postdoctoral researchers or supervisors’ opinions) that might influence students’ decisions (Fig. 1).

Participant

To recruit participants, we used the 2019 World University Rankings from the Times Higher Education²² and QS Top Universities²³ ranking to identify the top 10 universities in the European region (as defined by the World Health Organization)²⁴. We then searched university websites for administrative contacts related to any graduate programs in the biomedical sciences (e.g. pharmacy, medicine, public health, psychology, genetics, etc.). We also used the personal contacts of researchers within the MiRoR consortium to contact administrators of doctoral programs in their own universities. We sent semi-personalized email invitations to explain the purpose of the study and included the link to the survey (Appendix 2). We also used Twitter to disseminate the survey using the networks of the authors and the MiRoR consortium.

Environmental factors evaluated in the RCTs

We tested the impact of two environmental factors on PhD students’ research practices in two separate RCTs:

(1) Exposure to a post-doctoral researcher who experienced a similar situation and chose the DRP; (2) exposure to a supervisor who was consulted and did not object to the DRP.

1. (1)

   The first trial explored the impact of exposure to an environment in which a postdoctoral researcher described the experience of committing a detrimental practice in a similar situation vs. no exposure to this environmental factor (no consultation with colleagues).

2. (2)

   The second trial explored the impact of exposure to an environment where the supervisor was consulted, but the supervisor did not object to the detrimental practice (i.e. the supervisor agreed with the student’s decision) vs. no exposure to this environmental factor (no consultation with the supervisor).

An example of a vignette is provided in Table 2 and an additional example can be found in Appendix 3.

Outcomes

Participants were asked to indicate the solution that they preferred on a semantic differential scale, rated from definitely (5) to maybe (1) a preference for solution A or from maybe (1) to definitely (5) a preference for solution B. The DRPs were randomly reported as solution A or solution B to avoid bias related to the presentation of the scale. We derived a score from − 5 to + 5, where positive scores indicated the choice of DRP and negative scores indicated good practice or no DRP.

Sample size calculation

In each randomized trial, participants responded to a first vignette without environmental factors and then, after randomization, a second vignette with or without environmental factors. To assess a mean difference of 1.0 for the outcome between groups on a scale from − 5 to + 5 with a standard deviation of 2.5, for each trial, a sample size of 264 was needed to detect the effect size of 0.4 with a power of 90% and α risk of 5%. Accounting for a 10% attrition rate, we required a sample size of 300 participants (150 in each arm) so that each vignette was evaluated 15 times in each arm.

Randomization

For each trial, an IT manager generated a random assignment sequence using a block of 10; the list was not disclosed to the investigators. Allocation concealment was ensured using a computerized random assignment system. Randomization was programmed and piloted. Participants who did not complete the survey were replaced by other participants, whereas the intervention and control arms were ensured to be well balanced across vignettes. When the required sample size of the first trial was reached (i.e. 300 participants), subsequent participants were enrolled in the second trial.

Blinding

It was not possible to blind participants to the intervention. However, the participants were blinded to the study hypothesis. All the participants were informed of the first objective of the study, which was to explore the research practices of PhD students facing dilemmas. However, they were not informed of the second objective of evaluating the impact of environmental factors on their practices.

Analysis

Statistical analyses were performed using SAS software, version 9.4 (SAS Institute, North Carolina, US) and R 4.4.1.²⁵ The analysis population was comprised of participants who completed the trial, i.e. who read two vignettes. The primary outcome was the mean preference score. The difference in means between the groups in each trial was analyzed using an intercept and slope random linear model with a fixed group effect, random vignette effect, and vignette-group interaction effect (assuming an unstructured covariance matrix). Inferences were based on restricted maximum likelihood. Estimated coefficient on fixed group effect represent difference in means and its standard error allow to calculate 95% confidence intervals (CIs). Random effects allowed accounting for within-group clustering (i.e. each vignette was evaluated many times in each arm), between-group clustering (i.e. pairing of vignettes used in the two arms of the trial), and the possibility that scores may vary between vignettes.

The secondary outcome was the proportion of students who chose the DRP (i.e. rated at least 1 on a scale of − 5 to 5 for the DRP). For secondary outcomes, we estimated both the absolute risk difference of proportions of students who chose the DRP and relative risk (with 95% CIs) with a log binomial model under a GEE framework assuming an exchangeable correlation structure in the context of clustered data due to vignettes. P < 0.05 was considered statistically significant.

Furthermore, we conducted a post hoc analysis to examine factors associated with DRP. Vignettes assessed before randomization were pooled from the two trials. The factors considered were determined a priori: gender, research experience, training on research integrity, and pressure to publish. We estimated odds ratios (with 95% CIs) using univariate and multivariate logistic models under a GEE framework, assuming an exchangeable correlation structure in the context of clustered data due to vignettes.

Data sharing

The dataset will be shared on an open-access platform (https://zenodo.org) after the article has been published. The corresponding author could provide the dataset upon request.

Ethics approval and consent to participate

This study was approved by the Institutional Review Board of the Inserm (Comité d’Évaluation Éthique, IRB 00003888, Reference Number: 19-587).