Evaluation of Virtual Prenatal Care for Obstetric Care Delivery During the COVID-19 Pandemic: A Mixed Method Research Study Using the Consolidated Framework in Implementation Research

Background and Objective: Coronavirus 2 (SARS-CoV-2), a novel respiratory virus, rapidly spread, and placed patients at increased risk for short and potentially long-standing medical illnesses. The pandemic necessitated the rapid implementation of virtual prenatal care via telemedicine in obstetrics to maintain social distancing measures. The aim of this study was to assess and understand the patient perspectives of the rapidly implemented virtual prenatal care via a telemedicine model during the Coronavirus disease 2019 (COVID-19) pandemic utilizing the Consolidated Framework in Implementation Research (CFIR). Methods: Following the implementation of virtual prenatal care in March 2020, pregnant patients at a large urban clinic in the southeastern United States completed a 19-question anonymous survey that included open and closed-ended questions on their experience receiving virtual prenatal care via telemedicine or in-person prenatal care from May to December 2020. The survey and mixed-methods data analysis was guided by the CFIR framework. Results: A total of 59 patients completed the survey. One-third (31%, n=18) of the patients found virtual prenatal care to be an acceptable alternative model, and half (53%, n=31) found it acceptable only during a pandemic, preferring to return to in-person visits. Qualitative analysis found that some patients were deterred by limited in-person examinations and uncertainty with the virtual platform, while others appreciated the reduced need for transportation, childcare, and time spent. Conclusion and Global Health Implications: Most patients found virtual prenatal care to be easy to access and an acceptable alternative during the pandemic; however, most would prefer to return to in-person prenatal care visits. Future comparative research studies should examine how, among others, virtual prenatal care versus in-person prenatal care impacts specific maternal and fetal outcomes.


Background of the Study
Coronavirus 2 (SARS-CoV-2), a highly contagious novel respiratory virus that causes COVID-19, was first identified in December of 2019 1 and rapidly spread to become a global pandemic by the spring of 2020.With the rise in COVID-19 cases, the Centers for Disease Control and Prevention (CDC), along with national, state, and local governmental officials began encouraging and implementing social distancing guidelines in an attempt to limit the spread of the virus. 2 These new guidelines immediately impacted the care of millions of patients worldwide, particularly pregnant women.
During pregnancy, women experience several physiological changes that are thought to increase their susceptibility to COVID-19, including changes to the cardiorespiratory and immune systems. 3,4Most pregnant women only experience mild symptoms, predominantly described as a cough and fever, but may also present with myalgias, dyspnea, and fatigue. 4,5Yet, emerging data showed COVID-19-positive pregnant patients experienced increased rates of maternal, fetal, and perinatal complications.These complications included increased rates of cesarean delivery, fetal growth restriction, fetal distress, pre-term birth, pre-eclampsia, pre-mature pre-labor rupture of membranes, as well as intensive care unit (ICU) admissions, miscarriage, and death. 6Therefore, weighing the risks and benefits of traditional prenatal care, healthcare providers began to adopt and implement virtual prenatal care models to minimize exposure to patients and themselves. 7e use of virtual prenatal care facilitates and provision of basic antenatal care to pregnant women while reducing unnecessary hospital visits and potential exposures to COVID-19 among low-risk patients and with potential for modification among high-risk patients. 7,8Additionally, telehealth services to pregnant women during the COVID-19 pandemic offered the potential to decrease levels of prenatal distress and anxiety. 9It was previously found that provider sentiments toward virtual prenatal care were generally positive; however, while many felt that telehealth increased access to care for most patients, providers noted barriers to access experienced by patients with Medicaid. 10Providers cited benefits including limiting COVID-19 exposure and ensuring continued access to care during the pandemic.Barriers included technical difficulties, limited data on the use of telehealth in obstetrics, and increased difficulty with language barriers and translation services. 10

Objectives of the Study
The aim of this mixed-methods study was to assess and understand the patient perspective of the rapidly implemented virtual prenatal care via a telemedicine model during the COVID-19 pandemic utilizing the Consolidated Framework in Implementation Research (CFIR).

Overview
This study used the Consolidated Framework for Implementation Research (CFIR) 11 to assess the implementation of virtual prenatal care from the patient's perspective.Eight key constructs from three CFIR domains were employed, including (1) characteristics of the individual (knowledge and beliefs about the intervention, self-efficacy, individual stage of change, other personal attributes), (2) intervention characteristics (relative advantage, complexity, design quality, and packaging), and (3) outer setting (patient needs and resources) (Table 1).Questions were developed to assess the implementation of virtual prenatal care based on the CFIR constructs from the patient's perspective.

Survey Instrument
Our survey was broken down into three categories: (1) experience with telehealth visits, (2) needs in prenatal care, and (3) overall opinion of telehealth visits all guided by the CFIR framework as described above.Questions had open-ended, yes or no, or Likert scale answers to create the mixed-methods survey.An example of an open-ended question was "What advantages do obstetric telehealth visits have compared to traditional in-office visits?" versus the Likert scale option for "How confident are you that you can continue to do telemedicine visits if you would like?" with answers from extremely confident to not confident.

Data Collection
From May to December 2020, pregnant patients receiving care at the University of South (USF) Florida Department of Obstetrics and Gynecology were recruited during prenatal visits, both inperson and via telemedicine.Pregnant patients received a one-time anonymous questionnaire via a web link in their virtual prenatal care survey invite and flyers with QR code links were posted throughout the clinic.Additionally, paper surveys were handed out at the time when participants scheduled in-person prenatal visits.All pregnant patients seen were invited to participate regardless of gestational age.The questionnaire consisted of demographic characteristics, open-ended, and closed-ended questions related to participants' virtual prenatal care visits which began in March 2020.Participants were eligible to complete the survey if they were over the age of 18, receiving prenatal care at USF Health, and had participated in a virtual prenatal care visit.Exclusion criteria were participants under the age of 18, not receiving care at USF Health, or who had not participated in a virtual prenatal care visit.

Ethical Approval
This study was reviewed and determined to be exempt by the University of South Florida Institutional Review Board.

Mixed-Methods Data Analysis
To analyze the open-ended responses, the evaluation team developed an initial codebook using a priori structural codes based on the CFIR constructs.Ten percent of questionnaires were independently coded by two researchers (C.R. and K.F.) to assess inter-coder reliability, resulting in a kappa of 0.86, 12 indicating good agreement between coders.Coding discrepancies were then resolved and agreement on the final codebook was achieved.One of the researchers (C.R.) coded the remaining questionnaires.Thematic analysis was used to analyze open-ended responses, 13 and data analysis was conducted using MAXQDA 2020 software 14 (Table 2).Data saturation was determined when no new information emerged from participants' openended responses.
Trustworthiness was established through peer debriefings (credibility), independent coding by two research members resulting in a good kappa (reliability), and the use of participant quotes to represent themes (confirmability). 15,16Furthermore, patient evaluation findings were grouped by participants' preference for obstetric virtual prenatal

Characteristics of Individuals
Knowledge and Beliefs about the Intervention (Patient Preference) Participants' attitude and preference towards telemedicine.

Self-efficacy
Participants' belief in their ability to use virtual prenatal care as planned.
Individual Stage of Change Characterization of phase a participant is in, as they progress towards skilled, enthusiastic, and sustained use of telemedicine.
Other Personal Attributes* Broad construct that includes other personal traits such as motivation.

Intervention Characteristics
Relative Advantage Perception of the advantage of implementing virtual prenatal care compared to in-person visits.

Complexity
Perception about the difficulty of using telemedicine.
Design Quality and Packaging Perception of how well virtual prenatal care is bundled and presented.

Outer Setting Patient Needs and Resources
Extent to which the needs of participants are known and prioritized.
*Has been combined with Relative Advantage and presented as a Motivator and Challenge theme care (as described in Tables 2 and 3).We refer to participants who reported liking virtual prenatal care visits and would recommend it to a friend as 'liked telemedicine', participants who preferred virtual prenatal care visits only during the COVID-19 pandemic as 'liked telemedicine for now', and participants who reported not liking virtual prenatal care visits and wanting only in-person visits as 'did not like telemedicine'.

Study Sample and Setting
A total of 59 participants completed the survey, primarily aged 26-30 years (42%), White (63%), and utilizing employer insurance (71%) during their pregnancy.Participants were receiving care at a major urban academic clinic in the southeastern United States that sees approximately 2,000 racially and ethnically diverse patients a year.Participant demographic characteristics by obstetric virtual prenatal care preference are shown in Table 2.

Knowledge and Beliefs about the Intervention (Patient Preference)
Among participants, 53% (n=31) preferred virtual prenatal care via telemedicine during the COVID-19 pandemic but intended to return to in-person visits when possible.In contrast, 31% (n=18) liked virtual prenatal care and would recommend them to a friend.Nine participants (15%) did not like virtual obstetric prenatal care and wanted only in-person visits and one (2%) did not report a preference of the two modalities (Table 2).

Self-Efficacy
Participants were asked about their confidence in continuing virtual prenatal care visits after experiencing their first prenatal telemedicine visit if they wanted to, further defined by their ability to interact with the provider and receive care via the virtual visit.Generally, participants who liked virtual prenatal care (100%, n=16), and who liked virtual prenatal care for now (84%, n=26) were confident in using virtual prenatal care visits again if they wanted to.Only 22% (n=2) of those who disliked virtual prenatal care had self-efficacy in doing a virtual prenatal care visit again.

Individual Stage of Change
Because the virtual obstetric service delivery format was new, participants were asked if they felt prepared for their first virtual prenatal care visit.While most participants who liked virtual prenatal care (94%, n=17), or liked it for now (94%, n=30), reported that they felt prepared for their first virtual prenatal care visit, two-thirds (66%, n=6) of those who did not like virtual prenatal care felt unprepared for their first visit.Overall, participants found their first virtual prenatal care visit during the COVID-19 pandemic to be easy, they felt prepared for the visit and were confident in continuing virtual prenatal care visits.Additionally, instructional materials were considered helpful, with minimal additional guidance required, and most participants who favored virtual prenatal care visits reported that their needs were met.

Motivation and Challenges
This theme encompasses Motivators (Other Personal Attributes) and Relative Advantage.In general, participants discussed similar factors that either motivated them to continue using virtual prenatal care or posed challenges to their experience of using virtual prenatal care visits during the COVID-19 pandemic.The convenience of visits, performing procedures, safety from COVID-19, provider communication, and technology and privacy issues were the most prominent themes that either motivated or presented a challenge to using virtual prenatal care visits (Figure 1).Supporting quotes to illustrate patients' perspectives on their preference for using virtual prenatal care are shown in Table 3.

Convenience of Visits
The theme of convenience of visits encompassed participants' transportation, travel distance, childcare, scheduling, participant's availability for appointments, and financial needs, due to the associated decreased costs with travel and less loss of work time.Several motivators to prefer virtual prenatal care visits over in-person visits were reported: not needing to drive or have transportation to attend prenatal visits; a decreased need for childcare; ease and flexibility of scheduling appointments; ability to have prenatal visits in the comfort of their home; the benefit of having a partner attend visits; shorter appointment times; and decreased out-of-pocket costs.For example, one participant espoused the convenience of virtual prenatal care for someone with a busy schedule: "If someone has a busy schedule and cannot go to an in-office visit, this is a great alternative."(A2 -Hispanic, Age 18-25, Individual insurance plan) However, another participant noted that virtual prenatal care usurped the convenience of bundling other procedures or care into appointment times:

Performing Procedures
Regardless of participants' virtual prenatal care preferences, many described the inability of the provider to perform procedures such as fetal heartbeat checks, physical examinations, bloodwork, ultrasounds, and confirmation of pregnancy as challenges of telemedicine.This aspect of in-person prenatal care was noted as needing adaptations when providing virtual prenatal care.For example, some participants noted that not being able to hear their baby's heartbeat influenced their first virtual prenatal care experience: "I enjoy hearing the baby's heartbeat and I couldn't do that with telehealth."(A5 Black/African American (AA), Age 26-30, Employer insurance) One participant who disliked virtual prenatal care visits preferred having the provider perform examinations instead of requiring her to check her own vitals at home, which complicated the virtual prenatal care visit for her: "...ability for doctors to check vitals, measure, etc., versus me being the doctor at home trying to figure these things out."(A61 -White, Age 26-30, Employer insurance)

Safety from COVID-19
Participants also described a major motivator was the decreased exposure to COVID-19 infection that virtual prenatal care visits offered as a safer alternative to in-person prenatal care: "SAFETY!!Not wanting to risk unnecessary exposure for myself, my doctor, or the staff."(A63 -White, Age 31-35, Individual plan)

Provider Communication
Additionally, provider communication emerged as a motivator or challenge for using virtual prenatal care visits and was also described as an important aspect of prenatal care for participants when considering using virtual prenatal care visits.This theme encompassed open provider communication, having a personal connection with providers, providers being empathetic, caring, and providing reassurance when needed.It also encompassed having opportunities to ask questions and have questions answered, having additional information from providers, and not feeling rushed during a visit.When these factors were not met, this was viewed as a challenge for participants.
For example, one participant who liked prenatal virtual prenatal care visits for now, yearned for more information from her provider:

Technology/Privacy Issues
Technology issues were considered challenges to using virtual prenatal care for some, including difficulty connecting to the appointment because of faulty email links or internet connection.Other issues included unavailability or difficulty using blood pressure machines and Doppler machines to gather vitals such as participant's blood pressure and heart rate, or fetal heart rate.For example, one participant had difficulty operating her Doppler machine and had difficulty locating a fetal heartbeat.This resulted in undue momentary distress for the participant.Another participant who preferred "virtual prenatal care for now" noted the potential for these technology challenges: "If unable to get a blood pressure machine or internet working, this could pose a problem."(A49 -Black/AA, Age 31-35, Employer insurance) Finally, privacy issues were mentioned among a few participants who reported that during their virtual prenatal care visit, another patient inadvertently logged into their session.

Complexity
Participants were asked how difficult it was for them to do their first virtual prenatal care visit.Overall, participants did not find their first virtual prenatal visit to be difficult.However, two (6%) participants who preferred obstetric virtual prenatal care for now found it somewhat difficult to do.

Quality
Most participants who liked virtual prenatal care (94%, n=15) and who liked virtual prenatal care for now (80%, n=20) found handouts and instruction materials provided helpful.A little more than (56%, n= 5) of participants who disliked virtual prenatal care found the materials helpful.

Patient Needs and Resources
Only one participant (who liked telemedicine) required further assistance with the video connection for their virtual prenatal care visit when asked if more guidance or training with the video connection was needed.The majority of participants who liked virtual prenatal care (89%, n=16) and liked virtual prenatal care for now (84%, n = 26) reported that their needs were met.The unmet needs included not being able to hear the fetal heartbeat, check blood pressure, and lack of personal connection with the provider.

Discussion
In our mixed-methods survey, we were able to explore the patient perspective of virtual prenatal care using CFIR.The CFIR provided a valuable framework for determining the acceptability of virtual prenatal care implementation to patients and identifying ways to improve virtual prenatal care.In our patient survey, we found that a subset of patients found virtual prenatal care as an acceptable alternative healthcare delivery model (31%, n=18) indicating that post-pandemic virtual prenatal care may be a viable form of prenatal care.However, most patients (53%, n=31) considered virtual obstetric prenatal care acceptable only during the pandemic and preferred to return to in-person care afterward.Virtual prenatal care was seen as convenient and easy to access, with the advantage of no COVID-19 exposure risk.However, patients felt rushed, missed the elements of in-person care such as listening to fetal heart rate, and physical exam components and some had connectivity issues with the technology.
Our results correlate with other studies of virtual prenatal care.The percentage of patients who desired virtual prenatal care aligns with a recent study on patient desires for prenatal care delivery. 17High patient satisfaction was also seen in other studies of virtual prenatal care for those who desired it. 18,19

Strengths and Limitations of the Study
Our study has some strengths and limitations.Our study was conducted during the time of the pandemic to collect real-time feedback from participants to decrease recall bias.It was limited by a single-site design, few low-income or Medicaidinsured participants, and a low number of responses.The homogeneity of the socioeconomic status was due to the nature of the clinic studied which mainly accepts private payors.We were not able to assess patient outcomes and any significant impact on diagnosis, like gestational hypertension, which requires an increasing number of physical exams and in-person visits.Future studies have been planned to explore these limitations.

Conclusion and Global Health Implications
Patient feedback included several points with direct clinical implications.A large portion of respondents noted feeling unprepared for the virtual prenatal care visit, and therefore increasing availability to tutorials prior to the visit regarding how to work with the online platform, as well as what to expect would hopefully decrease anxiety going into the visit.To increase clinical safety and uptake of virtual prenatal care, the use of home blood pressure cuffs and fetal Dopplers may be beneficial.Many patients desired to hear fetal heart rate and missed not being able to be assured of fetal well-being during virtual appointments.Blood pressure cuffs can also be used to virtually assess maternal well-being and enhance safety if provided to patients and training patients on their use.Our findings indicate that most patients consider virtual care an acceptable alternative during the pandemic or other resource-limiting circumstances.Many emphasized how virtual care should be a choice for patients.
In conclusion, although most patients found virtual prenatal care to be easy to access and an acceptable alternative to in-person care during the COVID-19 pandemic, most patients would prefer to return to in-person prenatal visits for future care.With further improvements in obstetric virtual prenatal care, it could become a more acceptable mode of service delivery for patients that providers would be able to offer.Future research should include an examination of the long-term effects of obstetric virtual prenatal care on patient outcomes, or ways to improve the current virtual prenatal care model after a close examination of the patient-perceived challenges as found in this study.