Systematic Multiple Studies Review:

Supporting an Intentional Intervention for Graduate Engineering Students

Conducted by Jill Pearson under the guidance of
Dr. Lisa Giacumo

Click here to read the peer-reviewed published paper.

Background

The Boise State University College of Engineering received a grant from the National Science Foundation to address disparities in the number of low-income, first-generation, and underrepresented students participating in graduate engineering programs. To support the work of this grant, we conducted a systematic multiple studies review (Hong et al., 2018; Petticrew & Roberts, 2006; Pluye, et al., 2009) to determine what intervention programs were currently in place to increase the persistence and retention of these students, what components were common within the program designs, and what types of outcomes the programs were producing.

The literature review resulted in a written manuscript, which is currently under consideration for publication (Pearson, et al., 2020). It also inspired a new visual model that illustrates the relationship between common program components, the viewpoint of student academic persistence created by Tinto (2012), and the specific program design elements of the successful long-time Meyerhoff Scholars Program at the University of Maryland Baltimore County (UMBC). Finally, to support all future work on this grant, we also created an eLearning module for new graduate assistants about how to use interviews as a data gathering tool.

The project as a whole was designed with the ISPI (2020) Ten Standards in mind, paying particular attention to the first four standards:

Process

Our process started with the identification of research questions whose answers would help support the intervention work being done by the Boise State University College of Engineering.

1

What intentional intervention program components are used to increase retention and persistence among low-income (LI), first-generation (FG), and underrepresented (UR), students in science, technology, engineering, and mathematics (STEM) degree programs?

2

What can we learn from the empirical outcomes of interventions intentionally designed to support LI, FG, and UR students' retention and persistence in STEM degree programs?

The method was iterative in nature in that our search terminology and the subsequent results delivered by each helped us further define the precise research questions we used. For example, while the intervention program at Boise State University is focused entirely on engineering graduate students, using only “engineering” and “graduate” as part of our search string produced virtually no results. Therefore, through a systematic trial and error process that was captured in detail so that it could be replicated, we decided upon a traditional Boolean search string shown below:

We used the database Academic Search Premiere, and restricted our findings to peer-reviewed academic journals published between 2000 and the date of the search, March 27, 2020. This search resulted in 124 articles. Using a quality appraisal method that was outlined by Hong, et al. (2018) that built off the prior work of Pluye, et al. (2009), we reviewed each article to ensure it was relevant, empirical, and high quality, which resulted in a total of 31 sources.

Essential to understanding the data was the creation of a specific coding method so that articles could be analyzed consistently. Our codebook development began with a top down approach (Schensul, et al., 2013), guided by prior work in the field conducted by Tinto (1975, 2012). Tinto outlined four conditions that should be in place for an intentional intervention program to succeed in increasing student persistence and retention (Table 1). These conditions formed the foundation of our codebook development. Once our basic structure was established, we looked to the articles themselves as a source of the bottom-up coding by extracting data, developing definitions, and aligning program elements and outcomes.

Through this process, we were able to identify the ten most common elements present in intentional intervention programs aimed at increasing the persistence and retention of low-income (LI), first-generation (FG), and underrepresented students (UR). We aligned these elements to a traditional model created by Tinto (2012), as well as a current model that’s been experiencing positive outcomes for over 30 years at the University of Maryland Baltimore County (UMBC) (Ballen & Mason, 2017; Domingo, et al., 2019; Oseguera, et al., 2019).

Findings

Our study uncovered 13 components that were used consistently across many programs. These components included:

Admissions Requirements
Financial Aid (ranging from stipends to full scholarships)
Research Opportunities
Academic Tutoring & Study Skills
Professional Development/Networking
Graduate School Preparation
Peer Mentoring
Faculty Mentoring
Social Integration Experiences
Community Service
Targeted Academic Intervention (specific coursework or assignments)
Transition and Summer Bridge Programs
Focus placed on increasing persistence-related character traits

Despite the commonalities amongst program elements, overwhelmingly our research found a lack of consistency across program design specifics, measurement methods, and success factors. The data did not support overwhelmingly clear conclusions about which programs components were the most effective in terms of their ability to increase persistence and retention amongst underrepresented students.

That said, throughout our research, one program emerged as a possible model worthy of duplication.

The Meyerhoff Scholars Program at UMBC includes over a dozen program components including financial support, mentoring, community service, and research. Participants of the Meyerhoff Scholars Program graduate with STEM degrees at twice the rate of non-participants and they are five times as likely to go on to a PhD program in STEM (Stolle-McAllister, et al., 2010). The Meyerhoff Scholars Program has been so successful over the past 30 years that it was cited as the implementation inspiration for programs in three of the articles returned in our search.

Implications

Our work culminated in the creation of a model that combined the underpinnings of Tinto’s work (2012), the successes of the Meyerhoff Scholars Program and its elements, and the findings from our 31 empirical articles. The intention of the Low-income, First-generation, Underrepresented Student Support Model (Figure 1) is to demonstrate how an institution might design a student intervention program that addresses all four of Tinto’s conditions through carefully targeted elements.

Additional Project Components

Figure 1

LI, FG, UR student support model

The expectation is that 12 students per year will participate in the Stellar Engineering Students Graduate Program Scholarship (SENS-GPS) over the next five years, each receiving scholarships and additional support mechanisms as suggested by the multiple studies review. An essential component to this grant-funded program is ongoing research and data gathering to shape program details and determine program outcomes.

(Pearson, et al., 2020, Figure 1, p. 13)

One element of the data gathering will involve future graduate assistants of Dr. Giacumo conducting semi-structured interviews of SENS-GPS program participants. To support this specific research effort, we created an online course to prepare these graduate assistants to better perform semi-structured interviews. We used Cognitive Theory of Multimedia Learning Principles and Cognitive Load Theory (Clark, 2008) to ensure the most effective learning experience with the additional expectation that learners would be able to apply their knowledge by conducting interviews shortly after taking the eLearning.

Next Steps

The literature review is under consideration for publication in the Journal of Minorities and Women in Engineering and Science. In addition, the paper and corresponding ongoing project were just accepted for presentation at the 2021 American Society for Engineering Education annual conference. My hope is that this work contributes greatly to the professional community of practice, and that it can support the design and development of future intervention programs for FG, LI, and UR students in higher-education STEM programs.

References

Ballen, C. J., & Mason, N. A. (2017). Longitudinal analysis of a diversity support program in biology: A national call for further

assessment. BioScience, 67(4), 367-373. http://doi.org/10.1093/biosci/biw187

Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction : proven guidelines for consumers and designers of multimedia

learning (4th ed.). John Wiley & Sons. http://doi.org/10.1002/9781119239086

Domingo, M. R. S., Sharp, S., Freeman, A., Freeman, T., Jr., Harmon, K., Wiggs, M., Sathy, V., Panter, A. T., Oseguera, L., Sun, S., Williams, M.

E., Templeton, J., Folt, C. L., Barron, E. J., Hrabowski, F. A. III; Maton, K. I. Crimmins, M., Fisher, C. R. & Summers, M. F. (2019). Replicating Meyerhoff for inclusive excellence in STEM. Science, 364(6438), 335-337. http://doi.org/10.1126/science.aar5540

Hong, Q.N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M-P., Griffiths, F., Nicolau, B., O’Cathain, A.,

Rousseau, M-C., & Vedel. I. (2018). Mixed methods appraisal tool. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada.

International Society for Performance Improvement. (n.d.). ISPI's ten standards. https://ispi.org/page/10Standards

Institute of Medicine (2011). Expanding underrepresented minority participation: America's science and technology talent at the

crossroads. The National Academies Press. https://doi.org/10.1002/sce.21032

National Academies of Sciences, Engineering, and Medicine. (2019). Minority serving institutions: America's underutilized resource for

strengthening the STEM workforce. The National Academies Press. https://doi.org/10.17226/25257

Oseguera, L., Hyun Ju, P., De Los Rios, M. J., Aparicio, E. M., & Johnson, R. (2019). Examining the role of scientific identity in black student

retention in a STEM scholar program. Journal of Negro Education, 88(3), 229-248.

Pearson, J., Giacumo, L. A., Farid, A., Vacha-Haase, T., McLaughlin, B., Plumlee, D. & Sadegh, M. (2020). A Systematic multiple studies

review of low-income, first-generation, and underrepresented, STEM-degree support programs: Emerging evidence-based models and recommendations. [Manuscript submitted for publication]. College of Engineering, Boise State University.

Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A Practical guide. Blackwell Publishing.

Pluye, P., Gagnon, M. P., Griffiths, F., & Johnson-Lafleur, J. (2009). A Scoring system for appraising mixed methods research, and

concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4), 529-546. http://doi.org/10.1016/j.ijnurstu.2009.01.009

Rourke, L. & Anderson, T. (2004). Validity in quantitative content analysis. Educational Research Technology & Development, 52(1), 5-18.

http://dx.doi.org/10.1007/BF02504769

Schensul, S. L., Schensul, J. J., & LeCompte, M. D. (2013). Ethnographer's toolkit book 5. Initiating ethnographic research: A mixed methods

approach. AltaMira Press.

Stolle-McAllister, K., Domingo, M. R. S., & Carrillo, A. (2011). The Meyerhoff way: How the Meyerhoff scholarship program helps black

students succeed in the sciences. Journal of Science Education and Technology, 20(1), 5–16.
http://doi.org/10.1007/s10956-010-9228-5

Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89-125.

Tinto, V. (2012). Completing college: Rethinking institutional action. The University of Chicago Press.

Appendices

Appendix A: Program Goals and Project Tools