Unpacking the nuances: an exploratory multilevel analysis on the operationalization of integrated STEM education and student attitudinal change

Abstract

Integrated STEM education (iSTEM) is recognized for its potential to improve students’ scientific and mathematical knowledge, as well as to nurture positive attitudes toward STEM, which are essential for motivating students to consider STEM-related careers. While prior studies have examined the relationship between specific iSTEM activities or curricula and changes in student attitudes, research is lacking on how the aspects of iSTEM are operationalized and their influence on shifts in student attitudes towards STEM, especially when considering the role of demographic factors. Addressing this gap, our study applied multilevel modeling to analyze how different iSTEM aspects and demographic variables predict changes in student attitudes. Drawing on data from two five-year NSF-funded projects, we evaluated pre- and post-attitude survey responses from 948 students. Our analysis identified two key iSTEM aspects—relating content to students’ lives and engagement in engineering design—that significantly influence positive attitude change. The results highlight the importance of curriculum relevance and hands-on, problem-solving activities in shaping student attitudes. However, the impact of these instructional strategies varies across demographic groups. The study’s insights into the differential impact of iSTEM aspects on diverse student groups provide actionable guidance for educators, curriculum developers, and policymakers aiming to enhance STEM learning experiences and outcomes.

Introduction

The primary policy argument for STEM education is that national prosperity depends on addressing the ever-increasing demands to expand the STEM workforce (e.g., National Academy of Engineering and National Research Council, 2014; National Academy of Science, 2010; President’s Council of Advisors on Science and Technology [PCAST], 2011). Thus, much STEM education research addresses the reluctance of students to pursue STEM-related careers (Graham et al., 2013; Pinxten et al., 2017). Various factors have been found to influence this decision, including self-efficacy, social and familial influences, and perceptions of STEM careers (Abe & Chikoko, 2020; Amparo et al., 2022; Archer et al., 2010; Ozis et al., 2018; Tate et al., 2015). However, research shows the attitudes students develop toward STEM during their K-12 years are particularly influential (Abe & Chikoko, 2020; Bakar & Mahmud, 2020; Nugent et al., 2015; A. V. Maltese & Cooper, 2017), particularly for female students and students of color (Bieri Buschor et al., 2014; Rainey et al., 2019).

In the United States, reform documents for K-12 science education, such as the Next Generation Science Standards (NGSS), promote integrated STEM education (iSTEM) to improve student learning and promote STEM careers (National Research Council, 2012; NGSS Lead States, 2013). iSTEM emphasizes problem-solving and engagement of students in science and engineering practices to address real-world phenomena and issues. It is believed to enhance scientific and mathematical literacy and motivate students to pursue STEM careers (Moore, Stohlmann et al., 2014). Despite growing enthusiasm for iSTEM, there is a need to understand how it promotes positive student outcomes and addresses gender and racial disparities in the STEM fields. Additionally, empirical evidence on how iSTEM influences student attitudes toward STEM is lacking. While some studies have examined the impact of specific STEM activities and curricula on attitudes (Açıkay et al., 2023; McLure et al., 2022; Meng & Chen, 2023; Sari et al., 2018; Uğraş, 2018), they did not investigate the specific features of iSTEM driving these outcomes. Bridging this gap can inform more effective iSTEM implementation and ultimately encourage students’ interest in STEM careers. Thus, this study aimed to examine which critical aspects of iSTEM contribute to change in students’ attitudes towards STEM. Specifically, the study was guided by the following research questions:

(1) Which of the aspects of iSTEM are significantly associated with change in student attitudes toward STEM, when controlling for demographic variables, such as gender and race?

(2) To what extent do each of the aspects of iSTEM relate to change in student attitudes toward STEM while accounting for gender and race (interaction effects)?

Theoretical framework

Integrated STEM

Modernization in our increasingly technological world has surged the demand for STEM careers, yet educational systems lag in producing the necessary number and quality of STEM professionals (Anft, 2013; Grocholski, 2018). To address this, global efforts are promoting integrated STEM education (iSTEM) in K-12 schools (e.g. Executive Office of the President United States, 2012). Over the past twenty years, numerous scholars have attempted to define iSTEM. iSTEM requires explicit integration of STEM subjects (National Academy of Engineering and National Research Council, 2014). It demands proficiency implementing curricula that align with STEM’s broader goals (Lynch et al., 2014; Roehrig et al., 2021). Effective iSTEM pedagogical strategies involve contextualizing STEM learning through real-world problems (Roehrig et al., 2020; Kelley & Knowles, 2016; Kloser et al., 2018; Moore et al., 2020), incorporating students’ personal experiences (Roehrig et al., 2021; Ryu et al., 2018; Sias et al., 2017), designing multidisciplinary tasks (Fan & Yu, 2017; Guzey et al., 2017; Roehrig et al., 2021), engaging students in 21^st century skills and STEM practices (Roehrig et al., 2021; Siverling et al., 2019; Stehle & Peters-Burton, 2019; Trevallion & Trevallion, 2020), and introducing them to STEM careers (Kitchen et al., 2018; Roehrig et al., 2021; Ryu et al., 2018; Willis et al., 2010). While there remains disagreement about definitions of iSTEM (T.J. Moore et al., 2020), the field has majority consensus on defining characteristics of iSTEM (Roehrig et al., 2021). This recent framework presents seven defining iSTEM characteristics: focus on real-world problem, engineering design, context integration, content integration, authentic STEM practices, 21st-century skills, and promotion of STEM careers. Integrated STEM education utilizes motivating and relevant real-world problems to contextualize learning and engage learners in applying and increasing their STEM knowledge (Kelley & Knowles, 2016; Leammukda & Roehrig, 2020; Monson & Besser, 2015). In that regard, engineering design provides an opportunity for students to develop multiple solutions to these problems and help them learn from failure (Moore, Glancy, et al., 2014; Stretch & Roehrig, 2021). Contexts such as real-world problems and engineering design challenges facilitate interdisciplinary learning of STEM content (Arık & Topçu, 2020; Hiwatig et al., 2022). Subsequently, iSTEM features content integration as both means and end to such contextualized learning and, it emphasizes that the connections among the disciplines are made explicit to students (Dare, Keratithamkul, et al., 2021; English, 2016; Moore, Stohlmann, et al., 2014). Furthermore, iSTEM instruction highlights opportunities for students to engage in STEM practices, such as evidence-based reasoning and data practices, which allow learners to exercise agency in their learning activities (Guzey, Moore, & Morse, 2016; Hiwatig et al., 2022; Kelley & Knowles, 2016; E. Miller et al., 2018). It also supports the development of 21st century skills, such as collaboration, higher-order cognitive skills, and creativity (Asunda, 2014; Sias et al., 2017). Finally, iSTEM instruction provides students with opportunities to learn details about STEM careers and engage in authentic practices that STEM professionals engage in (Kitchen et al., 2018; Rodriguez et al., 2017; Ryu et al., 2018).

Subsequently Roehrig et al.’s (2021) theoretical framework has guided the design and development of an observation instrument (Dare, Hiwatig, et al., 2021) that aims to measure the extent of the occurrence of iSTEM in a given lesson. It effectively describes ten aspects of iSTEM which are anchored on the defining characteristics of iSTEM in the aforementioned framework. These iSTEM aspects include relating content to students’ lives, contextualizing student learning, developing multiple solutions, cognitive engagement in STEM, integrating STEM content, student agency, student collaboration, evidence-based reasoning, technology practices in STEM, and STEM careers. In light of the current study, these aspects of iSTEM represent instructional moves that can be optimized to present STEM learning in a positive way to students. For example, it can be argued that by making clear connections between the STEM lesson content and their personal experiences, students appreciate more the relevance of STEM to their lives.

Attitude and attitudinal change

In this paper, attitudes towards STEM encompass student attitudes about STEM learning, STEM careers, and social implications of STEM as characterized by Moore and colleagues (Moore et al., 2014). Additionally, it refers to underlying dimensions such as personal and social implications of STEM as well as the learning of science, technology, engineering, and mathematics and their respective relationship to STEM. Historically rooted in social psychology, attitudes reflect one’s summative evaluation of a psychological “object,” spanning positive and negative dimensions, such as good-bad or pleasant-unpleasant (Ajzen, 2001). These attitudes emanate from an individual’s beliefs about an object, with the strength of their overall sentiment determined by their subjective value of the object’s traits and its connection to their belief system. Attitudes encompass both affective and cognitive components (Haddock & Zanna, 2000; van der Pligt et al., 1998). The weight of each varies based on the nature of the attitude object. For example, attitudes towards knowledge domains like STEM are likely influenced more by cognitive elements, such as the learning process. Historically, scholars like Doob (1947) suggested that learning experiences shape most of our attitudes, while Hovland’s (1953) model highlighted the pivotal roles of educators and curricula in modifying attitudes towards subjects like science. Attitudes, though intricate, change based on experiences (Perloff, 2016; Petersen & Carlson, 1979), and pedagogical strategies can heavily sway this (Garcia-Carrion et al., 2020). Consequently, the manner in which STEM is presented in classrooms can either foster or alter student attitudes towards it, influenced by their classroom experiences and the teacher’s pedagogical approach.

Literature review

Aspects of iSTEM and student attitudes

Several iSTEM aspects have been linked to positive student outcomes: lessons tied to prior experiences spark interest (Djonko-Moore et al., 2018; Moll et al., 1992; T.J. Moore et al., 2020); real-world problem-centered learning boosts motivation (Kelley & Knowles, 2016; Monson & Besser, 2015); creativity in problem-solving fosters engagement (Berland & Steingut, 2016); seeing interdisciplinary connections propels interest (E.A. Dare et al., 2018; Moore, Stohlmann, et al., 2014); hands-on STEM practices inspire positive attitudes (E. Miller et al., 2018); collaboration elevates classroom experiences (Chen, 2018); and student engagement with technology heightens excitement (Bell & Bull, 2008; Ellis et al., 2020). Additionally, presenting equity-driven information about STEM careers can help dispel negative stereotypes (Avraamidou, 2020; Blotnicky et al., 2018). However, more unified research is needed to discern which particular iSTEM aspects most impact student attitudes.

Influences on iSTEM attitudes: demographic factors

While classroom experiences can reshape attitudes, intrinsic attributes like gender and race often establish initial standpoints (Jensen, 2017). For example, there exists a gender disparity in STEM with women significantly underrepresented compared to men (National Girls Collaborative Project, 2022; U.S. Department of Labor, 2024). Studies have shown that girls tend to develop negative attitudes towards STEM as they progress through their schooling (Sadler et al., 2012; Trott & Weinberg, 2020) because of issues such as gender stereotypes, male-dominated cultures, fewer female role models, and math anxiety (American Association of University Women [AAUW], n.d.). STEM fields are often perceived as masculine, especially those that are considered to be mathematics-heavy like physics, engineering fields, and applied mathematics. This perception prevents young female learners from developing a strong, positive STEM attitude that is supposed to propel them into pursuing future STEM careers. The lack of women studying and working in STEM allows the perpetuation of exclusionary, rigid, male-dominated cultures in the STEM fields that are not supportive of or appealing to female students. The gender disparity is aggravated by the lack of female role models in books, media, and pop culture. Finally, girls acquire math anxiety later on in their education because society impresses upon them that girls are not naturally good at Math (Gillibrand et al., 1999; Tai et al., 2006; Tolley, 2003). In light of these issues, some aspects of iSTEM are posited to address the gender disparity in terms of student attitudes toward STEM (Roehrig et al., 2021). For example, using real-world problems that are relevant to all students encourages participation and interest in STEM learning among all students of different backgrounds (Adams et al., 2014; Jethwani et al., 2016; Kessels, 2014; O’Brien et al., 2016). Providing examples of STEM careers featuring female STEM professionals can also temper the perception of STEM being only for men, leading to female students developing more positive STEM identities and interest in pursuing STEM careers (Cheryan et al., 2015; Koch et al., 2019; Weisgram & Diekman, 2017).

Racial disparities in STEM attitudes also persist. People of color, especially women of color, have been largely underrepresented in, and historically ostracized in, most STEM fields (Kessel & Nelson, 2011; National Science Foundation & National Center for Science and Engineering Statistics, 2017; Rodriguez et al., 2017). Kayumova and colleagues (2015, 2018) reported that Black and Latino/a students are less likely to pursue their interest in STEM due to the fact that they are less likely to receive the types of support required to foster their future aspirations in STEM. Students’ apprehension of confirming negative stereotypes of a group they belong to (gender, race, etc.) can also undermine their performance and affect their perceived STEM identity (Pew Research Center, 2022; Thoman et al., 2014). Meanwhile, existing studies such as Guzey, Harwell, et al. (2016) examined the effect of iSTEM on attitudes and how they vary across racial groups. They did not find significant difference between white and non-white students. This illustrates that iSTEM has the potential to eradicate existing racial disparity in student attitudes toward STEM (Rainey et al., 2018; Wiebe et al., 2018). Consequently, specific aspects of iSTEM that result to such have to be explored further in order to cater the lesson implementation better according to students’ backgrounds and racial identities.

Other possible factors influencing STEM attitudes

Studies show that younger students often exhibit more positive attitudes towards STEM compared to older students. For instance, research by Karakaya and Avgın (2016) and Unfried, Faber, and Wiebe (2014) indicates that middle school students’ attitudes towards STEM tend to decline as they progress through grades. In primary education, programs that integrate STEM education, such as the Project-based Integrated STEM Program, have been shown to positively influence students’ attitudes towards these subjects. Zhou et al. (2019) found that primary students who participated in such programs demonstrated improved attitudes towards STEM, regardless of their initial attitudes. This suggests that early exposure to engaging and hands-on STEM activities can foster a lasting interest in these fields. The decline in positive attitudes towards STEM as students progress to higher grades suggests the need for continuous and engaging STEM education throughout their schooling (Kurt & Benzer, 2020). Early and sustained interventions that provide hands-on, relevant, and enjoyable STEM experiences are crucial. Additionally, addressing the increased academic pressure and providing support as STEM subjects become more challenging can help maintain positive attitudes.

Different areas of science—such as physical science, life science, and earth science—can also elicit varying levels of affective outcome from students (Hiwatig, 2022). Understanding these differences is essential for developing targeted educational strategies that can enhance student engagement and achievement in STEM. For example, Tatar and Ergin (2019) showed that gender differences in attitudes towards physical and life sciences influence students’ overall perception of STEM. They found that female students exhibited more favorable attitudes towards biological sciences, which are often perceived as more relevant to personal and societal issues compared to physical sciences, which are seen as more abstract and technical. Additionally, a study by Faber and colleagues (Faber et al., 2013) highlighted that earth sciences, while less frequently emphasized in the curriculum, can significantly impact students’ environmental awareness and attitudes towards science. Their research indicated that integrating earth science topics into the STEM curriculum could foster a more comprehensive understanding of environmental issues, thereby increasing students’ overall interest in STEM. The varying attitudes towards different science areas suggest that educators need to adopt diverse strategies to engage students effectively. Tailoring STEM education to highlight the relevance and applications of each science area can help maintain students’ interest and foster positive attitudes.

Research framework

Informed by the theoretical frameworks and supporting literature presented above, a research framework (Fig. 1) was developed that illustrates unidirectional relationships between the outcome variable, change in student attitudes towards STEM, and predictor variables, aspects of integrated STEM and demographic variables such as gender and race/ethnicity. The conceptual framework also includes other contextual variables, science area and the grade level, of the curriculum unit implemented.

Fig. 1

Research framework

The research framework depicts unidirectional arrows for all iSTEM aspects even though supporting literature on each of their relationship with attitude toward STEM is still lacking. This study, recognizing the emerging nature of the field of integrated STEM (iSTEM) education, adopts a hypothesis-driven approach. While the literature on iSTEM’s impact on student attitudes toward STEM is still developing, this research framework is designed based on existing theoretical and empirical insights. It illustrates this study’s hypotheses about specific influences of key iSTEM aspects on attitudinal changes in students. This approach aligns with a strategic investigation of how distinct aspects of iSTEM education may contribute to shifts in student attitudes towards STEM.

The contextual variables (i.e. aspects of iSTEM, science area, and grade level) are separated from the demographic variables (i.e. gender and race/ethnicity) by a dash-dot line. The literature review presented claims from previous works about the differentiating effects of student characteristics on STEM attitudes. In that regard, the research framework also depicts the hypothesized moderating effects (dash lines) between the aspects of iSTEM and demographic variables, gender and race, in predicting change in student attitudes toward STEM.

Methodology

This study employed a correlational research design with regression analysis to examine the relationship between various facets of integrated STEM (iSTEM) instruction and student attitudes toward STEM. Hierarchical analysis and analysis of moderation effects were conducted to further explore these relationships. Using data from an attitude survey administered to K-12 students before and after the implementation of an iSTEM unit, the research investigated how classroom-level factors (i.e., aspects of iSTEM) influence student attitudes, accounting for student-specific attributes such as gender and race and considering the hierarchical nature of the data—students nested within classrooms. Hierarchical analysis allowed for the assessment of variability at both the student and classroom levels, while moderation analysis examined how the relationship between iSTEM instruction and student attitudes varied across different student subgroups.

Study design

The study was an ex-post facto as the iSTEM unit implementation data used in this study was collected as part of a prior project. Specifically, the dataset used came from a five-year NSF-funded project, Project STEM. Utilizing the Framework for Quality K-12 Engineering Education (Moore, Stohlmann, et al., 2014), Project STEM was designed to aid the learning of science and mathematical concepts by using an engineering design-based approach to curriculum development. Approximately 200 science teachers from the Midwestern United States participated. Teachers’ backgrounds varied by grade level (the majority of teachers taught grades 3–8) and science area taught, and years of teaching experience. Participating teachers learned about engineering design and engineering practices, teaching and learning science through engineering activities, and designing engineering design-based science units (see Guzey, Harwell, et al., 2016). Through collaboration with researchers and graduate students, the participant teachers produced over 50 curriculum units based on specific science topics (Table 1) throughout the duration of the ProjectSTEM. Each curriculum unit included an engineering design challenge (EDC) to situate the learning of target science and mathematics content. The curriculum writing process was guided by the state’s science standards, which included engineering practices, and was supported by frameworks for STEM integration that centralized the role of the engineering design process (EDP) to solve a real-world problem (Moore, Glancy, et al., 2014; Moore, Stohlmann, et al., 2014).

Table 1 Summary table of EngrTeams curriculum units

Day-to-day lesson implementation of the ProjectSTEM units was documented through video-recording of each lesson throughout the entire unit. The video observations, 50-min on average, represent a variety of classroom settings, including different grade levels, teachers, student demographics, science content, and engineering design challenges. The dataset included a wide range of teachers (106 separate teachers), classroom settings (434 earth science, 597 life science, and 999 physical science classrooms), curriculum units (48 in total), and grades (6 lower elementary, 879 upper elementary, 1071 middle school, and 74 high school observations).

In addition to the activities described above, a survey on attitudes toward STEM (Moore, Stohlmann, et al., 2014) was developed and administered before and after the unit implementations. The survey was administered to students over the span of three years of the project’s implementation. This endeavor was rooted in the broader goals of K-12 STEM education to increase student motivation to learn STEM subjects and develop interest in STEM careers.

Data and instruments

Attitudes survey

The dependent variable in this study, change in student attitude toward STEM, is derived from a vast dataset of pre and post surveys on student attitude toward STEM. Guided by frameworks on integrated STEM education and theories on attitudes, the attitude survey was designed to determine students’ attitudes toward STEM, STEM integration, and STEM careers and consisted of 28 five-point items, each scored from 1 (strongly disagree) to 5 (strongly agree). The questionnaire items are included in the Supplementary Materials. In assessing the validity and reliability of the instrument, Moore and colleagues (Moore et al., 2014) employed exploratory factor analysis (EFA) to provide internal structure validity evidence and had a panel of STEM experts to establish content validity (Moore, Stohlmann, et al., 2014). The overall survey score represents the mean of student responses on the 28 survey items, with higher scores indicating more positive attitudes toward STEM. EFA results indicated that the use of an overall attitude score (taken as the average of all 28 survey items) is valid, given a very high Cronbach alpha value of 0.91 for the entire survey. Change in student attitudes toward STEM was determined by calculating the difference between the post and pre-survey attitude survey scores. Demographic information such as students’ gender and race, and grade level were also included in the survey dataset.

STEM-OP

The second instrument, the STEM Observation Protocol (STEM-OP), was used to score the video dataset. The STEM-OP consists of ten items, each with four scoring levels (Dare, Hiwatig, et al., 2021, see Table 2). It was designed to measure the extent to which iSTEM occurs in a given period of classroom instruction. Consequently, it evaluates 10 key aspects of integrated STEM education that reflect the characteristics of integrated STEM outlined in Roehrig et al.’s (2021) theoretical framework. Table 2 summarizes details of each STEM-OP item. In establishing the validity of the instrument, the authors sought external review by a panel of STEM experts and subjected the instrument draft to multiple iterations to ascertain that each item sufficiently captures the aspects of their corresponding constructs. Furthermore, they provided additional validity evidence through examining the internal structure of the instrument, described in Roehrig et al. (2022). Principal Component Analysis (PCA) was conducted on data from 2030 classroom video observations. This sample size exceeded the recommended minimum ratio, ensuring robust analysis. PCA revealed three principal components explaining 60.7% of the total variance, with the first component accounting for 34.7%, the second for 15.1%, and the third for 11.0%. These components represented key dimensions of integrated STEM instruction, labeled as “Real-World Problem Solving” and “Nature of STEM Integration,” with a potential third dimension related to “Technology Practices in STEM” (Roehrig et al., 2022). Item reliability was demonstrated through inter-reliability with all items achieving an inter-rater reliability above the acceptability threshold of Krippendorff’s α ≥ 0.6 with the slight exception of an item referring to the integration of STEM content that achieved α ≥ 0.58.

Table 2 Description of the 10 STEM-OP items

Variables and data analysis

Correlational research design with regression analysis was used to investigate the different aspects of iSTEM and student attitudes toward STEM. Specifically, multilevel modeling was used to explore how contextual, classroom-level factors, such as the various aspects of iSTEM, contribute to change in student attitudes toward STEM when student-level factors such as gender and race are considered. This research design and approach were used in order to determine predictive relationships between the dependent and independent variables without making strong causal inferences and to account for the nested structure of the data (students within classrooms) (Ferron et al., 2008; Garson, 2013; Snijders & Bosker, 2012). SPSS v. 29 was the primary statistical software used in the analyses.

After data cleaning procedures, attitude survey data from 948 students were chosen for the analysis and were paired with the STEM-OP data of 49 classrooms students participated in. Consistent with Moore et al. (2014) and Guzey, Harwell, et al. (2016), overall student attitude towards STEM was calculated by getting the average of student responses to all the 28 items in the survey. Subsequently, change in overall student attitude towards STEM is computed as the difference between the student overall scores from the pre and post STEM Attitude survey. There were two main student demographic variables in this study, Gender and Race, and both were dummy-coded. For Gender, female is set as the reference category. For Race, there are four dummy variables corresponding to the non-White demographics, i.e. Native American, Asian/Pacific Islander, Hispanic/Latino, and African American. White is set as the reference category. The contextual variables included the ten aspects of iSTEM (as measured by STEM-OP items Dare, Hiwatig, et al., 2021), grade level, and science area. Both grade level and science area were dummy-coded and High School and Life Science were both set as reference categories, respectively (see Supplementary Materials for study variables and coding scheme).

Meanwhile, the values for each of the ten aspects of iSTEM were calculated by computing a unit score for that particular STEM-OP item, i.e. an exponential moving average (EMA) of the available scores of all days observed during a unit implementation. In the given study, the exponential moving average (EMA) was employed over a simple average due to its superior capability in handling datasets with varying time lengths and the dynamic nature of the observations (Klinker, 2011). Since classroom unit implementations can differ in duration, a simple average could distort the true performance trend by treating all units as if they had the same number of observations. Furthermore, EMA provides a more nuanced analysis for dynamic and evolving processes like lesson implementations by assigning more weight to the most recent observations. This methodological choice ensures that recent changes in the effectiveness or quality of the lesson are more prominently reflected, offering a clearer insight into the evolving trends of educational practices. This approach aligns with the need to accurately capture the fluctuating and time-sensitive nature of teaching environments, making EMA a more appropriate statistical tool for this analysis.

$$ {EM{A_{STEMOPx}} = P\,*\,\alpha + \left( {Previous\,EMA*\left( {1 - \alpha } \right)} \right)} $$

where P is the current STEMOP_X score,

$$\eqalign{\alpha \,{\text{is}}\,{\text{the}}\,{\text{smoothing}}\,{\text{factor}},\,{\text{given}}\,{\text{as}}\,\cr\frac{2}{{1 + number\,of\,observations\,in\,a\,unit\,}}}$$

Consistent with the conventions in educational research, a significance level of 0.05 was chosen for all the inferential statistical tests in this study. Literature on multilevel modeling provides a general rule-of-thumb for determining appropriate sample size to achieve a desirable power (Peugh, 2010; Spybrook, 2008). There were simulation studies illustrating that at least 30 groups, with at least 10 cases in each, yield unbiased estimates of fixed effects and contextual effects; and at least 50 groups are needed to estimate correct estimates of standard errors (Maas & Hox, 2004; Scherbaum & Ferreter, 2011). Given that there are 948 cases nested in 49 groups analyzed, it is reasonable to claim that the inferential tests conducted in this study are sufficiently powered.

Results

Table 3 presents the descriptive statistics for all the study variables. There was a total of 948 cases (students) subjected to the analysis. More than half of the sample were male and, 46% of the students were White. Of the 49 classrooms, the majority were middle school classrooms, and more than half were physical science classrooms. Overall, there is a positive mean change in the student attitude across classrooms (mean = 0.012; SD = 0.44). This relatively small but positive average change indicates that there might be a ceiling effect on the attitudinal change for students which may be attributed to several factors not included in this study such as prior exposure to iSTEM, quality of teachers recruited into the ProjectSTEM, and the prevalence of STEM programs in the state where the classrooms were sampled, among others.

Table 3 Descriptive statistics

RQ 1: Which of the aspects of iSTEM are significantly associated with change in student attitudes toward STEM, when controlling for demographic variables, such as gender and race?

An unconditional model for the variation of change in student attitude toward STEM was created to illustrate the variation between classrooms in terms of change in attitude scores and classroom-level characteristics. The subsequent analysis resulted in an intra-class correlation (ICC) of 0.0534, which means that 5.34% of the variance in attitude score change can be attributed to between-classroom differences. Such ICC value is considered as medium-level homogeneity (Kreft & De Leeuw, 1998). Consequently, design effect (DEFF) was calculated as follows:

$$ {DEFF = 1 + \left( {cluster\,size - 1} \right)*ICC)} $$

With a DEFF of 3.56, a value greater than 1.5, the analysis indicates that multilevel modeling (MLM) was warranted (Lai & Kwok, 2015).

Considering the research questions and research framework for this study, the effect of the student-level predictors, gender and race, are expected to vary between classrooms. To estimate this kind of variation, intermediate models were created (Ferron et al., 2008). As a note, all student-level predictors were selected and added as fixed effects given the exploratory nature of this study. The first set of models are constrained intermediate models that include in a stepwise, additive manner: student-level predictors and classroom-level predictors (STEM-OP scores, grade band, and science area) without any cross-level interactions. The second set of models are augmented intermediate models wherein slope residuals term is added. The slope residual corresponds to the differences between classroom-specific effects of the student-level predictors and the fixed slopes. Likelihood-ratio test was performed to compare the intermediate models and determine which one has a better fit. An online chi-square calculator was used to test whether one model is significantly different from another model (Social Science Statistics, 2022).

With the decision to include slope residuals, the estimation of cross-level interactions between each of the student-level predictors, race and gender, and classroom-level predictors of interest, aspects of iSTEM, was warranted. The addition of all the predictors (and random effects) in the model resulted in a conditional Pseudo R² of.109, which is higher than the marginal Pseudo R² of.094, indicating better fit. Figure 2 presents the final model for change in overall student attitude towards STEM. The full table of estimates from the analysis can be found in the Supplementary Materials. There are 70 fixed effects estimated, 50 of which are interaction effects. Table 4 shows the statistically significant estimates including the random effects.

Fig. 2

Final model for change in overall student attitude towards STEM

Table 4 Statistically significant MLM results for change in overall student attitude towards STEM

Two of the iSTEM aspects have a statistically significant effect on student attitudinal change towards STEM. They are STEMOP1 (Relating Content to Students’ Lives) and STEMOP3 (Student Engagement in Engineering Design). First, these findings imply that in classrooms where students perceive the STEM content as directly connected to their own experiences and personal lives, their attitudes towards STEM fields become more positive (${\gamma_{01}}$ = 2.916, SE = 0.92). This finding advocates for a curriculum design strategy that seeks to bridge the gap between abstract STEM concepts and tangible, real-world applications. By integrating examples, problems, and projects that reflect students’ interests and experiences, educators can make STEM subjects more accessible and engaging, thereby enhancing students’ attitudes towards these fields (Roehrig et al., 2021; Ryu et al., 2018; Sias et al., 2017).

Second, the hands-on, practical involvement in designing multiple solutions such as engineering design projects, which often requires creative problem-solving and application of scientific principles, significantly contributes to students developing a favorable attitude towards STEM (γ₀₃ = 2.878, SE = 1.15). Engagement in developing multiple solutions requires students to apply STEM content in creative problem-solving scenarios, fostering a sense of achievement and the practical relevance of STEM knowledge (Dare, Keratithamkul, et al., 2021; Grubbs & Strimel, 2015; Shahali et al., 2017). Such experiential learning opportunities allow students to experience firsthand the excitement and innovation inherent in STEM disciplines, thus contributing to a more favorable outlook towards STEM.

RQ 2: To what extent do each of the aspects of iSTEM relate to change in student attitudes toward STEM while accounting for demographic variables, such as gender and race (interaction effects)?

The study’s multilevel analysis revealed significant interaction effects that underscore the complex interplay between demographic variables and specific aspects of iSTEM on student attitudes towards STEM. These findings offer a nuanced understanding of how gender and racial/ethnic identity can influence the efficacy of iSTEM pedagogical strategies, highlighting the importance of culturally responsive teaching practices.

The relationship between change in overall student attitudes toward STEM and STEMOP4 (Cognitive Engagement in STEM) is significantly moderated by the Gender dummy variable. As classrooms score higher on Item 4, the change in attitude towards STEM becomes less pronounced for male students compared to female students (see Fig. 3). Conversely, compared to their male peers, female students are at an advantage in terms of shifting their STEM attitudes when lessons involve higher cognitive engagement.

Fig. 3

Interaction plot for gender and STEMOP4 on their effect on change in overall STEM attitude. Note Level 0: The teacher does not provide opportunities for students to learn S/T/E/M concepts. Level 1: The teacher provides opportunities for students to remember or understand S/T/E/M concepts and/or a design problem. Level 2: The teacher provides opportunities for students to use or apply S/T/E/M concepts and/or a design plan. Level 3: The teacher provides opportunities for students to analyze or evaluate S/T/E/M concepts and/or design solutions

Additionally, the relationship between change in overall student attitudes toward STEM and STEMOP1 (Relating Content to Students’ Lives) is significantly moderated by the NatAmer dummy variable (see Fig. 4). This means that compared to their white peers, Native American students experience lesser attitudinal shift as lessons score higher on STEMOP1. It also indicates that Native American students may not perceive STEM content as being as relevant to their personal lives as White students do. This underscores the importance of incorporating culturally relevant examples and contexts into STEM lessons that resonate with the experiences and backgrounds of Native American students.

Fig. 4

Interaction plot for NatAmer and STEMOP1 on their effect on change in overall STEM attitude. Note Level 0: The teacher does not acknowledge students’ everyday and/or personal experiences related to STEM. Level 1: The teacher mentions their own personal experiences or provides concrete examples to illustrate the STEM content in the lesson. Level 2: The teacher elicits students’ everyday and/or personal experiences related to STEM during the lesson. Level 3: The teacher elicits students’ everyday and/or personal experiences related to STEM and explicitly connects these to the lesson

Moreover, the relationship between change in overall student attitudes toward STEM and STEMOP3 (Developing Multiple Solutions) is significantly moderated by the NatAmer dummy variable (see Fig. 5). Similar to the finding for STEMOP1, the negative interaction effect estimate for Native American students suggests a discrepancy in how these students view the importance or applicability of divergent thinking and multiple solutions in STEM, compared to their White peers. This may reflect a need for pedagogical approaches that validate and incorporate diverse ways of knowing and solving problems.

Fig. 5

Interaction plot for NatAmer and STEMOP3 on their effect on change in overall STEM attitude. Note Level 0: The teacher does not encourage the development of multiple solutions. Level 1: The teacher encourages students to develop multiple solutions but does not provide opportunities for students to evaluate these solutions. Level 2: The teacher encourages multiple solutions and provides opportunities for students to evaluate the viability of different solutions. Level 3: The teacher encourages multiple solutions and provides opportunities for students to not only evaluate the viability of different solutions, but also use this information to redesign their solution

Meanwhile, there is a significant interaction effect between STEMOP4 and the dummy variable, AfrAmer, in terms of predicting attitudinal shift towards STEM (see Fig. 6). The positive estimate suggests that African American students, when compared to their White peers, derive more significant attitudinal benefits from lessons that engage them at various cognitive levels. This could imply that strategies which foster deep thinking and problem-solving can be particularly effective in enhancing STEM attitudes among African American students.

Fig. 6

Interaction plot for AfrAmer and STEMOP4 on their effect on change in overall STEM attitude. Note Level 0: The teacher does not provide opportunities for students to learn S/T/E/M concepts. Level 1: The teacher provides opportunities for students to remember or understand S/T/E/M concepts and/or a design problem. Level 2: The teacher provides opportunities for students to use or apply S/T/E/M concepts and/or a design plan. Level 3: The teacher provides opportunities for students to analyze or evaluate S/T/E/M concepts and/or design solutions

On the other hand, the relationship between change in overall student attitudes toward STEM and STEMOP8 (Evidence-Based Reasoning) is significantly moderated by the AfrAmer dummy variable (see Fig. 7). The negative estimate indicates that African American students may not benefit as much from practices of evidence-based reasoning in terms of attitudinal shifts towards STEM as their White counterparts. This finding could point to a need for more supportive structures or culturally responsive approaches in teaching evidence-based reasoning to African American students.

Fig. 7

Interaction plot for AfrAmer and STEMOP8 on their effect on change in overall STEM attitude. Note Level 0: The teacher does not provide students with opportunities to make claims and/or design choices. Level 1: The teacher provides opportunities for students to make claims and/or design choices, but these claims/choices are unsupported by evidence. Level 2: The teacher requires students to make claims and/or design choices based on evidence but does not require them to justify their reasoning. Level 3: The teacher requires students to make claims and/or design choices based on evidence and justify them using reasoning

Finally, the relationship between change in overall student attitudes toward STEM and STEMOP9 (Technology Practices in STEM) is positively moderated by the dummy variable, HIS (see Fig. 8). The positive estimate for Hispanic/Latino students indicates a particularly favorable impact of engaging with technology practices analogous to those used by professionals on their attitudes towards STEM, compared to White students. This suggests that emphasizing technology and its applications within the STEM curriculum can be a potent strategy to increase Hispanic/Latino students’ interest and positivity towards STEM fields.

Fig. 8

Interaction plot for HIS and STEMOP9 on their effect on change in overall STEM attitude. Note Level 0: Students do not use technology to collect, analyze or represent data, or to create or modify scientific models and/or design solutions. Level 1: Students use technology to collect data. Level 2: Students use technology to analyze and/or represent data. Level 3: Students use digital technology to create or modify a scientific model or design solution (e.g., CAD software)

Discussion of findings

Attitudes towards STEM

Our investigation into the factors influencing student attitudes towards STEM through integrated STEM instruction (iSTEM) reveals multifaceted insights that extend the current understanding of how educational environments and pedagogical approaches impact student perceptions and interest in STEM fields. The findings from this study, particularly the significant effects of specific iSTEM aspects such as relating content to students’ personal lives and engaging them in developing multiple solutions such as in engineering design, underscore the potential of targeted educational strategies to enhance student attitudes towards STEM. However, the discussion of these findings cannot be isolated from the broader context of classroom and extracurricular factors that also play crucial roles in shaping these attitudes.

The analysis of the null model indicated that over 5% of the change in overall student attitudes towards STEM could be attributed to differences between classrooms, suggesting a medium-level homogeneity. This observation points to a substantial variance in how different classrooms influence student attitudes towards STEM, highlighting the importance of the classroom environment and the teaching strategies employed. The variance suggests that while some classrooms witness a significant positive change in student attitudes towards STEM, others do not see the same level of impact. This differential effect underscores the complexity of attitudinal change and suggests that it is influenced by a confluence of factors beyond the iSTEM and curriculum content (L. Miller et al., 2002).

The unexplained portion of between-classroom differences in attitudinal change hints at the influence of other contextual factors such as teacher experience, class size, and the presence of existing STEM programs. These factors likely create unique learning environments that can either foster or hinder the development of positive attitudes towards STEM. This insight aligns with the socio-cognitive perspective on attitudinal change, which posits that classroom experiences and social influences play significant roles in shaping students’ attitudes (Hite & Milbourne, 2018; A. Maltese & Tai, 2011). The implications of this perspective are profound, suggesting that efforts to improve student attitudes towards STEM must consider the broader educational ecosystem, including teacher training, class dynamics, and school-wide initiatives that promote STEM engagement.

Furthermore, the observed small changes in students’ attitude scores raise questions about the potential ceiling effect in classrooms where students already have a high level of exposure to iSTEM. This finding suggests that students in some settings might begin with relatively positive attitudes towards STEM, thereby limiting the observable change. Such an outcome highlights the need for future research to account for varying levels of baseline exposure to iSTEM education and to differentiate between the impacts of iSTEM in different types of schools, such as those with a specialized focus on STEM versus general education schools.

Aspects of iSTEM

The findings make it evident that certain aspects of iSTEM play pivotal roles in fostering a more positive attitude among students towards STEM. This aligns with and expands upon the existing body of literature, such as the work of Guzey, Harwell, et al. (2016), which underscores the positive impact of such instructional approach on student attitudes. Our research delves deeper, dissecting the individual components of iSTEM to understand their distinct contributions to attitudinal change.

A notable finding from our study is the marked significance of certain iSTEM aspects, particularly those that connect STEM learning with students’ personal lives and those that engage students in hands-on, engineering design tasks where they develop multiple solutions. These findings underscore the importance of curriculum relevance and experiential learning in STEM education, reinforcing the need for educational strategies that are not only intellectually challenging but also personally meaningful to students (Kelley & Knowles, 2016; Moore, Glancy, et al., 2014). The positive effects of these iSTEM components suggest that when students can relate STEM content to their own experiences and engage in practical problem-solving, their interest and positivity towards STEM significantly increase.

Furthermore, the analysis reveals the intricate ways in which demographic variables such as gender and race interact with these iSTEM aspects, highlighting the differential impact these educational components may have on diverse student groups. This points to a gap in current literature and underscores the importance of considering demographic factors in iSTEM research and implementation. Our findings suggest that gender and race play a moderating role in how students respond to different iSTEM pedagogies. This reinforces the critical need for inclusive and culturally responsive teaching practices in STEM education, aimed at addressing the unique needs and backgrounds of all students (Edelen & Bush, 2021; O’Leary et al., 2020).

The significance of these findings lies not only in their contribution to the academic discourse on STEM education but also in their practical implications for educators, curriculum developers, and policymakers. By highlighting the impact of specific iSTEM aspects and the role of demographic factors in shaping student attitudes, our study provides actionable insights for the design and delivery of iSTEM curricula. These insights underscore the potential of tailored, contextually relevant, and hands-on STEM learning experiences to engage a diverse student population and foster a more inclusive and equitable STEM education landscape.

Gender equity

The exploration of gender equity in the context of integrated STEM (iSTEM) education reveals intricate dynamics that shape students’ attitudes towards STEM. While the overarching findings of our study suggest no significant difference between female and male students in terms of the magnitude and direction of change in their attitudes towards STEM following iSTEM exposure, a deeper dive into specific classroom contexts unveils nuanced disparities. Notably, as the degree of cognitive engagement in classroom activities increases, a negative interaction effect on male students’ attitudes towards STEM emerges, with females maintaining or improving their STEM attitudes under the same conditions. This challenges the assumption that higher cognitive engagement universally corresponds to more positive student attitudes and points to a nuanced reality where female students might actually derive more attitudinal benefit from such approaches than their male counterparts.

This finding reverses the commonly held notion that male students generally receive greater benefits from cognitive challenges within STEM learning environments. Instead, our study highlights that male students may not respond as favorably as females to higher levels of cognitive engagement. The evidence points to the possibility that iSTEM approaches might need to be refined to engage male students differently or that the nature of cognitive tasks must be revisited to ensure they cater equally to both genders (J. M. McCabe et al., 2020; M. T. Wang & Degol, 2017).

On the other hand, this intriguing reversal in gender dynamics within iSTEM classrooms reveals a promising shift: female students, traditionally viewed as disadvantaged in STEM due to various cultural and educational biases, may in fact be deriving greater benefit from iSTEM approaches that prioritize cognitive engagement. The significant and positive response of female students to these pedagogical strategies suggests that iSTEM may be serving as an equalizing force, effectively supporting and enhancing female students’ attitudes towards STEM. This aligns with research advocating for active learning environments that have been shown to reduce gender gaps in science and engineering education (Eddy & Hogan, 2014). Such findings are encouraging, as they highlight the potential of iSTEM to mitigate longstanding disparities by engaging female students in meaningful and cognitively demanding STEM learning experiences. This not only challenges the traditional narrative of gendered disadvantage in STEM but also demonstrates the capacity of iSTEM to foster an educational climate where female students can thrive and view themselves as capable and confident contributors to the STEM community.

In light of these insights, it becomes imperative for educators and curriculum developers to craft gender-responsive strategies within iSTEM education. This could involve diversifying cognitive tasks to appeal to different learning preferences, ensuring classroom environments are conducive to engagement for all genders, and re-evaluating the representation of gender within STEM materials. A concerted effort to promote gender inclusivity in teaching methods and to dismantle gender biases could lead to more balanced educational outcomes. By doing so, we work toward a more equitable STEM educational field where gender does not predict a student’s engagement level or attitudinal shift towards STEM subjects.

Racial disparity in iSTEM

Our study reveals critical insights into the racial disparities that exist in how iSTEM influences students’ attitudes towards STEM. The findings indicate significant interaction effects between specific racial/ethnic groups and certain aspects of iSTEM, underscoring the nuanced ways in which race and ethnicity intersect with educational experiences to shape students’ perceptions and attitudes towards STEM fields.

First, the negative interaction effects observed for Native American students in relation to Relating Content to Students’ Lives (Item 1) and Developing Multiple Solutions (Item 3) suggest that these aspects of iSTEM education do not resonate as effectively with Native American students as they do with their White peers. This discrepancy could be indicative of a broader issue within STEM education, where the curriculum and pedagogical approaches may not adequately reflect the cultural contexts and lived experiences of Native American students (Cannon et al., 2021; Turner et al., 2022). Such findings echo the call for culturally responsive teaching practices that are inclusive of and relevant to the diverse backgrounds of students (Bang & Medin, 2010; Castagno & Brayboy, 2008).

Conversely, African American students showed a positive response to Cognitive Engagement in STEM (Item 4), yet a negative reaction to Evidence-Based Reasoning (Item 8), highlighting varied attitudinal impacts within different aspects of iSTEM. This mixed outcome points to the complexity of engaging diverse student populations in STEM education and suggests that while certain iSTEM strategies may be effective in enhancing engagement and attitudes among African American students, others may require reevaluation or adaptation to be more culturally affirming and relevant (Emdin, 2016).

Furthermore, the positive estimate associated with Hispanic/Latino students and Technology Practices in STEM (Item 9) signifies the potential of technology integration in iSTEM to engage this group more effectively. This finding supports the argument for leveraging technology in education to bridge gaps and foster equity, aligning with research advocating for the strategic use of digital tools to enhance learning outcomes among underserved populations (Vakil & Ayers, 2019).

The observed racial disparities in the effectiveness of iSTEM highlight the imperative for educational research and practice to adopt a more nuanced understanding of how race and ethnicity influence students’ learning experiences and outcomes. These findings call for the development of iSTEM curricula that are not only interdisciplinary and integrative but also culturally relevant and responsive to the needs of diverse student populations. Ensuring that iSTEM education is equitable and inclusive involves recognizing and valuing the cultural assets that students of different racial and ethnic backgrounds bring to their STEM learning experiences (Litzler et al., 2014; Rodriguez & Blaney, 2021).

Limitations of the study

The aforementioned findings must be received in light of the limitations of this study. First, the data used in the analysis is limited by the available student-level characteristics and contextual variables. As the results demonstrated, there may be unaccounted but important contextual factors that are not included in this study. Furthermore, it is worth noting that the addition of the iSTEM aspects in the model, although significant, did not increase greatly the variance explained between classrooms. Second, the intersectionality of race and gender, although established theoretically in the literature, was not considered due to the increased complexity it will introduce to the interpretation of the results (i.e. three-way interaction among gender, race, and aspects of iSTEM. Third, the findings of this study are limited to the Project STEM data and similar study contexts. Generalizing the results pertaining to the aspects of iSTEM to the population is discouraged because random effects of these variables were not estimated. Furthermore, the unit implementations under Project STEM had different time frames and varied levels of quality. It is also worth noting that some unit implementations have less days (data points) than others. Fourth, the study design is not experimental; thus, causative claims cannot be made. While the word, “effect”, has been used multiple times throughout the paper, it was only limited to the terminology used in MLM. Nonetheless, predictive relationships between the outcome and predictors were described in the paper. Fifth, the study used a maximal model building strategy which entails including all available predictor variables and interaction terms. This put a strain on the significance test and may yield some false positive results. Sixth, there is an underlying assumption in this study that attitude survey (Moore, Stohlmann, et al., 2014) validly captures the construct of attitude towards STEM. The definition of the outcome variable in this study is limited by the theoretical and operational frameworks of the previous work that developed such research instrument. Seventh, some of the statistically significant findings presented have relatively small effect sizes. Interpretation and use of these results must be taken in light of their practical significance. Finally, the study focused heavily on the aspects of iSTEM and purposefully did not explore possible interactions between student-level factors and other contextual factors.

Conclusion and future directions

This study has made significant strides in understanding the multifaceted relationship between integrated STEM (iSTEM) education and student attitudes towards STEM fields, highlighting the complex interplay of various aspects of iSTEM and demographic factors such as gender and race. The findings reveal that while certain aspects of iSTEM significantly enhance student attitudes towards STEM, the effectiveness of these pedagogical approaches varies considerably across different demographic groups and classroom contexts. This variation underscores the importance of considering the nuanced effects of race and gender in the design and implementation of iSTEM curricula to ensure that these educational interventions are equitable and inclusive.

Moreover, the observed between-classroom differences in attitudinal changes towards STEM point to the critical role of contextual factors that extend beyond the measured variables in this study. Such factors may include, but are not limited to, teacher experience, class size, and the presence of existing STEM programs, which collectively suggest that the classroom environment itself is a significant determinant of student attitudes towards STEM. The moderating effects of race call for a reevaluation of iSTEM implementation to ensure that lesson content and activities do not inadvertently disadvantage non-White students.

The study findings present various opportunities for future research, such as a further exploration of other contextual factors at play in relation to attitudinal change, a closer examination of the interaction effects identified in the current study between aspects of iSTEM and demographic factors, or a replication study to confirm or refute the findings presented. Furthermore, similar future studies can collect school-level data to introduce a third level in the MLM analysis. By doing so, the classroom-level variables such as the aspects of iSTEM would be allowed to vary. This may provide additional insight especially pertaining to the random effects of each of the aspects of iSTEM. Furthermore, it would allow for generalizable insights about each of the aspects of iSTEM. A similar or replicate study can also benefit from a larger sample size and a higher Type I error rate given the exploratory nature of the research. The current study set α = 0.05, which while conventional in educational research, can be stringent with regard to the goal of the study. Finally, future studies may also employ a longitudinal design: collecting attitude survey at multiple points throughout the implementation of an iSTEM unit. By doing so, growth curves for the attitudinal change can be estimated which can provide more information about the effects of the predictor variables.

In conclusion, this study contributes to the emerging body of research on factors influencing student attitudes towards STEM, addressing a critical gap in the literature and laying the groundwork for future investigations. As the push for STEM education continues, understanding how to effectively leverage iSTEM to improve student attitudes is paramount for achieving broader educational goals, including increased participation in STEM careers and enhanced STEM learning outcomes.