An Empirical Analysis of Student Learning in Digital Versus Physical STEM Labs

Summary of Methods and Findings. The current study compared the effects of virtual versus physical laboratory manipulatives on 84 undergraduate non-science majors’ (a) conceptual understanding of density and (b) density-related inquiry skill acquisition.  A pre-post comparison study design was used, which incorporated all components of an inquiry-guided classroom, except experimental mode, and which controlled for curriculum, instructor, instructional method, time spent on task, and availability of reference resources. Participants were randomly assigned to either a physical or virtual lab group. Pre- and post-assessments of conceptual understanding and inquiry skills were administered to both groups. Paired-samples t tests revealed a significant mean percent correct score increase for conceptual understanding in both the physical lab group (M = .103, SD = .168), t(38) = -3.82, p < .001, r = .53, two-tailed, and the virtual lab group (M = .084, SD = .177), t(44) = -3.20, p = .003, r = .43, two-tailed.  However, a one-way ANCOVA (using pretest scores as the covariate) revealed that the main effect of lab group on conceptual learning gains was not significant, F(1, 81) = 0.081, p = .776, two-tailed. An omnibus test of model coefficients within hierarchical logistic regression revealed that a correct response on inquiry pretest scores was not a significant predictor of a correct post-test response, χ2(1, N = 84) = 1.68, p = .195, and that when lab mode was added to the model, it did not significantly increase the model’s predictive ability, χ2(2, N = 84) = 1.95, p = .377. Thus, the data in the current study revealed no significant difference in the effect of physical versus virtual manipulatives when used to teach conceptual understanding and inquiry skills related to density.

Importance of the Study. The results of the current study confirm recent quantitative findings that suggest non-traditional labs (NTL), in this case, virtual labs, can, at least under certain circumstances, be used to achieve learning outcomes related to conceptual understanding and scientific inquiry equally as well as traditional labs (TL) [1].  Previous studies supporting equivalent achievement in NTL seem to place emphasis on content knowledge and understanding (K), and thus quizzes and exams as the instrument of assessment, whereas studies supporting higher achievement in TL seemed to rely heavily upon qualitative data related to student and/or instructor perception (and thus surveys as the instrument of assessment) [1,2].  The current study assessed conceptual understanding and quantitatively confirmed the trend noted in the literature reviewed regarding the K outcome of the KIPPAS tool [1].  Given the lack of literature regarding the I outcome of the KIPPAS tool, despite inquiry being at the core of scientific learning, the current study also quantitatively confirmed the trend noted in the literature regarding the I outcome of the KIPPAS tool, and contributed much needed data to the discussion of NTL efficacy regarding the teaching of scientific inquiry skills.

The Literature Void. The current study contributes empirical data of lab type comparison (hands-on vs. virtual) to a body of literature in which comparative empirical data, especially quantitative data, is severely lacking.  Most studies on lab type efficacy have heretofore not been experimental or comparative, and have been largely based on measuring whether learning was enhanced when virtual labs are used as supplements, not as stand-alone instructional methods, or else how students and/or instructors “perceived” learning gains and the overall experience.  Most studies are survey-based or case studies, and very few large scale empirical efficacy studies have been performed.  It is known that virtual labs have been shown to enhance student learning, and some empirical studies have even shown that they can be suitable replacements for hands-on learning [3-8].  But such studies often fail to control for key variables, such as time spent on task, instructor, instructional methods, and curriculum, all of which were controlled in the current study.

Implications of the Study. Given the economic advantages of NTL, if the data continue to accumulate and provide empirical evidence that learning outcome achievement can occur at least equally as well with NTL as it can with TL, it may challenge current positions of some accrediting, certifying, and standards/quality assurance organizations.  Such evidence could, for example, make a case (at least situationally) for NTL being

. . . an acceptable, accessible, and cost-effective alternative to in-person, hands-on labs.  Research to confirm equivalent outcomes would also mean that governing organizations like the College Board, ACM, and NSTA should consider simulated labs equivalent to hands-on labs and, thus, acceptable practices for science laboratory requirements.  If this occurs, the definition of “hands-on” will no longer be limited to students touching physical materials, but will instead emphasize their mental “minds-on” engagement with the science topics they are studying [9].

Currently no consensus exists regarding NTL efficacy, and support of traditional laboratory learning environments over non-traditional ones has heretofore not been based on empirical evidence related to measured student learning outcome achievement. Thus, further empirical studies related to learning outcome achievement by learning objective category are needed. Large variability in the outcomes being measured could benefit the discussion, but discussions and conclusions must be within clear categorical boundaries (i.e., use a KIPPAS-like categorization), with any instructor preference or weighted importance of one category over the other made transparent.  Otherwise, meaningful, unambiguous comparisons cannot be made.  This research study is the first study to offer empirical and quantitative data within such boundaries.

As a specific example of the possible implications the data from this and similar studies could have on positions towards NTL, several years ago the College Board, the agency that accredits Advanced Placement (AP) secondary level classes for college credit in the United States, issued a position statement saying that virtual labs could not be part of a school’s AP curriculum, though this statement was recanted within months [10-11].  Currently, institution representatives wanting any courses to receive AP accreditation must submit a proposal and justification for the use of any virtual labs, and must receive written permission from the College Board, but curriculum standards are being rewritten for clarity considering this issue, and it is surmised that soon no conditional authorization will be permitted [10].  Also in the United States, the National Science Teachers Association (NSTA) and the American Chemical Society (ACS) explicitly denounced the substitution of NTL for TL [12-13].  These agencies influence or establish the standards by which science teachers and chemistry programs are accredited.

Data from the current study (and similar studies) has the potential to affect program entry requirements and transfer/articulation agreements as well.  Many pre-professional schools in the United States are not accepting for transfer online courses that utilize NTL. For example, per the 2007 Articulation Agreement between Mississippi Board of Trustees of State Institutions of Higher Learning (IHL) and Mississippi State Board for Community and Junior Colleges, the articulation agreement does not allow for online science courses to be accepted for admission into the School of Pharmacy, Medical School, or Dental School [14].  Community and Junior Colleges are a very important link to a four-year degree and/or a pre-professional program for many students [15].

The Take-Aways. (1) Attendees will have quantitative evidence and statistical data in support of no significant difference in conceptual learning gains or acquisition of inquiry skills between students who use virtual versus physical laboratory manipulatives. In other words, data that shows students do not necessarily learn scientific inquiry or acquire scientific knowledge better in a physical lab environment than they do in a virtual lab environment. Such information could have curricular and economic applications for future development and instruction of STEM lab courses. (2) Attendees will be equipped with data that supports the development of distance science courses using virtual labs, which can significantly reduce costs, increase demographic and geographic reach, and increase student access, all while retaining academic rigor, effective learning, and the same course learning objectives. (3) Attendees will have a clearer understanding of the efficacy debate regarding virtual labs in online STEM education and the impact it is having on distance science education, quality, development and offerings, and will be acquainted with empirical data that can help provide clarity in the conversation and movement towards a consensus. (4) Attendees will have a clearer understanding of the implications and importance of this topic to science education policy, such as with accreditation, transfer and articulation, economics, and professional school prerequisite acceptance.

References: http://bit.do/OLCrefs