Using a program evaluation of the online EdD in Educational Leadership as a model, participants will begin to design their own program evaluations. Participants will have opportunities to identify their own questions and brainstorm on ways to create and implement a program evaluation for online programs at their institution.
As a result of the increasing demand for tertiary education in many developing countries (Perraton, 2007) higher education institutions are seeking ways to increase educational access in difficult economic times. The University of the West Indies (UWI), a regional university serving seventeen countries across the Caribbean region, has charged the UWI Open Campus (UWIOC) with the mandate to deliver online education. Through the Doctor of Education in Educational Leadership, UWIOC is using online education to provide graduate education to students scattered across the islands of the Caribbean who may not have been able to access doctoral study in their home countries for a variety of reasons, including the inability to leave families and work to study face to face in another island.
Although there are still many traditional doctoral (i.e. residential PhD) programs, there is an increasing shift to non-traditional doctorates such as the EdD (Archbald, 2011) which are targeting geographically separated students seeking to apply research and theory to professional practice (Kumar & Dawson, 2012). Similarly, the UWIOC EdD is aimed at current and prospective leaders in K-12 and higher education across the Caribbean region and is the first professional doctorate being offered by the UWI.
Despite the fact that the first cohort has not yet graduated, it was decided that an evaluation of the program implementation would be useful in order to a) measure program effectiveness, b) identify areas of weakness for improvement (Rovai, 2003) and c) use the data for program decision making. This study is particularly concerned with evaluating the quality of the student experience in the programme so as to ensure that the students receive the highest quality education and are equipped to operate effectively as leaders in their educational context.
The research literature on online doctoral programs identifies several areas of concern, including development and implementation issues (Koehler et. al, 2013; Miller & Curry, 2014), the quality of student online experiences (Ivankova & Stick, 2007), student professional growth (Buss, Zambo, Zambo & Williams, 2014; Kumar & Dawson, 2014;), and student perceptions on the program (Kumar & Dawson, 2012).
An evaluation can be described as a measure of the value, merit or worth of an entity, which in this case is the educational program under consideration. There are relatively few examples of the application of program evaluation methods to online programmes in the literature, specifically online graduate programs (Chapman, 2005). While program evaluation is a relatively well developed academic field in its own right, the intersection with online graduate programs does not seem to be well developed as yet and this research still will be part of that emerging literature.
Research Questions
The research questions are as follows:
- What are stakeholder (i.e. students and facilitators) opinions of the programme design, course content, facilitation and support mechanisms?
- Are students being successful in the courses?
- Is the EdD programme cost effective thus far?
- Is the EdD meeting the programme objectives?
- What challenges have been encountered in the programme implementation?
- How can the programme can be improved to enable it to better meet its objectives?
Methods
The research study design is a qualitative case study evaluation of the EdD in Educational Leadership, using a single case, intrinsic case study design. This evaluation will also take into account the Joint Committee on Standards for Educational Evaluation’s Program Evaluation Standards, namely: feasibility, propriety, accuracy and evaluation accountability. The evaluation is bounded by the first three years of program implementation and is an implementation evaluation.
The data gathered to answer the research questions will include online surveys of administrators, faculty and students, student grades, program documents, and participant interviews. Data analysis will use qualitative methods for the program documents, surveys and interviews and quantitative methods for the student grades.
Early Results
At the time of writing the study is still in progress but early results indicate that students are generally satisfied with the program design, course content, facilitation and support mechanisms. Some implementation issues have been tentatively identified, including program structure, facilitation and costs, but these need to be fully developed.
Session Outcomes and Interactivity
The session will outline the methodology and key findings of the research in order to demonstrate how an evaluation of an online program can be done. Participants will be able to:
- Describe the research context, questions, methods and results of the evaluation
- Use the example given to outline a model for program evaluation
- Begin to create research questions for their own evaluations
- Begin to link methods of data collection to research questions in an evaluation
- Provide comments on the research
In order to promote engagement, participants will be asked to work together in small groups to complete a chart on designing a program evaluation which will link the evaluation type and research questions to data collection methods. Think-pair-share will be one of the techniques used to generate discussion and creativity.
Draft Presentation Plan
The following is an outline of how the presentation could be organised as an education session to provide participants with opportunities to consider their own online program evaluation needs:
- Presentation of the research context, type of evaluation, research questions and selection of data to answer the questions. 10 mins
- Think pair share activity on evaluation questions and data selection 15 minutes
- Presentation of key research findings 10 mins
- Questions and Answer 10 mins