Abstract

Various instructor qualities have been shown to have variable influences student outcomes in online courses. In this study, the impact of faculty quality on student performance was examined by comparing student performance when taught by excellent faculty and average faculty (determined through administrative evaluations). We examined 328 student grades from a 2019 January term 200 level elementary statistics course. Student performance measures included final course scores, grade distributions and pass rates. Learning modality (asynchronous vs. synchronous) and course design (e.g. syllabus, textbook, assignment and assessment design, etc.) as confounding variables were controlled through the template approach implemented at the study institution. The statistical analysis of the influence of modality ensured that modality was a controlled variable as there was no statistically significant difference in student outcomes across the online course modalities studied. There was no statistically significant difference in learner outcomes between students who were taught by faculty rated excellent compared to those taught by faculty rated average. It is likely that the standardized nature of the courses combined with the mandatory faculty training in online teaching and learning best practices minimized differences between instructors. Future research should examine more specific indicators of faculty quality such as social presence in online discussions and teaching presence in feedback. Furthermore, future research could examine the influence of faculty gender on student outcomes in online modalities.

Introduction

In the age of COVID-19 with so much instruction moved to distance learning, there is renewed interest in ensuring that undergraduate distance learning is as effective as traditional face-to-face learning. Although this research was conducted prior to COVID-19 (loosely defined as March 2020), lessons can still be learned about the efficacy of distance learning. This study was conducted at a campus with accelerated terms with a non-traditional student body. The advantage of this setting was that possible confounding variables of standardized syllabi, text, course content, homework, quizzes and assignments were eliminated due to award winning course design (U.S. News and World Report, 2019). Additionally, faculty training and evaluation procedures were robust to facilitate quality instruction. These conditions allowed for a more direct evaluation of intended variables such as the impact of instructor quality on end of course grades, grade distribution and pass rates.

Year over year, online learning is increasing in popularity within higher education, both by students as seen by enrollments and by institutions as seen by course offerings. Online pedagogy has distinct differences from traditional teaching. Online faculty describe effective online teachers as engaging in the following actions: 1) providing constructive feedback, 2) fostering interaction and engagement, 3) facilitating learning, 4) maintaining instructor presence to reduce transactional distance, 5) and establishing strong organization (Lewis & Abdul-Hamid, 2006). Additionally, cultural responsiveness is a key indicator of an effective online instructor (Heitner & Jennings, 2016). Higher education administrators describe online faculty effectiveness similarly, with the qualities essentially boiling down to the Community of Inquiry presences: social presence, teaching presence, and cognitive presence (Samora, 2013), with facilitation being identified as the most important characteristic. Facilitation is a broad term that encapsulates high interactivity, a strong student focus, responsiveness, encouragement, building a sense of community, and launching dialogue (Samora, 2013). It is important to note that instructor immediacy and presence in an online course may influence more than just pass rate and persistence, with other impacts including affective learning, cognition, and motivation (Baker, 2010).

As with traditional faculty, online faculty are evaluated by the administration of their institution to ensure effectiveness. Most assessment of online faculty relies on student surveys, followed by institutional methods that largely rely on course design, not the actions of the instructor (Pina & Bohn, 2014). Interestingly, student surveys tend to identify the same key characteristics of effective faculty that are identified by faculty and their administrators: organization, clarity, and feedback (Schubert-Irastorza & Fabry, 2011). Other assessment measures used include metrics from the learning management system and peer evaluations (Pina & Bohn, 2014). Assessment rubrics used by institutions tend to be Quality Matters (which primarily focuses on instructional design, not teaching) or those developed in-house (Pina & Bohn, 2014). Ideally, online faculty are evaluated based on objective and observable assessment criteria. Characteristics of quality evaluations of online faculty include critical commenting by generating narrative summaries of strengths and weaknesses of the instructor from observations by administrators and peers (Tobin, 2004). The evaluation should also include a formative component (Mandernach, Donnelli, Dailey, & Schulte, 2005). Characteristics identified for assessing online instructor quality include faculty login frequency, presence of a faculty biography, regular posting of announcements, conciseness of announcements, responding to student inquiries, timeliness of responses, participation in discussions, moderation of discussions, and providing high-quality feedback on assignments (Pina & Bohn, 2014).

Faculty quality inevitably influences student outcomes in a course. In fact, significant variation in student performance in online learning has been demonstrated across instructors (De Vlieger, Jacob, & Stange, 2017). Students who learn from an instructor with high quality (evaluated by students) are more likely to persist in the course and earn higher grades (Hoffmann & Oreopoulos, 2009). Academic learning time – the time students spend successfully on learning activities relevant to course goals – positively correlates with course grade (Frick, Chadha, Watson, & Zlatkovska, 2010). This supports the notion that high-quality instructors – ones who demonstrate social, teaching, and cognitive presence – will positively influence student learning outcomes by promoting productive time on task.

Moderating variables may influence the relationship between instructor quality and learner outcomes. There is some disagreement in the literature regarding the influence of faculty status (tenure-track, non-tenure track, adjunct, or graduate student) on student performance. Some report no influence on student outcomes (failure rate or withdrawal) (Hoffmann & Oreopoulos, 2009) while others report students are more successful (lower failure rate and less likely to withdraw) when learning from a full-time online faculty member (Mueller, Mandernach, & Sanderson, 2013). The research versus teaching focus of the faculty does not seem to influence student outcomes (Hoffmann & Oreopoulos, 2009). In traditional teaching, a same-gender instructor increases grade performance of students (Artz & Welsch, 2014; Hoffmann & Oreopoulos, 2009). It is unclear at this time if this holds true in online teaching and learning.

There are many moderating variables that may influence student performance and persistence in online learning beyond faculty performance. Instructor prior experience teaching the course may influence instructional effectiveness (De Vlieger et al., 2017). Class size does not appear to be a moderating variable in online learning (Bettinger, Doss, Loeb, Rogers, & Taylor, 2017) the way it is in traditional learning (Bedard & Kuhn, 2008). Student age and gender have mixed results in the literature on their moderation of student learning outcomes in online learning (Amro, Mundy, & Kupczynski, 2015; Kupczynski, Brown, Holland, & Uriegas, 2014; Meiselwitz & Sadera, 2008). While learner characteristics cannot be controlled in investigating the influence of instructor efficacy on outcomes, using a standardized curricula and assessments can help control moderating variables.

Purpose

This research has meaning to higher education learning institutions with regard to student performance based on instructor quality, particularly in (Science, Technology, Engineering and Mathematics (STEM) oriented courses such as introductory statistics. Instructor quality is but one factor in any classroom setting. Other factors include course design including syllabi, homework, quizzes testing, and the construction of the course in the learning management system. The purpose of this study was to determine if instructor quality impacted student performance when variables such as course design and course construction were standardized.

The course used in the study

The 200 level statistics course was instructed using all four learning modes. Because statistical thinking involves the ability to make sense of results; statistical thinking demands more than the ability to accomplish complicated calculations. Students develop statistical thinking skills through numerous “simulated real world” examples, exercises and discussions. The course is designed as an introduction to basic statistics. Curriculum emphasizes understanding concepts of statistics by incorporating illustrative aviation examples, identification of required assumptions, and underlying theory.

Hypotheses

Ha1. Student end of course scores in courses taught by faculty rated excellent will differ from student end of course scores taught by faculty rated average.

Ha2. Student end of course scores and faculty quality are related.

Ha3. Student pass rates and faculty quality are related.

Methods

Participants

The population for this study was undergraduate students enrolled in the January 2019 term of an online introductory statistics course at a medium-size private university (n = 328). Online course modalities included asynchronous online, video synchronous home, video synchronous classroom, and classroom. All sections were taught using streamlined content and learning outcomes.

Learning Modes

The campus used for this study offered courses in four formats:

Classroom – Students and the instructor were co-located in a traditional classroom environment. Classroom courses were nine weeks in length with regularly scheduled meeting times typically one session per week.

Videosynchronous (Classroom) - Students were located at multiple satellite locations accommodating smaller locations to increase class sizes. Students were connected synchronously with the instructor during regularly scheduled meeting times. The instructor could travel one or all locations for enhanced communications with students, however, all students meet videosynchronously at the same time at least once a week during nine planned class sessions.

Videosynchronous (Home) - Students were located at home and the instructor taught from their home. This delivery mode accommodates students who could not physically attend traditional courses at a campus. Students connected virtually with the instructor via videosynchronous software during nine regularly scheduled meeting times (once a week).

Online - Courses were created using a learning management system (Canvas software) and delivered using methods that do not rely on meeting times. Online courses were nine weeks in length and are completely asynchronous.

All four modes used the same text, discussions, homework, quizzes, tests syllabus and assignments. All courses were taught in 9 week terms. Classroom, Videosynchronous Classroom and Videosynchronous Home courses met for 3 hours and 20 minutes face to face and 90 minutes of interactive online instruction. For this reason, these possible confounding variables were controlled in this research (Lou, Bernard, & Abrami, 2006).

Student (n=328) performance data was collected from the learning management system. The data was used to calculate grade distribution and pass rate. Students enrolled during the study period were non-traditional students. Across all distance students at the institution, 47% were between the ages of 25 to 34 and 42% were over the age of 35. Only 3% of distance students at the institution were under 21 years old. Over 2,000 students per year take the statistics course to satisfy general education requirements of their degree programs.

Faculty quality was based on quality manager observations, student end of course surveys and department chair annual evaluations resulting in faculty ratings of 1 (excellent) or 2 (average). Thirteen faculty taught during the data collection period. One faculty was full time, the remaining 12 were adjunct faculty.

All data were aggregated, with no individually identifying information collected from either the student or instructor population, ensuring anonymity. The Institutional Review Board (IRB) granted this retrospective analysis exempt status (20-127).

Course Content

The course used for data analysis was in introductory statistics. Learning objectives covered types of data, sampling, measures of central tendency, and measures of variation, normal and binomial distributions, hypothesis testing using Z and T distributions and regression. The course used a complementary third party software providing homework, quizzes and tests providing effective prompts for the homework and randomly ordered questions for each student (Triola, 2018). Discussion boards focused on course concepts such as computing means, standard deviations and Z scores in Microsoft Excel and interpreting the results. Assignments focused on communicating the meaning of statistical results in realistic fictional scenarios.

Faculty Professional Development

All faculty completed at least one faculty development course prior to teaching their first course for the university. The initial faculty development course focused on how to teach online, use the Canvas learning environment and promote communication and interaction with students. A follow-up course was a just in time videosynchronous learning platform course prior to the first time each faculty member taught a video based course. Faculty completed coursework on how to use the videosynchronous platform. In the last part of the course, faculty demonstrated expertise with the Saba Centra videosynchronous learning platform to a university Center of Teaching and Learning evaluator prior to teaching a video based course. The completion of each course was codified in faculty records ensuring only pre-certified faculty would be able to teach online and videosynchronous courses.

Design

The research design was a cross sectional retrospective analysis of aggregate data.

Treatment of the Data

The first data analysis evaluated differences in learner outcomes by online course modality. This ensured that course design differences were eliminated as possible confounding factors (Lou et al., 2006). Data were first compared by instruction mode to include; online, classroom, synchronous video classroom or synchronous video home to control for modality. It is important to note that all four modes of instruction used the same course design, syllabus, book, homework assignments, quizzes, and tests eliminating these factors as possible confounding variables (Lou et al., 2006). Modality comparisons used a One Way Analysis of Variance for end of course grades and Chi Square for grade distributions and pass rate comparisons. Although a typical alpha for a study is .05, a Bonferroni corrected alpha (α=.017) was applied in all statistical testing in this research because of the multiple measures of the three hypotheses on the data set used for analysis. This adjustment was made to avoid any “Type 1” errors in hypothesis testing (Gay, Mills, & Airasian, 2009; Gould & Ryan, 2012).

Data were then examined based on instructor ratings of 1 (excellent) or 2 (average). The end of course score comparison used a t-Test for Independent Samples. The grade distribution comparison and pass rate comparison was conducted using Chi Square (α=.017) (Gay et al., 2009; Gould & Ryan, 2012).

Results

Evaluating Modality as a Possible Confounding Variable

The possible impact of learning mode as a confounding variable was assessed by reviewing data for end of course scores, grade distributions and pass rates with regard to student performance based on learning mode.

End of Course Grade

Summary data indicated that mean grades for the four modes of instruction were between 88.89% and 81.83%. Medians ranged between 92.40% and 90.80% (Table 1).

Table 1



The summary data shown in Table 1 were evaluated to determine if student end of course score was significantly impacted by learning mode. Levene’s Test for Homogeneity of Variance did not indicate significant variability between the four groups (p=.0874). Analysis of Variance results indicated that the difference between the four learning modes was not statistically significant at the α=.017 level [F(3, 324) = 1.01, p = 0.388. In this study, learning mode did not appear to have a significant impact on student end of course scores.

End of Course Grade distributions

Data were compared based on learning mode to determine its impact on student performance. These data were also used to determine if the pass rates were significantly different based on learning mode. Data are shown in Table 2.

Table 2
Summary Data: End of Course Grade Distribution Based on Learning Mode


A Chi-Square analysis was used to analyze the data in Table 2. The data did not provide enough evidence to reject the null hypothesis of no significance; α =.017, X2(12), 9.867, p = 0.627. In this study, learning mode and grade distribution did not appear to be related. These data can be visualized by looking at grade distribution based on percentages for each learning mode as shown in Figure 1.

Figure 1

Final Course Grade Distribution Percentage by Learning Mode


The distribution of final course grades was relatively even across the four different modes of learning. These data further support the argument of no discernible difference between learning modes.

Pass Rates
The overall (all learning modes) pass rate was 92%. When examined by modality, the range of pass rates was between 88% and 100%. Data can be better visualized by examining Figure 2.

Figure 2

Grade distribution percentages based on learning mode

The pass rates ranged between 88.89 for classroom to 81.83 for videosynchronous home. Frequency counts for each category were evaluated using a Chi Square test of independence. There was no discernable statistical relationship between pass rates and modality; α =.017, X2(3), 3.658, p = 0.301. In this study, learning mode and pass rates did not appear to be related.

Influence of Instructor Quality on Learner Outcomes

The hypotheses of this research involved instructor influence on student performance indicators identified as end of course scores, grade distribution and pass rates.

Instructor Influence on End of Course Scores

Data were then compared to evaluate the first hypothesis regarding the impact of instructor quality on student performance. The means were within one point of each other. The medians of students taught by faculty rated excellent and faculty rated average were within one point as well. Summary data are shown in Table 3.

Table 3

Summary Data: End of Course Scores Based on Instructor Quality



Student end of course scores from the two groups (instructors rated excellent and instructors rated average) from Table 3 were compared. Levene’s Test for Homogeneity of Variance did not indicate significant variability between the two groups of data (p=0.766). The t-Test for Independent Samples did not yield statistically significant difference in student performance based on instructor quality at α =.017, t(80.65) = 0.022, p = 0.982. In this study, instructor quality did not appear to have a significant impact on student end of course scores.

Instructor Influence on Grade Distribution

Grade distributions were then compared by instructor quality to determine its impact. These data were also used to compare the pass rate between students who were taught by faculty rated excellent and students who were taught by faculty rated average. The grade distributions are shown in Table 4.

Table 4

Summary Data: Grade Distribution Based on Instructor Quality


A Chi-Square analysis was used to analyze the data in Table 4. The data did not provide enough evidence to reject the idea of no significance; α =.017, X2(4), 4.40, p = 0.354. In this study, instructor quality and grade distribution did not appear to be related. This data can be more easily visualized in Figure 3.

Figure 3

Final Course Grade Distribution Percentages based on Instructor Quality

The final course grade distribution of faculty rated excellent and faculty rated average is relatively similar. Note the same rates for “As” and similar percentages for “F” grades. All percentages were within 6% between faculty rated excellent and faculty raged average.

Instructor Influence on Pass Rates

A final Chi Square test of independence was performed to determine the impact of instructor quality on pass rates. 94% of students who took the class with an instructor rated excellent passed the course. Likewise 91.7% of students who took the course with instructors rated average achieved a passing score. The data did not provide enough evidence to reject the idea of no significance; α=.017, X2(1), 0.457, p = 0.499. In this study, learning mode and pass rates did not appear to be related.

Conclusions

In this study, student end of course scores, were not significantly different based on learning mode or instructor quality. Some previous studies have reported differences in student performance based on learning mode which was why the researchers examined the data based on modality as well as instructor quality. Student grade distributions and pass rates did not appear to be related based on learning mode or instructor quality.

Reasons for the findings may include standardization of course design and delivery to include syllabi, text books, homework assignments, quizzes, testing and course layout within the learning management system. The university under study also had an instructor training program that each faculty member has to complete before teaching their first course. These measures controlled for many possible confounding variables.

Limitations

Results of this study should be viewed in light of some factors possibly limiting generalizability of results.

This study was at a Worldwide campus with non-traditional students that have an average age of 34. Approximately 50% of the students were serving in the military, another 30% were military affiliated (spouse or retired). The age or types of students involved in this study may differ from other college or university campuses. Military student demographics in higher education are similar to non-traditional students (Ford & Vignare, 2015). Like non-traditional students, military students tend to complete their coursework online (Ford & Vignare, 2015).
The term lengths at the campus studied are nine weeks. Many universities have terms or semesters of differing lengths which could impact a study on student performance. Additionally, data were taken from the January2019 term. Results from different terms during the year or different term lengths could differ (Austin and Gustafson, 2006).

The university under study had an instructor training program that each faculty member was required to complete before teaching their first course. Additionally, faculty understand they could be observed by a quality manager at any time. All faculty were observed and evaluated at least once per year. The method for training and evaluating faculty are additional factors to consider if this study were repeated in a different setting.

Recommendations

This study should be replicated in traditional college settings to determine if term lengths are related to student performance. The length of the term could be related to possible impacts of excellent or average faculty (e.g., a longer term could influence the impact of faculty on student performance).

Student age should also be examined to determine the impact of instructor quality. First time in college students are a subgroup that could be further examined.

Gender is another possible moderating variable on student performance. Flanagan noted that females tended to be less successful in online statistics courses (2012).

Learning styles and academic preparedness should be considered in future research to determine their impact on learning mode selection and performance.

References

Amro, H. J., Mundy, M. A., & Kupczynski, L. (2015). The effects of age and gender on student achievement in face-to-face and online college algebra classes. Research in Higher Education, 27

Artz, B., & Welsch, D. M. (2014). The effect of peer and professor gender on college student performance. Southern Economic Journal, 80(3), 816-838. https://doi.org/10.4284/0038-4038-2012.158

Austin, M., & Gustafson, L. (2006). Impact of course length on student learning. Journal of Economics and Finance Education, 5(1). http://citeseerx.ist.psu.edu/v...

Baker, C. (2010). The impact of instructor immediacy and presence of online student affective learning, cognition, and motivation. Journal of Educators Online, 7(1), 1-30.

Bedard, K., & Kuhn, P. (2008). Where class size really matters: Class size and student ratings of instructor effectiveness. Economics of Education Review, 27(3), 253-265. doi:https://doi.org/10.1016/j.econedurev.2006.08.007

Bettinger, E., Doss, C., Loeb, S., Rogers, A., & Taylor, E. (2017). The effects of class size in online college courses: Experimental evidence. The Economics of Education Review, 58, 68-85. doi:https://doi.org/10.1016/j.econedurev.2017.03.006

De Vlieger, P., Jacob, B. A., & Stange, K. (2017). Measuring up: Assessing instructor effectiveness in higher education. Education Next, 17(3), 68-75.

Ford, K., & Vignare, K. (2015). The evolving military learner population: A review of the literature. Online Learning, 19(1), 7-30.

Flanagan, J. (2012). Online versus face-to-face instruction: Analysis of gender and course format in undergraduate business statistics courses. Academy of Business Research, 11, 93-101.

Frick, T. W., Chadha, R., Watson, C., & Zlatkovska, E. (2010). Improving course evaluations to improve instruction and complex learning in higher education. Educational Technology Research and Development, 58, 115-136. https://doi.org/10.1007/s11423-009-9131-z

Gay, L. R., Mills, G. E., & Airasian, P. W. (2009). Educational research: Competencies for analysis and application (9th ed.). Upper Saddle, NJ: Pearson.

Gould, R. N., & Ryan, C. N. (2012). Introductory statistics (1st ed.). Boston, MA: Pearson.

Heitner, K. L., & Jennings, M. (2016). Culturally responsive teaching knowledge and practices of online faculty. Online Learning, 20(4), 54-78.

Hoffmann, F., & Oreopoulos, P. (2009). Professor qualities and student achievement. The Review of Economics and Statistics, 91(1), 83-92. https://doi.org/10.1162/rest.91.1.83

Kupczynski, L., Brown, M., Holland, G., & Uriegas, B. (2014). The relationship between gender and academic success online. Journal of Educators Online, 11(1)

Lewis, C. C., & Abdul-Hamid, J. (2006). Implementing effective online teaching practices: Voices of exemplary faculty. Innovative Higher Education, 31, 83-98. https://doi.org/10.1007/s10755-006-9010-z

Lou, Y., Bernard, R. M., & Abrami, P. C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research and Development, 54(2), 141-176. doi:10.1007/s11423-006-8252-x

Mandernach, B. J., Donnelli, E., Dailey, A., & Schulte, M. (2005). A faculty evaluation model for online instructors: Mentoring and evaluation in the online classroom. Online Journal of Distance Learning Administration, 8(3)

Meiselwitz, G., & Sadera, W. (2008). Investigating the connection between usability and learning outcomes in online learning environments. Journal of Online Learning and Teaching, 4(2), 234-242.

Mueller, B., Mandernach, B. J., & Sanderson, K. (2013). Adjunct versus full-time faculty: Comparison of student outcomes in the online classroom. Journal of Online Learning and Teaching, 9(3), 341-351.

Pina, A. A., & Bohn, L. (2014). Assessing online faculty: More than student surveys and design rubrics. Quarterly Review of Distance Education, 15(3), 25.

Samora, D. L. (2013). Characteristics of the effective online teaching faculty: Perspectives of online university administrators (Ed.D.). Retrieved from https://digitalcommons.liberty.edu/cgi/viewcontent.cgi?article=1748&context=doctoral

Schubert-Irastorza, C., & Fabry, D. L. (2011). Improving student satisfaction with online faculty performance. Journal of Research in Innovative Teaching, 4(1), 168-202.

Tobin, T. (2004). Best practices for administrative evaluation of online faculty. Online Journal of Distance Learning Administration, 7(2)

Triola, M. F. (2018). Elementary statistics: Using excel. (6th ed.). Boston: Pearson

U.S. News and World Report. (2019). Embry-riddle aeronautical university. Retrieved from https://www.usnews.com/best-colleges/embryriddle-aeronautical-university-1479