Abstract

Change management processes and learning adaptations that accompany quality assurance (QA) in higher education are an understudied phenomenon. This article describes a first-tier, phenomenological qualitative research study on the usability of the Continuum of Excellence in Quality Assurance (CEQA) model as a tool that institutions might use to identify, assess, and strategically embed institution-wide processes and actions for sustainable quality online learning for all stakeholders. Twelve educators with histories of implementation of Quality Matters tools and processes as part of their institutional approach to assure quality online learning were interviewed to gather information about their efforts to implement QA tools and processes and to map to the model. Five emerging themes are discussed.

Introduction and Background

Change management processes and learning adaptations that accompany quality assurance in higher education are an understudied phenomenon. While the term “quality assurance” (QA) is an often-used phrase in higher education, it is also a term fraught with vague definitions and disconnected activities. The range of options that are defined as QA fall on a continuum of implementations ranging from grand administrative, visionary schemes without practical execution, to faculty and staff champions informally practicing solutions at a course, department, or program level that may not be connected to any broader institutional context. However, random implementations of QA tools and processes makes it challenging for institutions to identify contextual inputs, to measure outputs, to strategically connect outcomes, and to assess efforts with the precision required to ensure continuous improvements and to provide evidence of implementation status. Additionally, simply having access to QA standards and tools does not ensure they will be implemented and measured.

It is vital, first, to define “quality assurance,” and, second, to examine online learning QA through an appropriate lens. Drawing on several experts (Chaloux & Miller, 2014; Christensen, 2018), we define QA as: identifying and learning in strategic, productive increments via established pathways and assessments of organization-wide targeted outcomes and build on previous research that has explored QA from a change management perspective (Adair and Shattuck, 2019). In previous research, Adair and Shattuck (2019), focused on Kotter’s eight-step model (1996, 2006) as the most relevant, and in this article, we extend our examination of that model. Kotter’s change management process begins with establishing a sense of urgency for the change and moves on to building coalitions in support of the change, articulating and communicating a vision, accomplishing small wins, and eventually instituting the change organization-wide. We also incorporate an alternative model of organizational change put forth by Nadler and Tushman’s (1998) framework where incremental change or frame bending is distinguished from radical or frame breaking change.

Adair and Shattuck (2019) coupled their change management approach with the learning organization theories of Argyris (1976), Garvin (1993), and Garvin, Edmondson, and Gino (2008). These theories describe the abilities of some organizations to engage in continuous improvement via the capacity for enhanced learning within an organization called double-loop learning. Adair and Shattuck noted,

In essence, successful innovations require the kind of organizational learning that is captured in meaningful quality assurance processes that reflect an organization’s capacity for systematic problem solving, benchmarking on self and others, and sharing and integrating that knowledge throughout the organization. Quality assurance, rather than stifling innovation with prescriptive and rigid processes, is fundamental to the successful dissemination of innovations. (p. 102)

Implementation of Quality Assurance in Higher Education: A Framework

The Continuum of Excellence in Quality Assurance (CEQA) model was developed by Dr. Deborah Adair, QM executive director, “for the recalibration of processes to make quality criteria relevant for online learning innovations” (Adair & Shattuck, 2019, p. 110). It is based on years of Dr. Adair’s observations and interactions with leaders and practitioners of online education, as well as Adair’s academic background in organization management. The model was informed by the Capability Maturity Model of Integration (CMMI) framework, which was developed at Carnegie Mellon University’s Software Engineering Institute (Paulk, Curtis, Chrissis, & Weber, 1993) and grew out of the needs associated with developing computer software and, later, the integration process focused on improvements. The CMMI framework “includes a self-assessment that presents the organisation’s best practices in key process areas (e.g., capabilities) and then shows how the organisation can redefine its capabilities as it evolves into a more mature state” (Yeh, Adams, Marshall, Dasgupta, Zhunushov, Richards, & Hay, 2017, para. 12).

The CEQA Model includes five levels of pathways to excellence, moving from informal, sporadic actions (Ad Hoc) through internal assessments (Quality Evaluation); adding regular and ongoing processes for action, feedback, and revision (Continuous Improvement); adding comparison with other organizations that have implemented the innovation (Benchmarking); and embedding institution-wide processes and actions resulting in sustainable quality online learning for all stakeholders (Institutional Change). The last stage of the model assumes the organization is capable of continuous improvement, consistent with previously mentioned capabilities of learning organizations.



In their 2019 presentation of this model, Adair & Shattuck posit which of Kotter’s change management processes are involved with each of these steps. The discussion section of this paper will revisit these steps and provide more illustrative examples of change management strategies at work, supported by data in this study.

The Problem

Models and frameworks are conceptual illustrations of relationships that can be used to organize, discuss, and analyze a phenomenon. Although the CEQA model has been well received and deemed as potentially very useful when presented during professional conversations and conference presentations, the model needed validation as a potentially useful framework for identifying, tracking, and assessing the implementation of QA tools and processes in higher education. At the same time, the change management processes used to ground the model have not been tested in any systematically collected organizational data.

Quality Matters, with an institutional membership in the U.S. of more than 1,200 higher education institutions, provided the population from which a sample was drawn to validate the usefulness of the CEQA model. Since 2006, institutions have adopted and adapted various features of the quality assurance tools and processes, which are focused specifically on design and faculty preparedness, made available to QM member institutions. The adaptation of these tools is voluntary and meant to be localized within an institution’s cultural and organizational context. Additionally, QM promotes the online learning Quality Pie (https://www.qualitymatters.org/why-quality-matters/process), which acknowledges the interactive, interrelated features that need to be present within an organization to foster quality assurance, including institutional support and infrastructure for student readiness, faculty readiness, course design, course delivery, and technology..

QM’s membership base comprises a full representation of higher education institutions, including community colleges, flagship institutions, and private four-year institutions. Among the membership, administrators, faculty, and staff who have an active history of implementing QM tools and processes provide an opportunity to gather data from a wide and varied community of practice (Shattuck, 2007) that has: shared histories of identifying and implementing solutions to common problems (e.g., the quality assurance of online learning); shared common language in describing course design and professional development markers (quality standards); and common QM tool and process artifacts. QM provides a framework of consistent and measurable official artifacts as well as flexible, localized adaptations and, for a number of institutions, constitutes their initial foray into quality assurance for online education.

While QM does not track or document the use of QM tools and processes at member institutions beyond the official course review processes and professional development offerings, it is known that member institutions around the country have been implementing selected tools and processes in a myriad of ways. QM has numerous QA tools and processes, ranging from a single faculty member informally using the QM Standards to design an online course, to formal certification of faculty as QM Peer Reviewers or certification of a single online course as meeting QM Standards, to certification of an entire degree program, to certification of teaching support, learner support, and learner success of online programs.

Methodology

The purpose of this phenomenological qualitative research study was to explore the validity and applicability of the CEQA model in higher education institutions that are engaging in implementation of QA processes for their online learning innovations. QM provided the population from which the sample pool was identified. The sample pool consisted of a dozen higher education faculty, staff, and administrators, representing a variety of institutional types, all of whom had extensive experience in online education and, additionally, had experience successfully implementing quality assurance tools and processes.

Yin (2009) pointed out that "how" and "why" questions are better answered through qualitative case study, which, in this instance, was done through interviewing the sample of QM implementation experts. Structured interview questions were developed by QM Senior Research Colleagues, a group of higher education faculty from QM institutions with expertise in online instruction and research. Interview questions asked about implementation strategies and efforts, and the interviews lasted approximately 30 minutes each.

Interviews

Interviews were conducted by the same interviewer, via Zoom, and were recorded with participant’s permission. All interviewees received the same questions, which were open-ended and included several prompts along the way. The interview questions were informed by the five levels of the CEQA and were asked in such a way as to encourage interviewees to describe what their institutional experiences had been and what their future vision of QA implementation was. The interviewer did not reveal the CEQA model until the end of the interview so as not to influence the responses and avoided steering the discussion with her own comments.

The interview script began,

To ground our conversation, QM is seeking to validate and refine a model of QM/QA implementation that our Director has created based on her observations over the years of QM’s existence. I don’t want to influence your comments today, so I’ll share the model with you at the end of our conversation. Basically, it is a continuum of implementation with five different levels that go from low to high.

The interview guide consisted of the following questions, and began with the: “Think back to when your institution was first introducing QM and describe what was done.”

  1. If you were to introduce QM at an institution, what would you do and how would you do it? What might you do differently from how your institution did it?
  2. To take the next step and be a little more systematic in your use of QM, describe what you would do and how you would do it.
  3. Would you use data to make improvements, and what would those be?
  4. Do you have or do you envision any activities that are examples of comparing organizational units or your institution to external peer group standards?
  5. How would you describe a comprehensive, institution-wide use of QM and quality assurance as a long-term strategy?

After initial interview questions were answered, the interviewer shared her screen to show interviewees the CEQA model and provided a brief explanation of the model. Reactions and comments from the interviewees were captured.

Analysis

Interviews were professionally transcribed, and a preliminary analysis by the interviewer was completed in October 2018. A deeper, qualitative analysis using Atlas, an established qualitative software program was completed during 2019. Using a constant comparative method (Boeije, 2002), this analysis confirmed thematic findings from the preliminary analysis and provided rich quality assurance and change management implementation examples.

Findings

While the themes that emerged from the interviews were closely related to implementation of QM tools and processes, these themes reveal points that can be mapped to a broader QA system.

Theme One: Early Implementation of a QA System

One of the critical factors necessary to implement a QA system within a higher education organization was the presence and active support of a “champion” at a high level in the institution. Support from this person, most often noted as a provost or vice president, lent credibility to the process and helped elicit faculty support. At the same time, this upper-level support was necessary, but not sufficient. In the majority of institutions included in the sample, there was initially widespread faculty resistance to implementation of the QA system. Faculty, who are used to working as independent agents and coin this independence as their “academic freedom,” were resistant to what they perceived as efforts to control or interfere in their teaching in some way. Accrediting agencies’ positive views of the use of a QA system was one factor identified by several interview respondents as a motivating factor for QA implementation, along with the potential for quality online education to be a marketing or differentiating strategy for the university.

Theme Two: Factors that Impact Moving the Implementation to Second Stages

Faculty “early adopters” were critical factors at many institutions in moving beyond an initial stage of implementation by promoting QM/QA faculty and instructional design staff training and use of the QA standards to design online courses. There was no particular profile of these early adopters identified, except to say they were often risk takers and envisioned early on the potential of online learning. Adoption of a QA system often involved gaining acceptance by the faculty governance system and/or the faculty union, if applicable, and this process was also positively influenced by the early adopters. Additionally, faculty early adopters also helped gain approval for QA efforts by comparing their experience to that of faculty at higher education institutions that were further along in their implementation.

While key administrators as champions were examples of “top-down” implementation strategies, there is data to support that “grass roots” initiatives were equally important. For example, instructional design staff, working in teams with faculty, were promoters of the QA processes and brought faculty along in growing coalitions. In addition, the existence of a statewide system approach to implementation also assisted in moving the process along. Those institutions that were members of a statewide consortium for QA or part of a higher education system that had adopted the QA system, quickly gained traction to train faculty, conduct reviews, and promote QA principles in larger numbers of classes and programs.

Theme Three: Factors Impacting Moving to a Middle Stage

Having systems and processes in place was identified as a way to enable institutions to move to a critical mass of QA adoption. For example, the use of syllabus templates that incorporated QA principles was mentioned by a number of institutions, as was faculty development and training. Faculty incentive systems, such as gaining recognition in promotion and tenure reviews if a course was QA certified, was another factor mentioned as high impact. Research and data collection showing that QA-certified courses promoted improved student success (examples of factors measured included higher GPAs or retention) were an additional factor in some institutions.

However, while faculty resistance did subside enough at some institutions to allow for QA implementation, external review of courses remained problematic for some institutions. In short, faculty may have been amenable to quality reviews from institutional peers, but had hesitancy to engage in quality reviews from peers external to their organization. This reluctance to engage in external validations of quality served as a barrier to further implementation. The majority of institutions included in the sample performed internal reviews until a fairly advanced stage, in fact, as part of a measured response to faculty resistance to external quality reviews.

Theme Four: Movement Back and Forth on Implementation

Several of the institutional online QA leaders mentioned that there was inconsistent implementation of QA processes and that implementation did not follow a linear process. Often, one college or program would make great progress while another lagged behind. Similarly, within the state systems, one institution would make great progress, while others would lag behind. Given fairly high turnover in some academic institutions, a major problem identified by several interview respondents was the loss of a champion, which would set the institution back many steps in the process. On the other side, when that champion moved to another institution, they often began a QM/QA implementation process at the new institution.

Theme Five: No One Has Achieved Stage 5 Implementation

For all of the institutions represented in the sample, QA is still being designed, implemented, and developed. Many seemed to identify themselves as being at stage 2 or 3, with a minority starting on stage 4. However, none of the academic institutions identified themselves as at stage 5, even though several were as many as 8-10 years in to their implementation cycles.

Related to viewing the CEQA model, interviewees noted that while the model made sense to them, their experience was one of movement back and forth between different levels on the continuum and that some parts of their institutions were at higher levels than others. For example, a given institution might advance fairly rapidly to establish and use quality standards but might lag in the faculty and staff training needed to implement those standards. Similarly, the institution might adopt design processes that embed QA standards, only to have faculty governance systems prohibit evaluation of those quality design efforts. In addition, they reported that the departure of a QM champion or key administrator had the largest impact on slipping from a higher level on the continuum to a lower one.

Discussion

Returning to the premise of this study, the collection of systematic implementation data to ascertain whether change management theories inform how QA processes play out in higher education, the themes above support critical theoretical applications. A sense of urgency s consistently mentioned by the change management theorists (Kotter, 1996; Nadler & Tushman, 1998) as a condition of change starting to take root. Implementation of the model for many of the institutions in this study paralleled the growth curve in online education. Further, the extent to which any institution was adopting online delivery was often dependent on student demand or far-sighted leaders who realized that online enrollment remained on the rise, even as face-to-face enrollments continued to decline. The majority of institutions noted a tenuous start, with only a fraction of faculty being willing to be early adopters – most of whom were visionary faculty. Additionally, many faculty were new to online delivery and were skeptical that online classes could match the quality of face-to-face classes, an attitude that often impacted faculty involvement in development and training. Conversely, several interviewees whose institutions promoted research on comparative results noted that collecting and using appropriate data often helped to spur implementation.

Kotter cited coalition building as an important next step in change implementation. In this study, grassroots teams of faculty and instructional design staff were one set of coalitions. Institutions where there was a state system and a decision was made at the state level to adopt QM was another form of coalition building. This made a huge difference in implementation, as training and sharing of “success” stories took place across system schools.

Further steps in Kotter’s model of change management are setting a vision, communicating it, and creating and celebrating short-term wins. There was very little discussion apparent in the interview transcripts of establishing a vision. However, a key organizational leader serving as champion and communicating support for QA/QM was often mentioned. Regarding the celebration of short-term wins, the data from this study support that recognizing and celebrating milestones and “wins,” such as improved student performance or faculty buy-in via training attendance, was an additional aid to furthering QA implementation. In essence, once these wins were communicated, others were more willing to adopt the institution’s Quality Assurance approach. Kotter’s final stages in the model are: producing more change based on larger sets of improvements and institutionalizing the new approach. Systems like creating syllabus templates and collecting student improvement data certainly worked as key drivers to promote larger scale adoption of QA/QM. Several higher education institutions in the study did seem to reach a level of implementation that, while it did not reach across the entire institution, were quite impressive across departments or colleges. Kotter’s model of change is inherently a trajectory, however, and this was not fully supported by the study’s data, as many indicated “fits” and “starts,” with a more circular implementation. In later work analyzing his change model, Kotter (2006), describes why change efforts often fail. Among the factors he identifies is not creating enough of a sense of urgency and/or not communicating enough about the new vision. None of the organizations in this study seems to have “failed”; yet some have not finished the cycle of institutionalization or have regressed. Determining whether any of Kotter’s “failure” factors are at work would need further research in these organizations.

Moving to Nadler and Tushman’s (1998) model, organizational change efforts are first categorized as reactionary or anticipatory. The data supports both, as some institutions did create a sense of urgency in Kotter’s terms or reactionary in Nadler and Tushman’s terms. About half, however, engaged in this work from an anticipatory perspective, noting the higher education trend towards online learning. Going further, Nadler and Tushman categorize change efforts as incremental or strategic, thereby, creating four categories of change: (1)tuning– anticipatory, incremental change; (2)reorientation– anticipatory, strategic change; (3)adaptation– reactive, incremental change (this is frame bending); and (4re-creation – reactive, strategic change (this is frame breaking). The four types of change as they are identified here range in the level of intensity needed to accomplish the change in the organization, with re-creation being the highest.

A number of the elements Nadler and Tushman identify as necessary for frame-breaking change were evident in this study. A magic leader is necessary for the high-intensity changes, as was shown by the champions identified in this study. “Many bullets” or levers for change are needed, with training, new templates, and changes in reward systems pieces identified in this study. These bullets help stage a multi-faceted approach to overcoming resistance to change; the data in this study supported the need for multiple approaches to overcoming the faculty resistors. Institutional commitment over the long term, including commitment of senior management time and resources, is less clear from the data in this study. What is also less clear when applying the Nadler and Tushman model is the extent to which implementation of QA/QM in each institution was “reorientation,” a high level of anticipatory change. If it is, then less radical efforts might bring about the needed changes without major organizational disruption. The steps in the Continuum of Excellence would potentially still apply but progression through the stages could be easier. None of the interviewees who participated in this study described the process of QM/QA implementation as being easy.

The theories of organizational learning introduced at the beginning of this article document organizations who are able to accomplish rapid adaptations because of the ability of individuals within the organization to engage in swift, anticipatory, and creative problem solving, often called double-loop learning. There is no doubt that such an ability to move forward quickly is an admirable quality in any organization and one that higher educational institutions should aspire to. While some individuals at the universities seem to have demonstrated double-loop learning capabilities (e.g., the early adopters), the data in this study do not support this capability at the institutional level.

As the interviews indicated, the CEQA model would be useful if considered as a circular model of continuous improvement within and between levels, rather than a unidirectional continuum. To begin, institutions might consider how the QA of online initiatives is being supported and evaluated prior to introducing a new set of standards and processes. Once the implementation is in place, future plans can be set based on the data and knowledge generated from the implementation. With this strategy, even the Ad Hoc level of the model would be more robust and successful because the quality assurance activities will be structured in a way that is consistent with the organization’s support of online education.

Significance

Online education is an ever-larger share of higher education (Ginder, Kelly-Reid, & Mann, 2018; Seaman, Allen, & Seaman, 2018) and, thus, has drawn interest as an institutional strategy to increase enrollments and access. However, the level of quality of online education has been critiqued since online education has been in existence, making quality assurance goals and processes paramount to a successful online learning strategy. In short, systematic quality assurance is the way to raise the level of quality online learning and promote continuous improvement. The finding that this model can serve as the basis of continuous improvement in an institution’s quality assurance can help institutions map and reach their strategic goals.

Next Steps and Recommendations for Future Research

The model should incorporate circular movement within and between levels on the continuum. Future research on the use of this model would provide further validation, as well as more institutional data that would make it a more nuanced and useful model. Additionally, unique, impactful events, such as 2020’s pandemic crisis, which prompted a rapid, full adoption of online learning in most higher education institutions, can be a potential rich data set of reactionary-driven change and warrants further study.

Summary

One of the largest notable takeaways of this study is that simply having access to standards and tools does not measure or ensure implementation of quality practices. Implementation of innovations for online learning will, in fact, be most successful and play a part in organizational learning when included in an institution’s quality assurance and continuous improvement efforts. The CEQA model was developed as a tool that institutions can use to identify, assess, and strategically embed institution-wide processes and actions for sustainable quality online learning for all stakeholders. This article describes a first-tier study of the usability of the CEQA model. Twelve educators with histories of implementation of Quality Matters tools and processes as part of their institutional approach to assure quality online learning were interviewed to gather information about implementation efforts and to map to the model. The findings, which enhance understanding of how change management theories inform the model’s implementation, can serve as the basis of continuous improvement in an institution’s quality assurance, and can also help institutions map and reach their strategic quality goals.


References

Adair, D., & Shattuck, K. (2019). Ensuring quality while creating and innovating. In K. E. Linder (Ed.), The business of innovating online: practical tips and advice from industry leaders (pp. 97-112). Stylus.

Argyris, C. (1976). Single-loop and double-loop methods in research on decision making. Administrative Science Quarterly, 21(3), 363-375. doi:10.2307/2391848. JSTOR 2391848.

Boeije, H. R. (2002). A purposeful approach to the constant comparative method in analysis of qualitative interviews. Quality and Quantity. Retrieved from https://www.researchgate.net/publication/226282093

Chaloux, B., & Miller, G. (2014). The impact of organizational context. In G. Miller, M. Benke, B. Chaloux, L. Ragan, R. Schroeder, W. Smutz., & K. Swan (Eds.). Leading the e-learning transformation of higher education: Meeting the challenges of technology and distance education (pp. 23–37). Stylus.

Christensen, C. M. (2018). Disruptive innovation. https://www.christenseninstitute.org/disruptive-innovations/

Garvin, D. A. (1993). Building a learning organization. Harvard Business Review, 71(4), 78–91. https://hbr.org/1993/07/building-a-learning-organization

Garvin, D. A., Edmondson, A. C., & Gino, F. (2008, March). Is yours a learning organization? Harvard Business Review. https://hbr.org/2008/03/is-yours-a-learning-organization

Ginder, S. A., Kelly-Reid, J. E., and Mann, F. B. (2018). Enrollment and Employees in Postsecondary Institutions, Fall 2017; and Financial Statistics and Academic Libraries, Fiscal Year 2017: First Look (Provisional Data) (NCES 2019-021rev). U.S. Department of Education. Washington, DC: National

Center for Education Statistics. https://nces.ed.gov/pubs2019/2019021REV.pdf

Kotter, J. P. (1996). Leading change. Boston, MA: Harvard Business School Press.

Kotter, J. P. (2006). Leading change: Why transformation efforts fail. In J. Gallos (Ed.), Organizational Development Reader (pp. 239-251). Jossey-Bass,.

Nadler, D. A. & Tushman, M. L. (1989). Organizational “frame bending” principles for managing reorientation. The Academy of Management Executive, 3(3), 194-204.

Paulk, M.C., Curtis, B., Chrissis, M.B., & Weber, C.V. (1993). Capability Maturity Model (Version 1.1) (CMU/SEI-93-TR-024). Software Engineering Institute, Carnegie Mellon University. http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=11955

Seaman, J. E., Allen, I. E., & Seaman, J. (2018). Grade increase: Tracking distance education in the United States. http://onlinelearningsurvey.com/reports/gradeincrease.pdf

Shattuck, K. (2007). Quality Matters: Collaborative program planning at a state level. Online Journal of Distance Learning Administration, 10(3). http://www.westga.edu/~distance/ojdla/fall103/shattuck103.htm.

Yeh, K. B., Adams, M. L., Marshall, E. S., Dasgupta, D., Zhunushov, A., Richards, A., & Hay, J. (2017). Applying a Capability Maturity Model (CMM) to evaluate global health security-related research programmes in under-resourced areas. Global Security: Health, Science and Policy, 2(1). https://www.tandfonline.com/doi/full/10.1080/23779497.2017.1279022?src=recsys

Yin, R. K. (2009). Case study research: design and methods (4th ed.) https://www.slideshare.net/nor...