John N. Gardner Institute for Excellence in Undergraduate Education

The JNGI Analytics Process Collaborative:
The Conceptual Framework and Components

Updated 02/02/16

Andrew K. Koch, Executive Vice President and Chief Academic Leadership & Innovation Officer, John N. Gardner Institute for Excellence in Undergraduate Education

Matthew D. Pistilli, Director, Assessment and Planning for the Division of Student Affairs, Indiana University Purdue University Indianapolis

Robert Rodier, Director of Technology & Informatics, John N. Gardner Institute for Excellence in Undergraduate Education

The Problems – Blurry Definition, High Cost & Limited Application 

The term “analytics” refers to a broad range of statistical techniques and predictive modeling approaches that have garnered great interest across U.S. higher education over the past decade (Campbell, DeBlois & Oblinger, 2007). But, as Hampson notes, “analytics in higher education is relatively new and descriptions are often imprecise. Different types of analytics, with little in common, are regularly lumped together” (2014). Analytics can be used for operational purposes in universities or colleges – such as maximizing classroom and course space usage, or for packaging financial aid to boost student recruitment yield. In fact, van Barneveld et al. list seven separate kinds of analytics, with upwards of two-dozen definitions for the various types. It is no wonder, then, that this term has a great deal of confusion surrounding it (van Barneveld, Campbell & Arnold, 2012).

Analytics tools offered by many for-profit vendors often cost tens, if not hundreds, of thousands of dollars per year to use. But, as a recent EDUCAUSE report makes clear, these expensive tools often are purchased only to “satisfy credentialing or reporting requirements rather than to address strategic questions, and much of the data collected are not used at all” (Bichsel, 2012, p. 3). As a result, at best, a small number of faculty make use of analytics for the benefit of a small subset of students. In addition, the high cost of commercially available analytics tools make them virtually inaccessible to smaller-enrollment institutions—the very institutions that enroll the largest levels of first-generation and low-income students (Rine & Eliason, 2015, p. 9-12). Thus, the college completion promise of analytics is far greater than the actual benefits realized for students on campuses across the United States.

The Solution

The JNGI Analytics Process Collaborative aims to address the fuzzy definitions, high cost and low application issues associated with analytics in higher education by: 1) placing a specific framework around the use of analytics within higher education; 2) providing that framework at an accessible, non-profit fee; and 3) providing support for the adoption and continuous application of analytics.

For starters, we do not believe that analytics is simply a tool, nor a singular “thing” that can simply be applied as a panacea for all challenges facing an institution. Rather, analytics are processes that combine large data sets in an effort to create actionable intelligence – information that can be used to directly and positively affect outcomes associated with students and institutions.

The assertion that analytics is a process is an important one. Many institutions believe that they “have analytics” once they have come upon or purchased a model or tool of some form. In reality, all they actually have is a mathematical formula or product that predicts, models, or represents an outcome – they have a “thing.” They do not have a process or processes to apply that data, to a high level, to effect positive change. Envisioning analytics as a process moves from simply having a model to being able to do something with the information fed into and obtained by an algorithm. In short, our definition of analytics as a process means that analytics is actually about using data to continuously shape and improve institutional and/or faculty actions surrounding teaching, learning and student success. Analytics is a tool that shapes process – not an answer in-and-of itself. To derive the full benefits of analytics, you need to focus on the process of applying the data output.

The Focus

The JNGI Analytics Process Collaborative is firmly focused on a form of analytics called “learning analytics.” As defined by the International Conference for Learning Analytics and Knowledge, learning analytics is, “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (2011).

Specifically, the JNGI Analytics Process Collaborative focused on the application of learning analytics to improve the way faculty teach and students learn in gateway courses. This highly defined focus allows for deep exploration into and work on the process of applying analytics outcomes into actions with gateway courses – thereby maximizing the benefits of analytics. In other words, the focus helps increase the likelihood that faculty and staff will actually do something with analytics-related reports and dashboards to affect positive changes in teaching and learning.

The Components

The JNGI Analytics Process Collaborative includes five key components. These include:

  • Learning Analytics Readiness Instrument
    Through the use of an adapted version of the Learning Analytics Readiness Instrument (LARI; Arnold, Lonn, & Pistilli, 2014), the JNGI’s Analytics Process Collaborative will strive to help campuses identify areas that could be addressed in order to create the most optimal environment possible for the application of an analytics process in gateway courses.
  • Historic Data Analytics (Longitudinal Trend Reporting)
    JNGI Analytics Process Collaborative participants will have access to the Gardner institute’s Gateway Course Success Analytics Inventory – a tool that puts data into the hands of faculty and staff working to transform gateway courses. The historic data shines a bright light on why changes in teaching and learning are necessary and for whom they are most necessary.
  • Predictive Analytics Models & Dashboards
    The JNGI Analytics Process Collaborative includes both predictive analytics models and dashboards that display prediction outcomes. These models predict the probability of success in a gateway course – currently defined as the likelihood of earning a C or better final course grade.Unlike for-profit proprietary approaches that do not reveal their models’ formulas, any participant in the non-profit JNGI Analytics Process Collaborative may see the models and, if they so desire, work with them outside the live system. The models belong to the Collaborative and, as long as an institution is involved in the collaborative, faculty and staff from that institution can see and work with the models. The only caveats are:

    1. any lessons learned about the models should be shared with the Gardner Institute and other members of the Collaborative;
    2. the models cannot be shared with people from institutions or organizations not presently involved in the Collaborative; and,
    3. if institutions leave the Collaborative, they cannot take the models with them or replicate them outside of the system.
  • Intervention and Implementation Process Planning Support
    The process of implementing analytics is not a one-time thing or an overnight activity; rather, it requires persistence, dedication, and energy focused on not only implementing something but also continuously nurturing a process over the course of time. The proper implementation of analytics tools and practices can result in a great many changes for a campus, most notably a positive change in teaching practice, increased use of interventions to drive students to resources related to their academic and personal needs, and a growth in student success, retention, and graduation rates. The JNGI Analytics Process Collaborative has been designed to support change – change in how analytics is viewed and used by faculty, change in pedagogy and practices and change in student performance and persistence.
  • National Survey of Analytics Approaches and Benefits
    Frequently, faculty and staff ask questions about specific analytics tools and/or examples of successful ways in which analytics have led to changes in teaching and learning. To our knowledge, there is no comprehensive survey that collects data to provide answers to these and other analytics-related question. As part of this effort, the JNGI Analytics Process Collaborative will address this gap with a national survey to be distributed in 2017. Findings will be shared broadly with in-depth overview and discussion occurring with institutions involved in the Analytics Process Collaborative.

 

The Benefits of the Approach

Our experience has led us to believe that when it comes to learning analytics, many institutions, on some level, struggle with similar issues, encounter comparable roadblocks, and need to address concerns regarding scale and scope. We believe that there is a real void in the educational space on how to even think about analytics, much less having capacity to build or scale an effort. The JNGI Analytics Process Collaborative seeks to fill this space for those institutions willing to embark on implementing an analytics process in a cooperative, mutually beneficial manner.


References

Arnold, K. E., Lonn, S., & Pistilli, M. D. (2014). An exercise in institutional reflection: The Learning Analytics Readiness Instrument (LARI). In K. E. Arnold, A. Pardo, & S. Teasley (Eds.), Proceedings from the 4th International Conference on Learning Analytics and Knowledge (pp. 163-167). New York: ACM. doi: 10.1145/2567574.2567621

Bichsel, J. (2012). Analytics in higher education: Benefits, barriers, progress, and  recommendations. EDUCAUSE Center for Applied Research.

Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE review, 42(4), 40.

First Annual Conference on Learning Analytics and Knowledge. (2011). Call for proposals. https://tekri.athabascau.ca/analytics/about.

Hampson, K. (April 25, 2014). Analytics in online higher education: Three categories. http://acrobatiq.com/analytics-in-online-higher-education-three-categories/

Rine, J. P. & Eliason, J. (2015). Expanding access and opportunity: How small and mid-sized colleges serve first-generation and low-income students. Washington, DC: The Council of Independent Colleges.

van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a common language. ELI Paper 1. Littleton, CO: EDUCAUSE.

Watson, H. J. (2011). Business analytics insight: Hype or here to stay? Business Intelligence Journal 16(1), 4-8.


Printer friendly version