Skip Navigation

Distress Signals from Canaries in the Data Mine

Posted on Feb 16, 2015

Center on Online Learning and Students with Disabilities Summit Series

During 2016 the Center will hold a summit related to the benefits and challenges of online learning for students with disabilities. To prepare for that meeting, The Center’s Blog and other venues will publish articles and position papers designed to frame the Summit agenda.

Canaries in the Data Mine: Who are they, where are they, and what are they doing there?

Bar graph

Level of agreement by state special education directors with the statement, Most students with disabilities who enroll in an online course complete it, as measured by total number of responses. From Perspectives from State Special Education Directors on Online Learning, 2013

When we ask education administrators  “How many students with disabilities are enrolled in online learning, which of these students do best in which types of environments and how are they progressing?” we are met with blank stares. They simply do not know. They may be able to tell us how many of these students were enrolled at the start of a semester, and whether the academic outcomes for these students were positive or negative.  Are these student still engaged in online learning? Do they complete and online course of study once enrolled?

Beyond enrollment and persistence, determining which factors actually promote learning – pathways, media, supports, activities, technologies, interpersonal connections (virtual or face to face) – is a more significant challenge.
The education personnel charged with reporting on the progress of students with disabilities really do not know.  They don’t know because they cannot get that information, this despite the fact that the patchwork of entities holding the data sets within which that information exists are unable or unwilling to share it, or unaware that the data they hold, when combined with the data held by others,  could benefit them and every other stakeholder in the education system.

A lot of data in; a trickle out

During the past three years the Center has been exploring the experience of K-12 students with disabilities in online learning: those involved in full-time virtual schooling, those taking supplemental online coursework, and, predominantly, those involved in Blended Learning environments. Our guiding focus has been to explore the intersection of student, context and system design: what strengths, challenges and needs do these students present, what are the factors affecting their involvement and progress (home, school, instruction, technology, etc.) and what aspects of the online system: its ease-of-use, accessibility and progress monitoring impact educational outcomes? In each of our inquiries, involving education personnel, vendors of online delivery systems and content, students and parents, one consistent factor has emerged: every stakeholder will acknowledge that while extensive data on these students are being recorded, it is providing little actionable information for anyone.  Despite this fact, a 2013 Fordham Center on Law and Information Policy study found that 95% of the nation’s school districts use cloud services for student data tracking and storage.  Online learning can provide real-time information on who these students are, what they are doing, where they are doing it (and how they got there, how long they stayed and where they went), and both formative and summative details of their academic achievement. Unfortunately, each of these data sets is maintained and controlled by separate entities, and various constraints: privacy concerns (110 bills addressing student data privacy were proposed in 2014), intellectual property anxieties, technical incongruities and ignorance of potential benefits, has significantly limited their real world usefulness.

Benchmarking Progress

For nearly 40 years tracking the achievement of students with disabilities has been the most personalized and detailed of any student population. Educational progress reporting for these students is mandated by both education and civil rights laws. As a result, this information has provided unique insights into the impact of curriculum or school reform efforts on the progress of these students. Tracking the progress of students with disabilities was a key part of the No Child Left Behind Act of 2001 (The Elementary and Secondary Education Act).

Since 2001 the primary measures of academic progress have been large-scale tests administered well beyond the point of instruction. Digital learning systems have altered the assessment possibilities by offering nearly immediate feedback of student progress. Today it’s common for students, (and teachers and parents) to have ready access to daily or weekly indicators of their academic trajectories. This timely access to information has expanded the existence of data-dependent competency-based education systems designed to individualize instruction for all students: effectively equalizing the individualized approach previously only available to students with disabilities.

So, As Dr. Phil says, “How’s that workin’ for ya?”

In full-time virtual, blended and supplemental instruction regular access to student achievement data appears to be very useful, especially in competency-based systems where students are working to achieve proficiency along a sequenced series of measurable academic levels. An initial disconnect often appears at the school or district level (in both competency and traditional settings) where daily or weekly student progress indicators are minimized or ignored in favor of quarterly, half-year or annual summative performance assessments. A second level of disconnection occurs when student growth is assessed by annual state-level achievement tests. In the best of circumstances the instruction, regular progress monitoring, local and statewide summative measures are all aligned, drawn from the same source or created by the same developers; this is an unrealized fantasy.

Are the statutes and regulations, and the reporting practices designed to address analog challenges simply off pace with the digital reality? Partially.  There’s also the privacy panics, intellectual property anxieties, technical incongruities and ignorance of potential benefits factors mentioned earlier, and these play no small role.

Water, water, everywhere, Nor any drop to drink.

Coleridge’s ancient mariner laments the fact that adrift on the ocean he and his shipmates are surrounded by water and cannot drink any of it, and so may perish in the very medium, that only for a saline adjustment, might guarantee their survival.  Educators, in general, and special education, specifically, are in a similar circumstance with respect to coordinating and taking advantage of the sea of data that exists. The diagram below (Exhibit A) depicts the relationships that exist between a school district or school and their provider of online instruction. Unfortunately, it is not a hypothetical scenario, but represents an actual system.

The yellow boxes represent 3rd party subcontracts or subsystems employed by the provider to manage aspects of online content delivery or online functionality. Nearly every K- 16 student in the country has an anonymized unique student identifier (a requirement of the US Department of Education’s Statewide Longitudinal Data System eligibility for funding). In most cases, this UID number is used by the school, the online provider, and by nearly all of the related subsystems and resources to track and record student pathways, activities and outcomes in the online environment, with one exception: the majority of Open Educational Resources or OERs, are educationally inert with respect to data collection, however, raising an entirely different complication).

While Exhibit A is striking for a host of reasons, some detail is more notable than others:  IEP information follows the registration of students with disabilities into the online environment, (sometimes); the information returned from the online environment to the district or school primarily consists of summative grades. The truly valuable information related to, for example, which resources a student accessed, for how long and using which supports (text-to-speech data tracking is often only provided at the course level, with student privacy issues cited as the rationale), is typically not shared for educational purposes.  This data will be used on an aggregated basis by the online provider and its sub-vendors to track usage patterns and associated outcomes, but it’s precisely the disaggregated data patterns – which students with which types of strengths and weaknesses do best with which materials under which circumstances – that would benefit both vendors and consumers alike, including the very students who are generating the data in the first place.

Flow chart of school and online data systems

A complex diagram depicting the interrelationships that exist between a school, and an online provider, and the subcontracts & subentities with whom the provider has relationships. Each of the entities depicted has access to student data relative to learning, but this data is unable to be shared for a variety of reasons.

While having access to disaggregated student data patterns that can be associated with greater than or less than anticipated academic achievement would be exceptionally useful for all students; it is essential, and mandated, for students with disabilities. The present system of isolated and unshared data sets does not currently provide sufficient information to meet the IEP-related expectation for a student with a disability; despite the fact that all the necessary data required to do so is available.

In a subsequent post we’ll explore approaches to providing students, parents, educators, vendors and other contributors to K-12 online learning with data that can be profoundly helpful in achieving meaningful learning outcomes. Any ideas?

    Leave a Reply

    Your email address will not be published. Required fields are marked *