1. Misalignment of service provision and student need is persistent.
2. Reliance on demographic data can lead to erroneous identification of “at-risk” students.
3. High-scoring students often miss out on advanced courses due to misidentification.
4. The majority of dropout prevention program participants may not fit the “at-risk” profile.
5. Resistance to data-based student identification can hinder progress.
6. Legislation alone cannot correct the misalignment in education.
7. Effective use of data is critical for ensuring equity and diversity in education.
Over the course of three decades, Edstar Analytics has analyzed various educational programs, finding a troubling, unchanging pattern. The problem centers on a persisting confusion surrounding data understanding among educators. This confusion, which first appeared when the No Child Left Behind Act (NCLB) introduced the concept of using objective data and outcome measures instead of aligning services by demographics, is still a significant issue. Our findings are echoed by school systems evaluating their own programs.
During NCLB, supplemental educational services program were designed for students receiving free or reduced-price lunch, with the goal of scoring at grade level proficiency. The programs we evaluated (in about 10 school districts or more) served students already performing at or above grade level proficiency using remedial curricular materials. This led to an unexpected outcome where proficient students, after service, scored below grade level. Control group students not receiving services did not.
In our evaluations of different educational programs, we noticed this pattern repeatedly. For instance, dropout prevention programs, intended to help “at-risk” students, were in fact serving students who were already performing well academically. The majority of these students had never been suspended, and they were consistently scoring at or above grade level in math and reading. The process for identifying these students was primarily based on demographic information rather than objective academic performance data. To counteract this, we initiated data academies to train educators to better use data for aligning services and opportunities. We’ve managed to identify high-achieving students who could benefit from advanced classes, and we have seen a high success rate when these students are given the chance. However, the approach has been met with resistance, especially from departments serving gifted and talented students, as these advanced classes are often reserved exclusively for students identified as “gifted.” Despite legislative measures in various states that mandate advanced course access for high-scoring students, we continue to see resistance and misunderstanding. Many institutions argue that these high-scoring students are not “gifted” hence they cannot be admitted into these advanced classes. Unfortunately, this mindset persists even in 2023, resulting in significantly misaligned services and a lack of opportunities in many programs and institutions. The current focus on equity and diversity training is commendable, but without data accountability or review mechanisms to ensure high-scoring students access enriched courses and low-scoring students receive needed support, the change will be limited.
In the data academies we initiated, a significant part of our program was challenging educators’ assumptions about student achievement and access to advanced educational opportunities. One of the exercises we frequently conducted was to ask participants to estimate the percentage of black male students scoring at the highest level on standardized math tests who were NOT enrolled in advanced math classes.
Educators would often predict zero or near-zero percentages. They believed that because of ongoing efforts to increase STEM access and close achievement gaps, any high-scoring black male students would surely already be in advanced math classes. However, the data told a different story. Participants were invariably taken aback when we revealed the actual numbers, demonstrating that a significant number of black male students, despite scoring at the highest level in math, were still placed in standard or even remedial math classes. This pattern underscores the disconnect between intentions and outcomes in educational programs. It also emphasizes the importance of rigorous data analysis to identify and address such disparities, aligning educational services with student needs and capabilities, rather than assumptions or preconceived notions.
During the implementation of our Data Academies, we discovered a common misconception among educators – many assumed that demographic characteristics could serve as a reliable stand-in for academic or behavioral data. For example, when asked how they identified students who scored below grade level, many would cite characteristics like the bus a student rode or the quality of their parent’s possessions – factors that have little to no direct correlation with academic proficiency. To address this, we developed specific lessons aimed at helping educators differentiate between various types of data: academic, demographic, and policy-related. We conducted activities where participants would categorize different types of data and utilized symbolic icons to reinforce the lessons.
We used apples to represent mutable data points, things that can be altered by educational intervention, such as reading scores. Oranges were used to symbolize immutable factors, like race or socioeconomic status, which cannot be directly influenced by educators. Lastly, mangos represented policy-related data, reflecting the effect of institutional decisions, such as using subjective judgement rather than academic data for math placement. Understanding the difference between these types of data was not always an immediate realization for many educators, requiring multiple reinforcing activities. However, grasping this distinction is crucial – particularly when creating SMART objectives. It’s important to remember that we cannot, and should not, aim to change immutable characteristics like race; instead, our focus should be on changeable academic and behavioral aspects that can positively influence student outcomes.