A school system asked a non-profit that worked with kids in their schools to document the effectiveness of their programs. Edstar was hired to help them do that. The non-profit had several different programs and services for at-risk kids. They told us their goals were to help kids improve in academics and behavior. They hoped to impact dropout and college going rates. Students had been referred to them by school staffs. They had no baseline academic or behavioral data on the kids they served, and although they had paper records kept in binders when students enrolled in one of their programs, in some of the blanks they either recorded the kids’ birthdays, or this blank contained their school IDs, or the name of the building they went to. Because of the way they kept records we didn’t have any way to create a data file that would contain the student information with their ID and programs they were in, which would be needed to get baseline and outcome data and match to services.
The school system gave us a huge file of names and IDs so we could identify the kids manually by searching on their names. This took us weeks. We then were able to create a file that contained the names and IDs of students served. We worked with the staff of the non-profit to create a file of what services each kid got.
We got baseline data for the kids. The academic and behavior data showed that they served the entire spectrum of kids, from very academically successful to struggling, and kids with no attendance or behavior problems to kids with lots of behavior problems. When we asked them which of their services were for supporting kids academically, they did not know. The same was true with services to support attendance and suspensions. We helped them review their services, and found research linking some of them to academic or behavior support for struggling students.
We were not able to evaluate the effectiveness of this program this first year because many of the students served already surpassed their outcome goals prior to service, the services were not aligned to the intended outcomes, and they didn’t keep the records in ways needed.
We helped them identify the data profiles of students they intended to serve. These included high school students who were behind in credits and not likely to graduate on time, students with more than 3 suspensions, attendance problems, and students who scored below grade level in math and reading. The school system was then going to produce rosters of students who fit these profiles, give them to the principals, and students would be referred from these lists. We also created record-keeping systems. We helped them align their services to these intended outcomes and the needs of the students.
The rosters provided by the school district to the principals did not contain kids who fit the intended profiles. They produced rosters, but they identified the wrong kids. They were not able to produce rosters identifying the intended target kids.
Beliefs and Skills of the Program and School Staffs
Cause and Effect: The program staff had been providing services for which there was no evidence of a link to the intended outcomes.
What At-risk Means: School staff were referring students who they thought were at risk. However, they referred many students who were already successful in school and had no behavior problems.
How to Classify Things–Organizing information so it can be used in analyses: The program staff kept student records in ways that made it not possible to create a roster of students served, service they received, and school ID for obtaining baseline data.
Outcome
Outcomes were analyzed for the kids who were in the target populations addressed by the SMART objectives of the new data-driven process. The school system was not able to provide outcome data for 6 of their 16 objectives. Among the 10 outcomes for which outcome data were available, none of the objectives were met. They had previously believed the services were effective from looking only at the outcomes for students who primarily already exceeded their intended outcomes.