To Measure What Tests Can't, Some Schools Turn To Surveys

Last year Susan Avey, the principal of Bogle Junior High School in Chandler, Ariz., had a heart-to-heart with one of her new teachers about how he was relating to students.

In a previous year, this might have been a conversation based on subjective impressions. The teacher might have gotten defensive. But this year, Avey had a new tool up her sleeve: a survey of her students.

"He came in to talk to me and said, 'I felt like I had really good relationships with kids, but reading my comments I was surprised that I wasn't rated as highly.' "

Drilling down into the results, they found that the teacher's relationships with girls, specifically, were weaker than those with the boys. The teacher was also a coach, and it turned out he liked to use a lot of sports analogies in class. Maybe that habit was missing the mark with non-sports enthusiasts?

This little data-driven a-ha moment is happening more often at schools around the country.

A growing battery of school leaders, researchers and policymakers think surveys are the best tool available right now to measure important social and emotional goals for schools and students — qualities like grit, growth mindset, student engagement or, as in the Chandler example, student-teacher relationships.

And a group of big-city districts in California, with the federal government's permission, is getting ready to incorporate this kind of survey data into their accountability systems this spring.

That means students' states of mind will be weighted alongside test scores and graduation rates when rating schools. It's a move other states will be watching, as the draft version of the new federal education bill includes language asking states to incorporate at least one measure that is not strictly academic in school accountability systems.

The CORE Movement

Nine of the biggest school districts in California — Fresno, Garden Grove, Long Beach, Los Angeles, Oakland, Sacramento City, San Francisco, Sanger and Santa Ana Unified — banded together to receive their own separate waiver from the requirements of the No Child Left Behind law in 2013.

A waiver is like a permission slip to create your own formula for improving schools; the Department of Education has granted 43 to various states in the past few years to provide flexibility from federal law. This is the only waiver that has gone to a group of districts instead of a whole state.

It's also the only waiver in the country to focus this much on social and emotional targets.

The CORE districts, as they were known, drew up a School Quality Improvement System that relied 60 percent on traditional academics, and 40 percent on "social-emotional and culture-climate" factors. This 40 percent includes indicators the schools were already collecting: on chronic absenteeism, suspension and expulsion rates. But the districts also wanted to know how well schools were faring in building nontraditional, noncognitive skills.

We've written before about how researchers are becoming increasingly convinced that students' attitudes about learning, their ability to control themselves and persevere and their proficiency in working well with others account for more than half the picture of their long-term success.

Enter Chris Gabrieli, a wealthy Boston businessman passionate about these factors.

Gabrieli is the co-founder of Transforming Education, a nonprofit working with the CORE districts to design their social-emotional measures. They created a framework with four attributes: Growth mindset, self-efficacy, self-management and social awareness.

• Growth mindset refers to the belief that working hard and persisting will help you improve at anything, and that abilities are not fixed at birth.
• Self-efficacy refers to a sense of confidence that you are capable of achieving what you set out to do.
• Self-management refers to productive habits such as organization and delaying gratification.
• And social awareness is defined as the ability to work in a diverse group, to recognize sources of support and to have empathy for others.

For the past two years, TransformED field-tested a survey designed to allow students to rate themselves on these skills. This past spring 450,000 CORE district students took it.

And this week, TransformED announced that it is going to be releasing that survey and those results. That means any school in the country can start giving the survey this spring and can then benchmark the results against a similar population from one of the CORE districts — whether that be English Language Learners in middle school, or low-income girls in urban schools.

Gabrieli has pulled in an impressive roster of advisors, including Penn's Angela Duckworth (best known for her work on grit), Stanford's Carol Dweck (growth mindset) and Linda Darling-Hammond (assessment reformer), and Harvard's Martin West. West has done research for TransformED showing that the student surveys, as short as 20 minutes, combined with teacher reports, can meaningfully predict outcomes such as GPA, test scores, attendance and suspensions.

"We're in the early innings," Gabrieli acknowledges, whether you look at how to identify these skills, how to measure them, or most importantly, how to cultivate them.

But, "I would argue that until the education system begins to value these attributes, they won't focus on them. That's why we started with measurements."

The Panorama View

Many of the CORE districts have partnered with a for-profit startup, Panorama Education, to administer and analyze these surveys, both online and on paper.

The Chandler district southeast of Phoenix, where Avey is a principal, is one of Panorama's 220 district clients, which are located in 40 states serving 3 million students.

Aaron Feuer founded Boston-based Panorama. He got interested in the change-making power of surveys while an undergraduate at Yale.

"Surveys have been around forever, but they're not usually rigorous or getting good data," he says.

Panorama's tools enable schools to ask questions about whatever they find important. But the company also offers guidance on which types of questions are supported by the best research. It provides access to questions that other schools have asked — and information about what they did to improve on the results. "Our focus is using data to help schools improve," Feuer says.

Many school leaders, even outside the CORE districts, already believe these non-academic qualities are important, and they are looking for ways to track their progress and get credit for their efforts.

"We're a data-driven school," says Avey. "I tell my teachers the academic piece, the standardized test results, those are one little piece of this big puzzle for every kid."

TransformED's own research shows schools are already spending about $650 million per year on curricula and professional development designed to address social and emotional skills directly. And the organization says that teachers estimate they spend about 10 percent of their total time emphasizing these skills, a cost of another $30 billion per year that puts social and emotional learning not too far behind math and reading.

Schools need data to understand which of these efforts are working.

That's what educators at Bogle Junior High learned from their survey.

"The schoolwide theme the last three years has had to do with grit," says Avey. "But 31 percent of our kids did not respond favorably to the questions under that domain." Based on the results, Avey and her staff decided to retool their "grit" curriculum and explicitly teach more specific skills, like goalsetting and decision making.

And if that doesn't work? Avey says they'll look at the survey results and try something else.