For decades, test developers have attempted to create a generalized electronic scoring system for texts. It's even been called the "holy grail" of educational assessment, because it would free us from the tyranny of multiple choice items. Products developed exclusively with data analytics, like IBM's Merlin, are making some inroads, but they require huge amounts of text to produce results, and aren't yet reliable enough enough for use in individual assessment. And like the electronic scoring systems now being produced by educational test developers, they measure easily quantified aspects of texts (like vocabulary and punctuation) rather than what we really want to know—how deeply students are learning the skills and knowledge they need for success in life and work.
CLAS is the first computerized developmental scoring system that measures what educators really want to know—how well people can put their knowledge to work in real-world contexts—reliably enough to be used ethically in individual assessment contexts. It's a truly general system that can readily be adapted for use in diverse knowledge domains. Rather than measuring whatever aspects of texts are most easily quantified, CLAS measures growth along a well-validated developmental scale called the Lectical Scale. It's algorithms are based on a deep study of learning and development that involves a blend of human expertise and analytics. When CLAS produces a score, it tells us where an assessment performance lands on the Lectical Scale, what the score means in terms of a test-taker's mastery of the skills and concepts targeted by the assessment, and what the test taker is likely to benefit from learning next.
CLAS algorithms are based on human and computer analysis of tens of thousands of texts (and growing). At its core is the Lectical™ Dictionary—the world's first developmental dictionary—a constantly evolving taxonomy of the development of meanings. (More about the Dictionary below.)
Like the LAS (Lectical Assessment System, our human scoring system), CLAS can be used:
CLAS has several advantages over the LAS:
Since its introduction in 2014, CLAS has been used to score thousands of assessments for several research and evaluation projects.
The Lectical Dictionary is at the core of everything we do here at Lectica. It not only makes CLAS possible, but we use it to describe learning sequences and develop customized learning resources and other report feedback. We've also used it to create a better spell checker—one that takes the developmental level of a performance into account when it makes spelling suggestions.
The Dictionary is composed of units of meaning called "Lectical Items"—words or phrases like "evidence" and "reliable evidence" that carry meaning. Each Lectical Item is assigned to a Lectical Phase (1/4 of a Lectical Level), based on a combination of empirical evidence, the judgment of our analysts, and a variety of helper algorithms. The goal is to assign items to the lowest level at which the simplest meaning they carry is likely to be useful. Dictionary entries begin with first speech and cover the full span of development. As of August, 2019, the Dictionary contained about 530,000 curated items, which CLAS uses to score 6 different assessments.
The Dictionary is continuously monitored and improved by a team of trained analysts. Over time, as we build new assessments in new subject areas, the Dictionary will become an increasingly comprehensive taxonomy of learning.
Unlike the purely computational scoring algorithms of conventional "big data" analysis or "meta-analytics," CLAS's algorithms are developed through a human/machine collaboration. CLAS and our analysts are constantly engaged in "conversations" about the placement of Lectical Items. CLAS's algorithms represent our current knowledge, and our analysts interact with CLAS to integrate new knowledge. This partnership approach required a great deal of human labor and learning upfront, but the resulting system produces scores that are more accurate and much more meaningful than those produced with analytics alone.
(We call the process of building the Dictionary Lexicating. It's so much fun—for people who love puzzles—that we're thinking about building a game around it. Think of it as a kind of meaningful Sudoku!)
CLAS is not only making it possible to fulfill our educational mission—to optimize learning for everyone—but its technology will be useful in a wider range of contexts, including marketing, software, and psychological and sociological research. Over the next few years, we'll be exploring a range of applications.
If you'd like to know more about CLAS or have ideas about how CLAS technology could enhance your work, please contact us.