The Computerized Lectical Assessment System 

For decades, test developers have attempted to create a generalized electronic scoring system for texts. It's even been called the "holy grail" of educational assessment because it would free us from the tyranny of multiple-choice items. Products developed exclusively with data analytics, like IBM's Merlin, are making some inroads, but they require huge amounts of text to produce results and aren't yet reliable enough for use in individual assessment. Moreover, like the electronic scoring systems now being produced by educational test developers, they measure easily quantified aspects of texts (like vocabulary and punctuation) rather than what we really want to know—how deeply students are learning the skills and knowledge they need for success in life and work.

CLAS is the first computerized developmental scoring system that measures what educators really want to know—how well learners can put their knowledge to work in real-world contexts—reliably enough to be used ethically in individual assessment contexts. It's a truly general system that can readily be adapted for use in diverse knowledge domains. Rather than measuring whatever aspects of texts are most easily quantified, CLAS measures growth along a well-validated developmental scale called the Lectical Scale. Its algorithms are based on a deep study of learning and development that involves a blend of human expertise and analytics. When CLAS produces a score, it tells us where an assessment performance lands on the Lectical Scale, what the score means in terms of the development of a test-taker's skills, their understanding of concepts targeted by the assessment, and what the test taker is likely to benefit from learning next.

CLAS algorithms are based on human and computer analysis of tens of thousands of texts (and growing). At its core is the Lectical^™ Dictionary—the world's first developmental dictionary—a constantly evolving taxonomy of the development of meanings. (More about the Dictionary below.) 

Like the LAS (Lectical Assessment System, our human scoring system), CLAS can be used:

• to score Lectical Assessments;
• to evaluate the effectiveness of interventions designed to promote cognitive development; 
• to track development over time; 
• to trace conceptual development; 
• in comparative research on conceptual development; and 
• in research on developmental processes.

CLAS has several advantages over the LAS:

1. it makes it possible, for the first time, to conduct large scale developmental assessment,
2. it is easily expandable into new knowledge domains,
3. it is entirely objective, and
4. it dramatically reduces the cost of scoring large numbers of assessments.

Since its introduction in 2014, CLAS has been used to score thousands of assessments for several research and evaluation projects.

The Lectical Dictionary

The Lectical Dictionary is at the core of everything we do here at Lectica. It not only makes CLAS possible, but we use it to describe learning sequences and develop customized learning resources and other report feedback. We've also used it to create a better spell checker—one that takes the developmental level of performance into account when it makes spelling suggestions.

The Dictionary is composed of units of meaning called "Lectical Items"—words or phrases like "evidence" and "reliable evidence" that carry meaning. Each Lectical Item is assigned to a Lectical Phase (1/4 of a Lectical Level), based on a combination of empirical evidence, the judgment of our analysts, and a variety of helper algorithms. The goal is to assign items to the lowest level at which the simplest meaning they carry is likely to be useful. 

For example, an examination of items containing the word evidence reveals easily observed progressions in the development of its meaning, such as the following:

• Phase 09b: something that I know is true
• Phase 09c: good information that comes from something people have seen or proved (same as a fact)
• Phase 09d: more or less proven facts that you can use to persuade others
• Phase 10a: information that comes from good research and can be used to support arguments
• Phase 10b: information that comes from different kinds of research and sources and needs to be evaluated before you use it to support arguments

Lectical Dictionary entries begin with first speech and cover the full span of development. In addition to being assigned to a Lectical Phase, many of the items in the Lectical Dictionary have been assigned to thematic strands such as deliberation, conflict resolution, the physics of energy, or evidence. For example, at this writing, there are over 85,000 terms in the Lectical Dictionary that relate to evidence. Longitudinal and cross-sectional analyses of sequences like the evidence sequence has demonstrated that each successive conception builds upon previous conceptions (Dawson & Gabrielian, 2003; Dawson- Tunik, 2004). These findings are consistent with the developmental theory upon which the dictionary is based (Fischer, 1980; Piaget, 1985), and suggest that Lectical Items assigned to a particular phase can be said not only to represent the understandings of that phase but also play the role of building blocks for future conceptions.

The distribution of Lectical Items from different phases within a given performance can, therefore, be thought of as evidence of the historical pattern of an individual’s development. The rate of dictionary development has increased as what we learn about patterns in the acquisition of concepts is gradually integrated into our methods and technology. For example, we have learned that there are regularities in the progression of verb conjugation within particular developmental levels and that, in some cases, new single-word Lectical Items typically don’t appear next to “and” or “or” until the phase following the phase to which they have been assigned. Patterns like these, when adequately regular, allow a degree of automation in the lexicating process. The Lectical Dictionary is continuously monitored, refined, and added to by a team of trained analysts. Over time, as we build new assessments in new subject areas, the Lectical Dictionary will become an increasingly comprehensive curated taxonomy of learning.

As of May 2021, the Dictionary contained over a million curated items, which CLAS uses to score 6 different assessments.

Unlike the purely computational scoring algorithms of conventional "big data" analysis or "meta-analytics," CLAS's algorithms are developed through a human/machine collaboration. CLAS and our analysts are constantly engaged in "conversations" about the placement of Lectical Items. CLAS's algorithms represent our current knowledge, and our analysts interact with CLAS to integrate new knowledge. This partnership approach required a great deal of human labor and learning upfront, but the resulting system produces scores that are more accurate and much more meaningful than those produced with analytics alone. 

In addition to providing a Lectical Score, CLAS provides several indicators of confidence in that score. We use these indicators to help us determine when a score needs human review and to identify cheating. The application that computes confidence statistics documents word count, calculates the density of spelling errors, and provides several statistics that allow us to determine if the curve for a given performance deviates from an expected pattern. 

Working with CLAS

CLAS is not only making it possible to fulfill our educational mission—to optimize learning for everyone—but its technology will be useful in a wider range of contexts, including marketing, software, and psychological and sociological research. Over the next few years, we'll be exploring a range of applications.

If you'd like to know more about CLAS or have ideas about how CLAS technology could enhance your work, please contact us.