Table 3 Uncovering Student Thinking about Living Systems. Concept Inventories (CIs) and their associated design solutions for the valid assessment of biological understanding in light of the assessment triangle

Embarking on assessment design without a robust and explicit cognitive framework.

Assessment design does not consider the cognitive theories that are implicitly being assumed when eliciting observations from assessment tasks.

Integrate theoretical work in developmental and cognitive psychology with the NRC (2001) assessment triangle to start assessment design with theoretical grounding.

Offering only either-or item structure (concepts vs. misconceptions) but not both.

Mixtures of correct and incorrect ideas have been shown to be common or the norm in some domains.

Multiple-True-False formats in which all options may be selected as correct or incorrect.

Using pieces to infer holistic reasoning.

Selection of all of the normative pieces of an idea does not necessarily indicate that the pieces are assembled, understood, or applied in a normative manner.

Ordered Multiple Choice formats present different arrangements and combinations of ideas reflective of levels of understanding or expertise. Assesses holistic understanding.

Situating tasks in one context or no context.

Items typically measure knowledge in one situation, whereas studies show that students often display different understandings across contexts. Or, items measure knowledge in an abstract, generalized situation but not in a real-world context.

Measure knowledge across at least two situations or contexts (e.g., different taxa, different cell types) to determine stability or robustness of knowledge.

Prioritizing recognition over recall.

Recognizing information is a different cognitive process that recalling information. Nearly all CIs ask students to recognize the accuracy of information but do not have them construct or recall understanding of a concept.

Constructed response assessments are well-suited to measuring knowledge via recall. Computerized scoring of responses is becoming more common, eliminating the drawbacks of time-intensive human scoring.

Interpreting observations using outdated psychometric models and sampling.

Classical Test Theory has many limitations that can distort interpretations of instrument scores. Rasch and IRT methods offer significant advantages.

Gather sufficient sample sizes across diverse participant samples to foster robust interpretations of biological understanding or attitudes.