Next generation assessment and test development

1.

Next generation assessment
and test development
Tarasova Ksenia

«Technologically Enhanced Items»- technology-enabled items,
innovative items, technology-enhanced innovative items, computerbased items, innovative computerized test items…
…contain content or functionality that is not possible in text-based or multiplechoice tasks…
Wendt & Harmes, 1999
…are computerized tasks that utilize the capabilities and functions of the computer and
that are difficult to implement in traditional paper form…
Parshall & Harmes, 2007
TEI:
and what's
that?
…interactive, computer-based tasks, which makes them different from traditional formbased tasks…
Parshall, Davey, and Pashley, 2000
TEIs (technologically enhanced item types) are computer-based test items
that involve specialized interaction to produce an answer and/or
associated response data.
Stimulus vs. Response Interaction

3.

Stimulus: Multimedia

4.

Stimulus: Simulation tool

5.

Stimulus: Simulation

6.

TEI: response interaction
Focuses on the method of receiving or recording a response

7.

(K.Scalise, 2009)

8.

«…evolution from technology-enhanced assessment to assessmentenhanced technology as passing through four stages» (DiCerbo &
Behrens, 2012)
It's a 4-level process:
• use of technology to create new item types and improve feedback on
assessments that are still close neighbors of pre-digital «paper andpencil tests»; creating new types of elements, mechanics, and
feedback improvements
TEI:
next
generation
assessment
• is defined by the creation of new assessments that are focused on a
central, context-providing problem to be solved; these might be
simulations that are designed and administered as discrete assessment
activities (simulation tasks)
• creating natural digital environments designed with embedded
assessment arguments (using of stealth assessment in a game (Shute &
Ventura, 2013)
• consists of an ecosystem that accumulates information from various
natural digital environments like those in level 3.

9.

Virtual Performance-Based Assessments (VPBAs) - are environments for test
takers to interact with systems, sometimes including other persons or agents, in
order to provide evidence about their knowledge, skills, or other attributes.
Digital data capture makes it possible to acquire rich details about test taker
actions and the evolving situations in which they occur.
VPBA
Scenario-Based Assessments
Simulation-Based Assessments
Game-Based Assessments
Collaborative Assessments

10.

Scenario-Based Assessments
A scenario-based assessment introduces a narrative context for
a student’s experience, to add a layer of meaning to
motivations, thoughts, and actions. The simplest version
introduces an ongoing context to a sequence of tasks, which
are frequently in a traditional format (e.g. multiple-choice or
short answer). More complex scenario-based assessments can
add elements of simulation and gaming.
Scenario-Based
Assessments

11.

Simulation-Based Assessments
In simulation-based assessments, the goal is to create a particular
situation that resembles the real-world context (e.g., conducting a
science experiment, administering medical care, or flying a jet fighter) to
provide opportunities to assess individuals’ capabilities in such a
situation” (Davey et al., 2015).
SimulationBased
Assessments
Simulation-based assessments are particularly important to afford
opportunities for people to demonstrate their knowledge, skills, and
abilities in situations that can be too expensive, too time consuming, or
too dangerous to carry out in real-world contexts (Snir et al., 1993).

12.

Game-Based Assessments
Game-based assessments incorporate principles of game design in
addition to assessment design to measure individuals’ knowledge,
skills, and abilities. These types of assessment can increase
motivation by making the tasks more engaging.
In designing a game-based assessment, it is important to find the
balance between design considerations for game elements and
assessment elements.
Given that game designers build complex, interactive experiences,
game-based assessments are particularly suitable for complex
constructs such as systems thinking and crosscultural competence.
Game-Based
Assessments

13.

Collaborative Assessments
Collaborative assessments are designed to create opportunities for individuals to
demonstrate their proficiency in working or communicating effectively with others.
For this purpose, an environment is created that ensures communication and
cooperation with one or more persons.
Virtual environments are well-suited for this, as they can present complex problems
and scenarios for individuals to work through in concert with others.
• communication can be asynchronous, such as streaming online conversations in a
learning management system or massive open online courses (MOOCs) (Rosé,
Ferschke, 2016; Wise, Chiu, 2011)
Collaborative
Assessments
synchronous, where people communicate in real time via chat or audio and video
channels (Andrews-Todd & Forsyth, 2020; Andrews-Todd et al., 2019; Bower et al.,
2017; van Joolingen et al., 2005)
• can include environments that support communication between a human
participant and one or more computer agents (Biswas et al., 2010; Graesser et al.,
2012; OECD, 2013).
• can include two human participants and artificial agents (e.g., Liu et al., 2015).

14.

Collaborative Assessments
In virtual environments, all the actions and discourse among
team members can be captured rather than just the answers
they provide. The resulting data can illuminate the processes
and strategies team members use during performance, as to
both the substance of the problem and the nature of the
interactions.
Collaborative
Assessments

15.

The work and mastery of the
test developer in today's digital
world is to solve some of the
outlined problems even before
the psychometric analysis stage.

16.

https://us.dqworld.net/lang:en_GB/#!/landing
Digital intelligence - a set of social, emotional and cognitive abilities that
enable people to face challenges and adapt to the demands of digital life.
International
instruments:
DQ

17.

Pay attention to the instructions that familiarize with the interface and system rules. The
more interactive the test, the more detailed and interactive the instructions should be.
International
instruments:
DQ

18.

19.

20.

21.

The tool's interface engages the target audience in the performance of the test.
International
instruments:
DQ