1EdTech Question & Test Interoperability: ASI Outcomes Processing Final Specification Version 1.2

IPR and Distribution Notices

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

1EdTech takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on 1EdTech's procedures with respect to rights in 1EdTech specifications can be found at the 1EdTech Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © 2002 1EdTech Consortium. All Rights Reserved.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

Use of this specification to develop products or services is governed by the license with 1EdTech found on the 1EdTech website: http://www.imsglobal.org/license.html.

The limited permissions granted above are perpetual and will not be revoked by 1EdTech or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

Table of Contents

1. Introduction
     1.1 Question & Test Interoperability Overview
     1.2 Scope & Context
     1.3 Structure of this Document
     1.4 Nomenclature
     1.5 References

2. Information Model
     2.1 Exchanging ASI Objects
     2.2 Use Cases
     2.3 ASI Issues
     2.4 Outcomes Processing Solution
           2.4.1 The In-built Scoring Algorithms
           2.4.2 Defined Proprietary Algorithms
     2.5 Tabular Representation
           2.5.1 QTI Outcomes Processing Data Objects

3. XML Binding
     3.1 outcomes_processing> Elements
           3.1.1 <qticomment> Element
           3.1.2 <outcomes> Element
           3.1.3 <objects_condition> Element
           3.1.4 <processing_parameter> Element
           3.1.5 <map_output> Element
           3.1.6 <outcomes_feedback_test> Element
     3.2 <outcomes> Element
           3.2.1 <qticomment> Element
           3.2.2 <decvar> Element
           3.2.3 <interpretvar> Element
     3.3 <objects_condition>
           3.3.1 <qticomment> Element
           3.3.2 <outcomes_metadata> Element
           3.3.3 <and_out> Element
           3.3.4 <or_out> Element
           3.3.5 <not_out> Element
           3.3.6 <objects_parameter> Element
           3.3.7 <map_input> Element
           3.3.8 <objectscond_extension> Element
     3.4 <outcomes_feedback_test> Element
           3.4.1 <test_variable> Elements
           3.4.2 <displayfeedback> Element
     3.5 <outcomes_metadata> Element
     3.6 <and_objects> Element
           3.6.1 <outcomes_metadata> Element
           3.6.2 <and_objects> Element
           3.6.3 <or_objects> Element
           3.6.4 <not_objects> Element
     3.7 <or_objects> Element
           3.7.1 <outcomes_metadata> Element
           3.7.2 <and_objects> Element
           3.7.3 <or_objects> Element
           3.7.4 <not_objects> Element
     3.8 <not_objects> Element
           3.8.1 <outcomes_metadata> Element
           3.8.2 <and_objects> Element
           3.8.3 <or_objects> Element
           3.8.4 <not_objects> Element
     3.9 <test_variable> Element
           3.9.1 <variable_test> Element
           3.9.2 <and_test> Element
           3.9.3 <or_test> Element
           3.9.4 <not_test> Element
     3.10 <and_test> Element
           3.10.1 <variable_test> Element
           3.10.2 <and_test> Element
           3.10.3 <or_test> Element
           3.10.4 <not_test> Element
     3.11 <or_test> Element
           3.11.1 <variable_test> Element
           3.11.2 <and_test> Element
           3.11.3 <or_test> Element
           3.11.4 <not_test> Element
     3.12 <not_test> Element
           3.12.1 <variable_test> Element
           3.12.2 <and_test> Element
           3.12.3 <or_test> Element
           3.12.4 <not_test> Element

4. Best Practice & Implementation Guide
     4.1 Overall Data Model
     4.2 Relationship to the Other 1EdTech Specifications
     4.3 Basic Example XML Instances
           4.3.1 Example ('Number Correct' and 'Number Correct Attempted')
           4.3.2 Example ('Weighted Number Correct', 'Weighted Number Correct Attempted',
'Parameter Weighted Number Correct' and 'Parameter Weighted Number
Correct Attempted')
           4.3.3 Example ('SumofScores' and 'SumofScoresAttempted')
           4.3.4 Example ('Weighted Sum of Scores', 'Weighted Sum of Scores Attempted', 'Parameter Weighted Sum of Scores' and 'Parameter Weighted Sum of Scores Attempted')
           4.3.5 Example ('Best K from N')
           4.3.6 Example ('Guessing Penalty' and 'WeightedGuessingPenalty')
           4.3.7 Example (Remapped Output 'Sum of Scores')
           4.3.8 Example (Remapped Input 'Sum of Scores')
           4.3.9 Example (Parameterized 'Sum of Scores')
     4.4 Advanced Example XML Instances
           4.4.1 Example (Multiple-choice Quiz)
           4.4.2 Example (True/false Quiz)
     4.5 Implementation Guidance
           4.5.1 The In-built Scoring Algorithms
           4.5.2 Defining Proprietary Scoring Algorithms
           4.5.3 Using Meta-data
           4.5.4 Consideration of Selection & Ordering
     4.6 Example XML Instances
     4.7 Proprietary Extensions
     4.8 V1.x/V2.0 Issues & Compatibility
     4.9 1EdTech Harmonization

Appendix A - Glossary of Terms
     A1 - General Terms
     A2 - Elements & Attributes

Appendix B - In-Built Scoring Algorithms
     B1 - 'Number Correct' Scoring Algorithm
     B3 - 'Weighted Number Correct' Scoring Algorithm
     B4 - 'Weighted Number Correct (Attempted)' Scoring Algorithm
     B5 - 'Parameter Weighted Number Correct' Scoring Algorithm
     B6 - 'Parameter Weighted Number Correct (Attempted)' Scoring Algorithm
     B7 - 'Sum of Scores' Scoring Algorithm
     B8 - 'Sum of Scores (Attempted)' Scoring Algorithm
     B9 - 'Weighted Sum of Scores' Scoring Algorithm
     B10 - 'Weighted Sum of Scores (Attempted)' Scoring Algorithm
     B11 - 'Parameter Weighted Sum of Scores' Scoring Algorithm
     B12 - 'Parameter Weighted Sum of Scores (Attempted)' Scoring Algorithm
     B13 - 'Best K from N' Scoring Algorithm
     B14 - 'Guessing Penalty' Scoring Algorithm
     B15 - 'Weighted Guessing Penalty' Scoring Algorithm

Appendix C - Logic Rules

About This Document
     List of Contributors

Revision History

Index

1. Introduction

1.1 Question & Test Interoperability Overview

The 1EdTech Question & Test Interoperability (QTI) specification describes a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports [QTI, 02i]. Therefore, the specification enables the exchange of this item, assessment and results data between Learning Management Systems, as well as content authors and, content libraries and collections. The 1EdTech QTI specification is defined in XML to promote the widest possible adoption. XML is a powerful, flexible, industry standard mark-up language used to encode data models for Internet-enabled and distributed applications. The QTI specification is extensible and customizable to permit immediate adoption, even in specialized or proprietary systems. Leading suppliers and consumers of learning products, services and content contributed time and expertise to produce this final specification. The 1EdTech QTI specification, like all 1EdTech specifications, does not limit product designs by specifying user interfaces, pedagogical paradigms, or establishing technology or policies that constrain innovation, interoperability, or reuse.

The 'Outcomes Processing' specification contains the description of how the aggregated scores at the Assessment and Section levels can be derived. These scoring outcomes are based upon the child Sections and/or Items. A wide range of scoring algorithms is supported through the usage of a predefined set of parameterized instructions; these avoid the realization of the algorithms within the XML. This document contains the relevant information model, XML binding and best practices guidance but it should be read in the context of the core ASI documents.

1.2 Scope & Context

This document is the 1EdTech Question & Test Interoperability ASI Outcomes Processing Specification and must be read in conjunction with the core documents:

• 1EdTech QTI: ASI Information Model [QTI, 02a];
• 1EdTech QTI: ASI XML Binding [QTI, 02b];
• 1EdTech QTI: ASI Best Practice & Implementation Guide [QTI, 02c].

It defines the outcomes processing features that are to be applied to the Section and/or Assessments. These new Assessment and Section features are not backwards compatible with the other versions of the 1EdTech QTI specifications. These new features have no effect on the 1EdTech QTILite specification [QTI, 02h] because that refers only to the Item data structure.

1.3 Structure of this Document

The structure of this document is:

1.4 Nomenclature

1.5 References

2. Information Model

2.1 Exchanging ASI Objects

The possible advanced Assessment structures that can be exchanged using QTI are shown in Figure 2.1:

• The Assessment could contain a single Section block (Figure 2.1a);
• The Assessment could consist of a reference to an external Section block (Figure 2.1b);
• The Assessment could consist of a mixture of contained and externally referenced Section blocks (Figure 2.1c). There is no preferential order in the blocks and references.

(a)	(c)
(b)

In the case of Sections, some of the possible data structures that can be exchanged are shown in Figure 2.2:

• The Section could contain one or more Items (Figure 2.2a);
• The Section could contain one or more references to external Items (Figure 2.2b);
• The Section could contain one or more Sections (Figure 2.2c);
• The Section could contain one or more references to external Sections (Figure 2.2d);
• The Section could contain a mixture of blocks and references to Section and Items. There is no constraint on the order in which the Sections and Items are declared/referenced i.e. interleaving is supported.

(a)	(b)
(c)	(d)

(e)

The wide range of different data structures that can be exchanged is especially significant when developing mechanisms that support 'Outcomes Processing'. 'Outcomes Processing' is responsible for:

• The aggregation of scores assigned to the Items and/or Sections to create a score or set of scores for the parent Section. The ways in which this roll-up is achieved reflects the different scoring algorithms;
• The aggregation of scores assigned to Sections to create a score or set of scores for the parent Assessment. The ways in which this roll-up is achieved reflects the different scoring algorithms;
• Any assignment of scores based upon more than one Item and/or Section.

It is important to note that the type of scoring algorithm that is to be used is not mandated by the 1EdTech specification. Instead, the specification is capable of supporting most of the available scoring algorithms. Also, the algorithms themselves are not encoded within the specification itself. Instead the type of algorithm is specified and the necessary parametric information supplied to enable the assessment engine to execute the corresponding algorithm.

2.2 Use Cases

The following representative use-cases have been identified for support within QTI ASI outcomes processing V1.2:

1. Multiple-choice quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete (single response) questions whose outcomes are a dichotomous variable e.g. multiple-choice questions. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "CORRECT" for each Item. This takes on the value "True" if the response was correct and "False' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the percentage of total score obtained by a candidate and (3) the percentage of items actually attempted which were correct;
2. True/false quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete true/false. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "isCorrect" for each Item. This takes on the value '1' if the response was correct and "-1' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the weighted total score;
3. Multiple-response quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are multiple-response questions (two answers for each) with partial scores being awarded when at only one correct answer is given. For the sake of being definite we assume there are 5 Items and that we have a defined outcome, "SCORE" for each Item. This takes on the value "+1" for each correct response "-1' for each incorrect response. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the percentage of items actually attempted which were correct;
4. End of chapter test - this is the same as use-case (a) except that all of the items have different weights. We wish to see (1) the number of correct items, (2) the percentage of total weight answered correctly by the candidate and (3) the "weighted" percentage of questions attempted correctly of those questions answered;
5. English comprehension/composition test with essay - this is a placement test designed to assess English language ability. The test consists of two sections: (1) Short answer items (multiple-choice) and (2) the Essay. Short answer items come in two kinds. The first are 15 discrete vocabulary, grammar and usage Items that all produce a single dichotomous outcome "isCorrect". The second are the reading comprehension Items (although they may actually test other skills than just reading) that consist of a stimulus followed by five discrete questions. We assume that these are done as Items with five responses and hence five outcomes: these are Boolean variables whose meanings are as above. All of these Items can be given different weights (as can the 5 outcomes of the two reading comprehension items). The essay question is graded "A,B,C,D,E,F" by either a human or computer rater. The divisions do not necessarily represent equal spacings. The essay represents 1/3 of the total grade. Out of ten points, the mapping of the grades is as follows A=10, B=8, C=7, D=5, E=2, F=0. The total score for the assessment is required;
6. Diagnostic scoring - this is an additional score report which has been retrofitted onto the 'English comprehension test ((e) above). In addition to the overall score, "diagnostic" scores are going to be given in four different sub-areas: "Reading", "Writing", "Vocabulary" and "Grammar" (the dependencies among these sub-scores and the overall score are ignored for computational simplicity). The 15 discrete items and each of the 10 outcomes from the reading items all load onto these sub-scores differently. Some outcomes are completely irrelevant to some of the sub-scores. An Item/outcome can have a high "vocabulary" load and a low "grammar" load, or vice-versa. The essay is also added onto each of the sub-scores, but with a different loading;
7. Diagnostics complex response biology lite - this is a classroom assessment designed to assess the students understanding of the scientific method as it applies to Biology. The assessment returns not a single score but three scores reflecting "Domain Knowledge" (knowledge of the biological subjects in the assessment), "Methodological Knowledge" (understanding of the scientific method) and "Integrated Knowledge" (the ability to apply the scientific method to a given problem). The items are a series of complex "tasks". Each task gives a piece of scenario of a scientific investigation and has the students work through several steps of the process, answering questions about what they are doing at each step. Each task has many complex responses, and hence many observable outcomes. The outcomes are related to the number of steps in the task. Each is a variable given one of the values "Low", "Medium", or "High" depending on how the student did on one part of one step. The outcomes are labelled "OUT***" where '***' is a sequence number unique to the item. For more definite, assume that there are five tasks: Task1 has 7 outcomes, Task2 has 5, Task3 has 8, Task4 has 6 and Task5 has 13. For each of the reporting variables, the scoring model should add an appropriate "weight" if the outcome has reached an appropriate level (either "Medium" and above or "High"). The weight and the level will, in general, be different for each of the reporting variables. Not all outcomes are used for all of the reporting variables;

2.3 ASI Issues

The core features within the ASI structures that are related to the process of selection and ordering are shown in Figure 2.3.

The relationship of these features to outcomes processing are:

• <qtimetadata> - contains the QTI-specific meta-data about the object. This meta-data may be used to decide which objects are selected to support a particular aggregation mechanism;
• <outcomes_processing> - the set of aggregation processing instructions that are to be applied to the child objects. The variables to contain the aggregated scores are declared using the <outcomes> element;
• <feedback> - the feedback that is to be presented if the conditions defined within the <outcomes_processing> occur;
• <selection_ordering> - the selection and ordering rules that are to be applied to the set of child objects contained within the parent. Only those objects that are presented to the participant can contribute to the scoring algorithm but these may or may not be attempted by the participant;
• Internal and externally referenced Section and Item objects - the set of contained and referenced objects which can be selected and ordered;
• External metadata - the external meta-data descriptions that are linked to the object and which conform to the 1EdTech Meta-data specification.

The scoping of the scoring algorithms that are supported is summarized using the example Section/Item combination shown in Figure 2.4. The scope of the scoring algorithm is:

• For Section 'Section Ident_1' the set of children objects is - 'Item Ident_1', 'Section Ident_2', 'Section Ident-3' and 'Item Ident_2'. This means that the scoring algorithm identified in Section 'Section Ident_1' is applied to the default variables associated with each child object;
• For Section 'Section Ident_2' the set of children objects is - 'Item Ident_2.1', 'Item Ident_2.2' and 'Item Ident_2.3'. The Section-level default variables are formed from the aggregation of these child objects;
• For Section 'Section Ident_3' the set of children objects is - 'Item Ident_3.1', 'Item Ident_3.2' and 'Item Ident_3.3'. The Section-level default variables are formed from the aggregation of these child objects;
• If for all of these three Sections only some of the objects are selected then the corresponding aggregation is based upon those objects presented.

It is the responsibility of the scoring engine that is implementing the scoring algorithm to maintain the scoping of the variables. If the default variable names are used then all of the variables will be named 'SCORE'. For instance, the scoping should ensure that:

Section_Ident_2.SCORE = Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE

At the level of the 'Section_Ident_1' then the variables 'Item_Ident_2.1.SCORE' etc. are only perceived through the variable 'Section_Ident_2.SCORE'.

2.4 Outcomes Processing Solution

The outcomes processing capability is based upon two complimentary mechanisms:

• In-built scoring algorithms - in which the appropriate scoring algorithm is just named and so the corresponding assessment engine is responsible for applying the algorithm to the default input and output variables;
• Defined proprietary algorithms - in which the relationship between the input and output scoring variables is defined using a parameterized approach.

2.4.1 The In-built Scoring Algorithms

The algorithms that are to be supported are:

1. Number of right answers (including multivariate responses) - this is identified by the algorithm name "NumberCorrect" and has associated with it the default integer variable 'COUNT'. This is the store for the number of correctly answered objects. An object is defined as correctly answered if the value assigned to the default Boolean variable 'COUNT' for the object is 'True'. The number correct algorithm can only operate on Boolean scoring variables. The default integer scoring variables COUNT.max (equal to the number of objects attempted), COUNT.min (set to zero) and COUNT.normalized (a value of between zero and one) are also available. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
2. Number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'NumberCorrectAttempted';
3. Weighted number right (including multivariate responses) - this is similar to the 'number of rights answers' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedNumberCorrect". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
4. The weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedNumberCorrectAttempted';
5. Parameterized weighted number correct - name 'ParameterWeightedNumberCorrect' this is similar to the 'WeightedNumberCorrect' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of CORRECT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
6. The parameterized weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedNumberCorrectAttempted';
7. Percentage correct (including multivariate responses) - this score is derived from both the "NumberCorrect" and "NumberCorrectAttempted" algorithms in which the percentage correct is defined by the equation-

1. Sum-of-scores - this is identified by the algorithm name "SumofScores" and has associated with it four default integer variables SCORE, SCORE.min, SCORE.max and SCORE.normalized. These variables store the sum of the scores, the minimum possible score (set to the sum of the 'minvalue' attributes for all of the corresponding objects), the maximum possible score (set to the sum of the 'maxvalue' attributes for all of the corresponding objects) and the normalized score (a value of between zero and one) respectively for the set of objects (the minimum, maximum and normalized values are only valid if they have been explicitly defined for all of the corresponding objects). The value of 'SCORE' is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
2. Sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'SumofScoresAttempted';
3. Weighted sum of scores (including multivariate responses) - this is similar to the 'sum of scores' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedSumofScores". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
4. The weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedSumofScoresAttempted';
5. Parameterized weighted sum of scores - name 'ParameterWeightedSumofScores' this is similar to the 'WeightedSumofScores' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
6. The parameterized weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedSumofScoresAttempted';
7. Best 'K' out of 'N' (including multivariate responses) - this is identified by the algorithm name "BestKofN" and is used to calculate the outcome by using the highest 'K' scores from the set of presented objects generically defined as 'N' objects. The values are placed in the default variables 'SCORE', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
8. Negative scores (including multivariate responses) - called 'GuessingPenalty'. Each object has three associated variables for the values of the number of correct answers ('COUNT.correct'), the number of incorrect answers ('COUNT.incorrect') and the number of unattempted answers ('COUNT.unattempted');
9. Item Response Theory (IRT) - supported using the processing parameter mechanism;
10. Time remaining - for further study in V2.0;
11. Par scores - for further study in V2.0;
12. Cut scores (pass/fail, right/wrong and including multivariate responses) - for further study in V2.0;
13. Multiple forms/equating (table based) - for further study in V2.0;
14. Partial credit/Graded response - for further study;
15. Critical Item - for further study in V2.0;
16. Critical Outcome - for further study in V2.0;
17. Factor analysis/ Multivariate IRT (research) -supported through extensions. For further study in V2.0;
18. Bayes net (research) - supported through extensions. For further study in V2.0;
19. Computer Adaptive Testing - for further study in V2.0.

2.4.2 Defined Proprietary Algorithms

Proprietary algorithms can be used but these must be characterized using the parameter definition mechanisms supported in the specification i.e. the <objects_parameter> and <processing_parameter> elements. Any number of parameters can be passed using these elements and there is no restriction on the naming convention used for these parameters - all parameters are specific to an algorithm. The names of the proprietary algorithms (passed in the scoremodel attribute) should not clash with the intrinsic names and some form of naming convention based upon the creating organization should be adopted.

2.5 Tabular Representation

The tables in this Section provide a conceptual, informative description of the elements in the data objects. The columns in these tables refer to:

Note: Additional descriptive information about the element.

2.5.1 QTI Outcomes Processing Data Objects

Table 2.1 describes the data objects that are used in the construction of the QTI outcomes processing elements.

3. XML Binding

3.1 outcomes_processing> Elements

Description: The <outcomes_processing> element is the container for all of the outcomes processing instructions for Assessments and Sections. Multiple outcomes processing containers can be use when multiple scoring algorithms are to be applied to produce the aggregated outcomes. If multiple outcomes_processing elements are supplied, it is the intention that all should be run and the outcomes from all of them together should be reported as the outcomes of the enclosing Section or Assessment. The outcome variables defined by each <outcomes_processing> element should be unique across all outcomes_processing elements defined by a section. In particular, it is an error for multiple <outcomes_processing> elements to set the same outcome variable and the results will be undefined.

Multiplicity: The <outcomes_processing> occurs zero or more times within the <assessment> and <section> elements.

Attributes:

• scoremodel (optional. Default='SumofScores'). The type of the scoring algorithm that is to be used.
  Data-type = String (max of 32 chars).

Elements:

• qticomment
• outcomes
• objects_condition
• processing_parameter
• map_outcome
• outcomes_feedback_test

Example:

<outcomes_processing scoremodel="SumofScores">
  <outcomes>
    <decvar/>
  </outcomes>
</outcomes_processing>

3.1.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes processing structure as a whole.

Multiplicity: Occurs zero or once within the <outcomes_processing> element.

Attributes:

• xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
  Data-type = string.

3.1.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0).

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: See sub-section 3.2.

3.1.3 <objects_condition> Element

Description: This element contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value using the in-built set of algorithm definitions.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes: See sub-section 3.3.

3.1.4 <processing_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes:

• pname (mandatory). The name of the proprietary parameterized value that is to be used by the scoring algorithm.
  Data-type = string (1-64 chars).

3.1.5 <map_output> Element

Description: This element is used to re-map the named variable to another named variable (given in the body of the element). The target variable name must have been declared using <decvar> in the <outcomes> element of the enclosing <outcomes_processing> element. When a variable is remapped, all of its derived variables are to be remapped as well. Thus if remapping 'SCORE' to 'myScore', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized' would be remapped to 'myScore.min', 'myScore.max' and 'myScore.normalized' respectively.
Data-type = string (1-256 chars).

Multiplicity: This occurs zero or many times within the <outcomes_condition> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is normally used by the scoring algorithm e.g. 'SCORE', 'CORRECT', etc. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also to be remapped.
  Data-type = String (max of 256 chars).

3.1.6 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: See sub-section 3.4.

3.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0). The declaration of each variable is accompanied by the implicit declaration of the <vname>.min, <vname>.max and <vname>.normalized variables. These are used to store the minimum value, maximum value and normalized value (in the range zero to one) of the scoring variable.

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: None.

Elements:

• qticomment
• decvar
• interpretvar

3.2.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes.

Multiplicity: Occurs zero or once within the <outcomes> element.

Attributes:

• xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
  Data-type = string (1-2048 chars).

3.2.2 <decvar> Element

Description: The <decvar> element permits the declaration of the scoring variables. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also declared and made available from the scoring algorithms. A <decvar> with no attributes is assumed to define the integer variable 'SCORE' with a default value of zero.

Multiplicity: This occurs one or more times within the <outcomes> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is to be declared.
  Data-type = String (max of 256 chars).
• vartype (required. Enumerated list: String, Decimal, Scientific, Boolean, Integer, Enumerated, Set. Default=Integer). The type of the variable declared.
  Data-type = Enumerated list.
• defaultval (optional). The default value to which the variable is to be initialized.
  Data-type = String (max of 16 chars).
• cutvalue (optional). The value/grade above which the subject is considered to have been mastered.
  Data-type = String (max of 16 chars).
• minvalue (optional). The minimum value permitted for a numeric score.
  Data-type = String (max of 32 chars).
• maxvalue (optional). The maximum value permitted for a numeric score.
  Data-type = String (max of 32 chars).
• members (optional). The set of enumerated values that constitute the member.
  Data-type = String (max of 1024 chars).

3.2.3 <interpretvar> Element

Description: The <interpretvar> element is used to provide statistical interpretation information about the associated variables.

Multiplicity: This occurs zero or more times within the <outcomes> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable whose interpretation details are to be described. The default name is 'SCORE'.
  Data-type = String (max of 256 chars).
• view (optional with selection from the enumerated list of: All, Administrator, AdminAuthority, Assessor, Author, Candidate, InvigilatorProctor, Psychometrician, Scorer, Tutor. Default=All). The view defines the scope for the display of the associated information i.e. to whom the material can be presented.
  Data-type = Enumerated list.

Another description of this element and its sub-components is given in 1EdTech QTI: ASI XML Binding V1.2
[QTI, 02b].

3.3 <objects_condition>

Description: Each <objects_condition> element defines a subset of the objects (Items and Sections) selected by the selection algorithm that are to be used in this score. This allows for the construction of subscores. If no <outcomes_metadata> element is present within the <objects_conditions> element to select a subset of objects, then the <objects_conditions> applies to all objects selected by the selection algorithm. If multiple <objects_conditions> are given within a single <outcomes_processing> element, then the algorithm is applied to the union of all objects selected. The <objects_parameter>, <map_input> and <objectscond_extension> elements within the <objects_condition> element apply only to those objects selected in the condition. If an object is selected by more than one <objects_condition> element, it should receive parameter assignments and input mapping as if it were the first element so selected. These conditions include the identification and mapping of the variables for input.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: None.

Elements:

• qticomment
• outcomes_metadata
• and_objects
• or_objects
• not_objects
• objects_parameter
• map_input
• objectscond_extension

3.3.1 <qticomment> Element

Description: This element contains the comments that are relevant to the presentation of the Item.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes:

• xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
  Data-type = string (1-2048 chars).

3.3.2 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the QTI-specific and/or 1EdTech Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule. This data is a string of up to 64 characters length.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.5.

Elements: None.

3.3.3 <and_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.6.

3.3.4 <or_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.7.

3.3.5 <not_out> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.8.

3.3.6 <objects_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters. These parameters are applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outomes_condition> select the same object, the parameters are taken from the first <outcomes_condition> that selects the element.

Multiplicity: Occurs zero or more times within the <objects_condition> element.

Attributes:

• pname (mandatory). The name of the proprietary parameterized value that is to be used by the scoring algorithm.
  Data-type = string (1-64 chars).

3.3.7 <map_input> Element

Description: This element is used to re-map the named input variable to another variable. The default variable names are derived from the type of scoring algorithm identified in <outcomes_processing> element. The target variable name must have been declared in the evaluation objects that undergoing aggregation and must be of the same type as the default variable. When a variable is remapped, all of its derived variables are to be remapped as well. The input mapping is applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outcomes_condition> select the same object, the input mapping used is the one contained in the first <outcomes_condition> that selects the element.
Data-type = String (1-256 chars).

Multiplicity: This occurs zero or many times within the <objects_condition> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is normally used in the scoring algorithm e.g. 'SCORE', 'COUNT', etc.
  Data-type = String (max of 256 chars).

3.3.8 <objectscond_extension> Element

Description: This element contains the proprietary extensions that can be used to extend the functionally capabilities of the <outcomes_condition> element.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: None.

3.4 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes:

• title (optional). The title of the feedback test.
  Data-type = String (max of 256 chars).

Elements:

• test_variable
• displayfeedback

3.4.1 <test_variable> Elements

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs once within the <outcomes_feedback_test> element.

Attributes: See Sub-section 3.9.

3.4.2 <displayfeedback> Element

Description: The <displayfeedback> element is responsible for assigning an associated feedback to the aggregated scoring if the 'True' state results.

Multiplicity: This occurs zero or more times within the <outcomes_feedback_test> element.

Attributes:

• feedbacktype (optional - enumerated list: Response, Solution, Hint. Default = Response). The type of feedback that has been triggered by the associated outcomes condition. Within the Assessment and Section feedback elements only the 'Response' type of feedback is supported.
  Data-type = Enumerated list.
• linkrefid (required). The identifier of the associated feedback. An <assessfeedback> or <sectionfeedback> element must exist with this identifier as defined by the 'ident' attribute.
  Data-type = String (max of 256 chars).

3.5 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the 1EdTech QTI-specific and/or 1EdTech Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = String (1-64 chars).

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes:

• mdname (mandatory). Identifies the QTI-specific or 1EdTech Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
  Data-type = string (1-256 chars).
• mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
  Data-type = string (1-16 chars).

Elements: None.

Example:

<objects_condition>
  <outcomes_metadata mdname="qmd_topic" mdoperator="EQ">Trigonometry</outcomes_metadata>
</objects_condition>

3.6 <and_objects> Element

Description: The <and_objects> element supports the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'. The logic rules for this element are described in Appendix B.

Multiplicity: This occurs zero or once within the <objects_condition>, and zero or many times within the <and_objects>, <or_objects> and <not_objects> elements.

Attributes: None.

Elements:

• outcomes_metadata
• and_objects
• or_objects
• not_objects

3.6.1 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the 1EdTech QTI-specific and/or 1EdTech Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = string (1-64 chars).

Multiplicity: Occurs zero or many times within the <and_objects> element.

Attributes:

• mdname (mandatory). Identifies the 1EdTech QTI-specific or 1EdTech Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
  Data-type = string (1-64 chars).
• mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
  Data-type = string (1-16 chars).

Elements: None.

3.6.2 <and_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or many times within the <and_objects> element.

Attributes: See Sub-section 3.6.

3.6.3 <or_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or many times within the <and_objects> element.

Attributes: See Sub-section 3.7.

3.6.4 <not_objects> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or many times within the <and_objects> element.

Attributes: See Sub-section 3.8.

3.7 <or_objects> Element

Description: The <or_objects> element supports the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'. The logic rules are described in Appendix B.

Multiplicity: This occurs zero or once within the <objects_condition>, and zero or more times within the <and_objects>, <or_objects> and <not_objects> element.

Attributes: None.

Elements:

• outcomes_metadata
• and_objects
• or_objects
• not_objects

3.7.1 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the 1EdTech QTI-specific and/or 1EdTech Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = string (1-64 chars).

Multiplicity: Occurs zero or many times within the <or_objects> element.

Attributes:

• mdname (mandatory). Identifies the QTI-specific or 1EdTech Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
  Data-type = string (1-64 chars).
• mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
  Data-type = string (1-16 chars).

Elements: None.

3.7.2 <and_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or many times within the <or_objects> element.

Attributes: See Sub-section 3.6.

3.7.3 <or_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or many times within the <or_objects> element.

Attributes: See Sub-section 3.7.

3.7.4 <not_objects> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or many times within the <or_objects> element.

Attributes: See Sub-section 3.8.

3.8 <not_objects> Element

Description: The <not_objects> element supports the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'. The logic rules are listed in Appendix B.

Multiplicity: This occurs zero or once within the <objects_condition>, and zero or many times within the <and_objects>, <or_objects> and <not_objects> element.

Attributes: None.

Elements:

• outcomes_metadata
• and_objects
• or_objects
• not_objects

3.8.1 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the QTI-specific and/or 1EdTech Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = string (1-64 chars).

Multiplicity: Occurs zero or once within the <not_objects> element.

Attributes:

• mdname (mandatory). Identifies the 1EdTech QTI-specific or 1EdTech Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
  Data-type = string (1-64 chars).
• mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
  Data-type = string (1-16 chars).

Elements: None.

3.8.2 <and_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or once within the <not_objects> element.

Attributes: See Sub-section 3.6.

3.8.3 <or_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or many times within the <not_objects> element.

Attributes: See Sub-section 3.7.

3.8.4 <not_objects> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or many times within the <not_objects> element.

Attributes: See Sub-section 3.8.

3.9 <test_variable> Element

Description: The <test_variable> element contains the conditions that are applied to a defined set of input scoring variables to determine if feedback is to be presented. Complex test structures can be constructed using the associated logic elements.

Multiplicity: This occurs once within the <outcomes_feedback_test> element.

Attributes: None.

Elements:

• variable_test
• and_test
• or_test
• not_test

3.9.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or once within the <test_variable> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable whose condition is to be tested. The default name is 'SCORE'.
  Data-type = String (max of 256 chars).
• testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
  Data-type = string (1-16 chars).

3.9.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or once within the <test_variable> element.

Attributes: See Sub-section 3.10.

3.9.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or many times within the <test_variable> element.

Attributes: See Sub-section 3.11.

3.9.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or once within the <test_variable> element.

Attributes: See Sub-section 3.12.

3.10 <and_test> Element

Description: The <and_test> element is used to define variable test conditions all of which must be defined as 'True' to result in the output variables being assigned the corresponding values.

Multiplicity: This occurs zero or once within the <test_variable> element, and zero or more times in the <and_test>, <or_test> and <not_test> elements.

Attributes: None.

Elements:

• variable_test
• and_test
• or_test
• not_test

3.10.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or more times within the <and_test> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is to be tested. The default name is 'SCORE'.
  Data-type = String (max of 256 chars).
• testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
  Data-type = string (1-16 chars).

3.10.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or many times within the <and_test> element.

Attributes: See Sub-section 3.10.

3.10.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or many times within the <and_test> element.

Attributes: See Sub-section 3.11.

3.10.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or many times within the <and_test> element.

Attributes: See Sub-section 3.12.

3.11 <or_test> Element

Description: The <or_test> element is used to define variable test conditions at least one of which must be defined as 'True' to result in the output variables being assigned the corresponding values.

Multiplicity: This occurs zero or more times within the <test_variable> element.

Attributes: None.

Elements:

• variable_test
• and_test
• or_test
• not_test

3.11.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or more times within the <or_test> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is to be tested. The default name is 'SCORE'.
  Data-type = String (max of 256 chars).
• testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
  Data-type = string (1-16 chars).

3.11.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or many times within the <or_test> element.

Attributes: See Sub-section 3.10.

3.11.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or many times within the <or_test> element.

Attributes: See Sub-section 3.11.

3.11.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or many times within the <or_test> element.

Attributes: See Sub-section 3.12.

3.12 <not_test> Element

Description: The <not_test> element is used to define variable test conditions that result in the condition being defined as 'False' to result in the output variables being assigned the corresponding values.

Multiplicity: This occurs zero or more times within the <test_var> element.

Attributes: None.

Elements:

• variable_test
• and_test
• or_test
• not_test

3.12.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or once within the <not_test> element.

Attributes:

• varname (optional. Default = 'SCORE'). The name of the variable that is to be tested. The default name is 'SCORE'.
  Data-type = String (max of 256 chars).
• testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
  Data-type = string (1-16 chars).

3.12.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or once within the <not_test> element.

Attributes: See Sub-section 3.10.

3.12.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or once within the <not_test> element.

Attributes: See Sub-section 3.11.

3.12.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or many times within the <not_test> element.

Attributes: See Sub-section 3.12.

4. Best Practice & Implementation Guide

4.1 Overall Data Model

The overall 1EdTech QTI ASI Outcomes Processing data model is shown in Figure 4.1.

4.2 Relationship to the Other 1EdTech Specifications

The relationship of this 'Outcomes Processing' specification to the other parts of the 1EdTech QTI specifications is:

• The 'Outcomes Processing' root element is a part of the Assessment and Section core data objects in the 1EdTech QTI ASI: Information Model and 1EdTech QTI ASI: XML Binding. The <response_processing> element (defined within the QTI ASI Information model and XML Binding documents) defines the variables that are used as inputs to the response processing when the algorithm operates on items. Similarly, the <outcomes_processing> element (defined in this document) defines the variables that are used as inputs when the outcomes processing operates on sections. Note that in predefined algorithms, these operate on either an integer variable 'SCORE' or a boolean variable 'CORRECT'. To be used with the maximum number of outcomes processing algorithms an item's <response_processing> element should set both variables. Also, some algorithms treat items that were not attempted or not presented differently from those which were attempted so there must be some way of querying the items attempted or presented status. Finally, other algorithms assume that a default value for the outcome variable (usually a null score) is available even if the item was never presented. All of these data can be extracted from a 1EdTech QTI results report, however, most implementations will probably use equivalent internal data structures;
• The 'Selection & Ordering' component of the 1EdTech QTI specification [QTI, 02d] interacts with the 'Outcomes Processing' part. This interaction is a result of the fact that the aggregated score for a Section and/or Assessment depends upon which Items and Section were selected for presentation to the participant;
• The 1EdTech QTI Results Reporting specification contains the description of how the results are recorded for a participant that has undertaken an evaluation. These results will depend upon the aggregated scores derived from the outcomes processing mechanism. The QTI Results Reporting specification describes how the results from any 1EdTech QTI-compliant Assessment, Section and Item can be reported.

4.3 Basic Example XML Instances

The following set of basic examples demonstrates how the different use-cases are supported. Table 4.1 lists the examples used to show the operation of each of the outcomes processing algorithms.

4.3.1 Example ('Number Correct' and 'Number Correct Attempted')

A Section consists of 10 Items each of which has a 'score' assigned to its default scoring boolean variable (CORRECT) that is used to denote if the Item has been answered correctly i.e. 'True'. The number of correct responses is assigned to the default score variable for the Section (COUNT). The outcomes processing algorithms for calculating the number of correct responses both in terms of the Items presented and the Items presented and attempted are given below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S01" ident = "IMS_QTIV1p2_S_OUT_01">
    <outcomes_processing scoremodel = "NumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
    </outcomes_processing>
    <outcomes_processing scoremodel = "NumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_Attempted" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_Attempted</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">

51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">

111
112
113
114
115
116
117
118
119
120
121

      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'NumberCorrect' scoring algorithm that is supported for the Section is:

  COUNT = 0
  COUNT.min = 0
  COUNT.max = 0
  COUNT.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(CORRECT) = True
    then
      begin
        COUNT.max = COUNT.max + 1
        if item.CORRECT = True
        then COUNT = COUNT + 1
      endif
    enddo
  COUNT.normalized = COUNT / COUNT.max

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT = 7

COUNT.min = 0

COUNT.max = 10

COUNT.normalized = 0.7

The 'NumberCorrectAttempted' scoring algorithm that is supported for the Section is (the variable 'COUNT_Attempted' must be used to ensure that there is no scoping clash when two algorithms attempt to operate on the COUNT variable):

  COUNT_Attempted = 0
  COUNT_Attempted.min = 0
  COUNT_Attempted.max = 0
  COUNT_Attempted.normalized = 0
  for ALL_ATTEMPTED_ITEMS do
    if ITEM_HAS_VARIABLE(CORRECT) = True
    then
      begin
        COUNT_Attempted.max = COUNT_Attempted.max + 1
        if item.CORRECT = True
        then COUNT_Attempted = COUNT_Attempted + 1
      endif
    enddo
  COUNT_Attempted.normalized = COUNT_Attempted / COUNT_Attempted.max

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT_Attempted = 7

COUNT_Attempted.min = 0

COUNT_Attempted.max = 8

COUNT_Attempted.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample01.xml'. All more complete description of the 'Number Correct' and 'NumberCorrectAttempted' scoring algorithms is given in Appendices B1 and B2.

4.3.2 Example ('Weighted Number Correct', 'Weighted Number Correct Attempted', 'Parameter Weighted Number Correct' and 'Parameter Weighted Number Correct Attempted')

A Section consists of 10 Items (a mixture of implicit and referenced) and each Item has a 'score' assigned to its default scoring boolean variable (CORRECT) that is used to denote if the Item has been answered correctly i.e. 'True'. The four different weighting algorithms are presented using the appropriate weighting calculation mechanism. The number of correct responses is assigned to the default score variable for the Section (COUNT).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S02" ident = "IMS_QTIV1p2_S_OUT_02">
    <outcomes_processing scoremodel = "WeightedNumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_WNC" vartype = "Integer" 
             minvalue = "0" maxvalue = "15" cutvalue = "10"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_WNC</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "WeightedNumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_WNCA" vartype = "Integer" 
             minvalue = "0" maxvalue = "15" cutvalue = "10"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_WNCA</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedNumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_PWNC" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="COUNT">COUNT_PWNC</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedNumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_PWNC" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="COUNT">COUNT_PWNC</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">

42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">

161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212

      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'WeightedNumberCorrect' scoring algorithm that is supported for the Section is:

  COUNT_WNC = 0
  COUNT_WNC.min = 0
  COUNT_WNC.max = 0
  COUNT_WNC.normalized = 0
  for ALL_SELECTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = item(qmd_weighting)
        COUNT_WNC.max = COUNT_WNC.max + item.weighting
        if item.CORRECT = True
        then COUNT_WNC = COUNT_WNC + item.weighting
      endif
    enddo
  COUNT_WNC.normalized = COUNT_WNC / COUNT_WNC.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_WNC = 10

COUNT_WNC.min = 0

COUNT_WNC.max = 15

COUNT_WNC.normalized = 0.667

The 'WeightedNumberCorrectAttempted' scoring algorithm that is supported for the Section is:

  COUNT_WNCA = 0
  COUNT_WNCA.min = 0
  COUNT_WNCA.max = 0
  COUNT_WNCA.normalized = 0
  for ALL_ATTEMPTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = item(qmd_weighting)
        COUNT_WNCA.max = COUNT_WNCA.max + item.weighting
        if item.CORRECT = True
        then COUNT_WNCA = COUNT_WNCA + item.weighting
      endif
    enddo
  COUNT_WNCA.normalized = COUNT_WNCA / COUNT_WNCA.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_WCNA = 10

COUNT_WCNA.min = 0

COUNT_WCNA.max = 12

COUNT_WCNA.normalized = 0.833

The 'ParameterWeightedNumberCorrect' scoring algorithm that is supported for the Section is:

  COUNT_PWNC = 0
  COUNT_PWNC.min = 0
  COUNT_PWNC.max = 0
  COUNT_PWNC.normalized = 0
  for ALL_SELECTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = objects_parameter(qmd_weighting)
        COUNT_PWNC.max = COUNT.max + item.weighting
        if item.CORRECT = True
        then COUNT_PWNC = COUNT_PWNC + item.weighting
      endif
    enddo
  COUNT_PWNC.normalized = COUNT_PWNC / COUNT_PWNC.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_PWNC = 14

COUNT_PWNC.min = 0

COUNT_PWNC.max = 20

COUNT_PWNC.normalized = 0.7

The 'ParameterWeightedNumberCorrectAttempted' scoring algorithm that is supported for the Section is:

  COUNT_PWNCA = 0
  COUNT_PWNCA.min = 0
  COUNT_PWNCA.max = 0
  COUNT_PWNCA.normalized = 0
  for ALL_ATTEMPTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = objects_parameter(qmd_weighting)
        COUNT_PWNCA.max = COUNT_PWNCA.max + item.weighting
        if item.CORRECT = True
        then COUNT_PWNCA = COUNT_PWNCA + item.weighting
      endif
    enddo
  COUNT_PWNCA.normalized = COUNT_PWNCA / COUNT_PWNCA.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_PWNCA = 14

COUNT_PWNCA.min = 0

COUNT_PWNCA.max = 16

COUNT_PWNCA.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample02.xml'. A more complete description of the weighted number correct' scoring algorithms is given in Appendices B3, B4, B5 and B6.

4.3.3 Example ('SumofScores' and 'SumofScoresAttempted')

A Section consists of 10 Items each of which has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and 0 for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE). The outcomes processing algorithms for calculating the sum of scores both in terms of the Items presented and the Items presented and attempted are given below.

1
2
3
4
5
6
7
8
9
10

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S03" ident = "IMS_QTIV1p2_S_OUT_03">
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
    </outcomes_processing>
    <outcomes_processing scoremodel = "SumofScoresAttempted">
      <outcomes>

11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="SCORE">SCORE_Attempted</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>                                    </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...

71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'SumOfScores' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
    then
      begin
        SCORE.min = SCORE.min + item.SCORE.minvalue
        SCORE.max = SCORE.max + item.SCORE.maxvalue
        SCORE = SCORE + item.SCORE
      endif
    enddo
  SCORE.normalized = (SCORE-SCORE.min) / (SCORE.max-SCORE.min)

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

SCORE = 7

SCORE.min = 0

SCORE.max = 10

SCORE.normalized = 0.7

The 'SumofScoresAttempted' scoring algorithm that is supported for the Section is (the default output variable name has been remapped to avoid overloading the 'SCORE' variable during the concurrent application of the two scoring algorithms):

  SCORE_Attempted = 0
  SCORE_Attempted.min = 0
  SCORE_Attempted.max = 0
  SCORE_Attempted.normalized = 0.0
  for ALL_ATTEMPTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
    then
      begin
        SCORE_Attempted.min = SCORE_Attempted.min + item.SCORE.minvalue
        SCORE_Attempted.max = SCORE_Attempted.max + item.SCORE.maxvalue
        SCORE_Attempted = SCORE_Attempted + item.SCORE
      endif
    enddo
  SCORE_Attempted.normalized = (SCORE_Attempted - SCORE_Attempted.min) / 
                      (SCORE_Attempted.max - SCORE_Attempted.min)

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

SCORE_Attempted = 7

SCORE_Attempted.min = 0

SCORE_Attempted.max = 8

SCORE_Attempted.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample03.xml'. All more complete description of the 'Sum of Scores' and 'Sum of Scores Attempted' scoring algorithms are given in Appendix B7 and B8.

4.3.4 Example ('Weighted Sum of Scores', 'Weighted Sum of Scores Attempted', 'Parameter Weighted Sum of Scores' and 'Parameter Weighted Sum of Scores Attempted')

A Section consists of 10 Items each of which has a score assigned to its default scoring integer variable (SCORE). A score of '1' is given for a correct answer and '0' for an incorrect answer. The four different weighting algorithms are presented using the appropriate weighting calculation mechanism. The sum of these scores is assigned to the default score variable for the Section (SCORE).

1
2

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S04" ident = "IMS_QTIV1p2_S_OUT_04">

3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

    <outcomes_processing scoremodel = "WeightedSumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "19" cutvalue = "12"/>
      </outcomes>
      <map_output varname="SCORE">SCORE_WSOS</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "WeightedSumofScoresAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "15" cutvalue = "10"/>
      </outcomes>
      <map_output varname="SCORE">SCORE_WSOSA</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedSumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="SCORE">SCORE_PWSOS</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedSumofScoresAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="SCORE">SCORE_PWSOSA</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>3</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>

51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>

111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170

        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>6</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>

171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
201
203
204
205
206
207
208
209
210
211
212

          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'WeightedSumOfScores' scoring algorithm that is supported for the Section is:

  SCORE_WSOS = 0
  SCORE_WSOS.min = 0
  SCORE_WSOS.max = 0
  SCORE_WSOS.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_WSOS.min = SCORE_WSOS.min + (item.SCORE.minvalue * item(qmd_weighting))
        SCORE_WSOS.max = SCORE_WSOS.max + (item.SCORE.maxvalue * item(qmd_weighting))
        SCORE_WSOS = SCORE_WSOS + (item.SCORE * item.qmd_weighting)
      endif
    enddo
  SCORE_WSOS.normalized = (SCORE_WSOS - SCORE_WSOS.min) / (SCORE_WSOS.max - SCORE_WSOS.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_WSOS = 16

SCORE_WSOS.min = 0

SCORE_WSOS.max = 19

SCORE_WSOS.normalized = 0.842

The 'WeightedSumOfScoresAttempted' scoring algorithm that is supported for the Section is:

  SCORE_WSOSA = 0
  SCORE_WSOSA.min = 0
  SCORE_WSOSA.max = 0
  SCORE_WSOSA.normalized = 0.0
  for ALL_ATTEMPTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_WSOSA.min = SCORE_WSOSA.min + (item.SCORE.minvalue * item(qmd_weighting))
        SCORE_WSOSA.max = SCORE_WSOSA.max + (item.SCORE.maxvalue * item(qmd_weighting))
        SCORE_WSOSA = SCORE_WSOSA + (item.SCORE * item(qmd_weighting))
      endif
    enddo
  SCORE_WSOSA.normalized = (SCORE_WSOSA-SCORE_WSOSA.min)/(SCORE_WSOSA.max-SCORE_WSOSA.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_WSOSA = 16

SCORE_WSOSA.min = 0

SCORE_WSOSA.max = 17

SCORE_WSOSA.normalized = 0.941

The 'ParameterWeightedSumOfScores' scoring algorithm that is supported for the Section is:

  SCORE_PWSOS = 0
  SCORE_PWSOS.min = 0
  SCORE_PWSOS.max = 0
  SCORE_PWSOS.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_PWSOS.min = SCORE_PWSOS.min + 
                  (item.SCORE.minvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOS.max = SCORE_PWSOS.max + 
                  (item.SCORE.maxvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOS = SCORE_PWSOS + (item.SCORE * objects_parameter(qmd_weighting))
      endif
    enddo
  SCORE_PWSOS.normalized = (SCORE_PWSOS-SCORE_PWSOS.min)/(SCORE_PWSOS.max-SCORE_PWSOS.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_PWSOS = 14

SCORE_PWSOS.min = 0

SCORE_PWSOS.max = 20

SCORE_PWSOS.normalized = 0.7

The 'ParameterWeightedSumOfScoresAttempted' scoring algorithm that is supported for the Section is:

  SCORE_PWSOSA = 0
  SCORE_PWSOSA.min = 0
  SCORE_PWSOSA.max = 0
  SCORE_PWSOSA.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_PWSOSA.min = SCORE_PWSOSA.min + 
                  (item.SCORE.minvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOSA.max= SCORE_PWSOSA.max + 
                  (item.SCORE.maxvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOSA = SCORE_PWSOSA + (item.SCORE * objects_parameter(qmd_weighting))
      endif
    enddo
  SCORE_PWSOSA.normalized =               (SCORE_PWSOSA-SCORE_PWSOSA.min) /
                  (SCORE_PWSOSA.max-SCORE_PWSOSA.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_PWSOSA = 14

SCORE_PWSOSA.min = 0

SCORE_PWSOSA.max = 16

SCORE_PWSOSA.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample04.xml'. All more complete description of the 'Sum of Scores' scoring algorithm is given in Appendices B9, B10, B11 and B12.

4.3.5 Example ('Best K from N')

A Section consists of 10 Items and each has a score assigned to its default scoring integer variable (SCORE). The 'maxvalue' score is given for a correct answer and the 'minvalue' for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE). The aggregated score is set as the best seven scores.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S05" ident = "IMS_QTIV1p2_S_OUT_05">
    <outcomes_processing scoremodel = "BestKofN">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <processing_parameter pname = "BestK">7</processing_parameter>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">

25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>                                    </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">

84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'BestKfromN' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  For HIGHEST_SCORES_FROM_PRESENTED_ITEMS(7) and ITEM_HAS_VARIABLE(SCORE) do
    begin
      SCORE.min = SCORE.min + item.SCORE.minvalue
      SCORE.max = SCORE.max + item.SCORE.maxvalue
      SCORE = SCORE + item.SCORE
    Enddo
  SCORE.normalized = (SCORE - SCORE.min) / (SCORE.max - SCORE.min)

In the case where all ten Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

SCORE = 7

SCORE.min = 0

SCORE.max = 7

SCORE.normalized = 1

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample05.xml'. A more detailed description of these scoring algorithms are presented in Appendices B13.

4.3.6 Example ('Guessing Penalty' and 'WeightedGuessingPenalty')

A Section consists of 10 Items each of which has a 'score' assigned to its default scoring boolean variable (CORRECT) that is used to denote if the Item has been answered correctly i.e. 'True'. The number of correct responses is assigned to the default score variable for the Section (COUNT). The outcomes processing algorithms for calculating the number of correct responses both in terms of the Items presented and the Items presented and attempted are given below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S01" ident = "IMS_QTIV1p2_S_OUT_01">
    <outcomes_processing scoremodel = "GuessingPenalty">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Decimal" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
    </outcomes_processing>
    <outcomes_processing scoremodel = "WeightedGuessingPenalty">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_Weighted" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_Weighted</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>

53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>

113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
142
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171

            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>

172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230

      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>

231
232
233
234
235
236
237
238
239
240
241

      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'GuessingPenalty' scoring algorithm that is supported for the Section is:

  COUNT = 0
  COUNT.correct = 0
  COUNT.incorrect = 0
  COUNT.unattempted = 0
  for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
    begin
      if child.object(ATTEMPTED) = True 
      then
        begin
          if childobject.CORRECT = True
          then 
            begin
              parentobject.COUNT.correct = parentobject.COUNT.correct + 1
              parentobject.COUNT = parentobject.COUNT + 1
            end
          else 
            begin
              parentobject.COUNT.incorrect = parentobject.COUNT.incorrect + 1
              parentobject.COUNT = parentobject.COUNT - item(qmd_penaltyvalue)
            end
        endif
      else childobject.COUNT.unattempted = childobject.COUNT.unattempted + 1
    enddo

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT = 6.8

COUNT.correct = 7

COUNT.correct = 7

COUNT.incorrect = 1

COUNT.unattempted = 2

The 'WeightedGuessingPenalty' scoring algorithm that is supported for the Section is:

  COUNT_Weighted = 0
  COUNT_Weighted.correct = 0
  COUNT_Weighted.incorrect = 0
  COUNT_Weighted.unattempted = 0
  for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
    begin
      if child.object(ATTEMPTED) = True 
      then
        begin
          if childobject.CORRECT = True
          then 
            begin
              parentobject.COUNT_Weighted.correct = 
                parentobject.COUNT_Weighted.correct + 1
              parentobject.COUNT_Weighted = parentobject.COUNT_Weighted + 1
            end
          else 
            begin
              parentobject.COUNT_Weighted.incorrect = 
                parentobject.COUNT_Weighted.incorrect + 1
              parentobject.COUNT_Weighted = 
              (parentobject.COUNT_Weighted * item(qmd_weighting) - 
              item(qmd_penaltyvalue)
            end
        endif
      else childobject.COUNT_Weighted.unattempted = 
        childobject.COUNT_Weighted.unattempted + 1
    enddo

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT = 13.8

COUNT.correct = 7

COUNT.correct = 7

COUNT.incorrect = 1

COUNT.unattempted = 2

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample06.xml'. All more complete description of these scoring algorithm is given in Appendices B14 and B15.

4.3.7 Example (Remapped Output 'Sum of Scores')

A Section consists of 5 Items (a mixture of implicit and referenced) and each Item has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and -1 for an incorrect answer. The sum of these scores is assigned to the new score variable for the Section (Section_06_SCORE).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S07" ident = "IMS_QTIV1p2_S_OUT_07">
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval= "0" varname= "Section_07_SCORE" vartype= "Integer" 
             minvalue= "0" maxvalue= "10" cutvalue= "6"/>
      </outcomes>
      <map_output varname = "SCORE">Section_07_SCORE</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>

60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The output remapped 'SumofScores' scoring algorithm that is supported for the Section is:

  Section_07_SCORE = 0
  Section_07_SCORE.min = 0
  Section_07_SCORE.max = 0
  Section_07_SCORE.normalized = 0.0
  For ALL_ATTEMPTED_ITEMS and ITEM_HAS_VARIABLE(SCORE) do
    begin
      Section_07_SCORE.min = Section_07_SCORE.min + item.SCORE.minvalue
      Section_07_SCORE.max = Section_07_SCORE.max + item.SCORE.maxvalue
      Section_07_SCORE = Section_07_SCORE + item.SCORE
    Enddo
  Section_07_SCORE.normalized = 
  (Section_07_SCORE - Section_07_SCORE.min)/(Section_07_SCORE.max - Section_07_SCORE.min)

In the case where all ten Items have been attempted and eight have been answered correctly then the final set of output remapped Section variable states will be:

Section_07_SCORE = 8

Section_07_SCORE.min = 0

Section_07_SCORE.max = 10

Section_07_SCORE.normalized = 0.8

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample07.xml'.

4.3.8 Example (Remapped Input 'Sum of Scores')

A Section consists of 5 Items (a mixture of implicit and referenced) and each Item has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and -1 for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S06" ident = "IMS_QTIV1p2_S_OUT_06">
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval= "0" varname= "SCORE" vartype= "Integer" 
             minvalue= "0" maxvalue= "10" cutvalue= "6"/>
      </outcomes>
      <objects_condition>
        <map_input varname = "SCORE">item_SCORE</map_input>
      </objects_condition>
    </outcomes_processing>
    <selection_ordering>
      <selection seltype/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>

31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>

90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117

          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The input remapped 'SumofScores' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  For ALL_ATTEMPTED_ITEMS and ITEM_HAS_VARIABLE(item_SCORE) do
    begin
      SCORE.min = SCORE.min + item.item_SCORE.minvalue
      SCORE.max = SCORE.max + item.item_SCORE.maxvalue
      SCORE = SCORE + item.item_Score
    Enddo
  SCORE.normalized = (SCORE - SCORE.min)/(SCORE.max - SCORE.min)

In the case where all ten Items have been attempted and eight have been answered correctly then the final set of Section variable states will be:

SCORE = 8

SCORE.min = 0

SCORE.max = 10

SCORE.normalized = 0.8

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample08.xml'.

4.3.9 Example (Parameterized 'Sum of Scores')

A Section consists of 10 Items and each has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and -1 for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE). The intention is to produce a result score for all those Items on the topic that are 'Advanced' and 'Basic'.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S08" ident = "IMS_QTIV1p2_S_OUT_08">
    <outcomes_processing scoremodel = "SumOfScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "6"/>
      </outcomes>
      <objects_condition>
        <or_objects>
          <outcomes_metadata  mdname="qmd_levelofdifficulty" 
                      mdoperator= "EQ">basic</outcomes_metadata>
          <outcomes_metadata  mdname="qmd_levelofdifficulty" 
                      mdoperator="EQ">advanced</outcomes_metadata>
        </or_objects>
      </objects_condition>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>3</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...

53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...

113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172

      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>6</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>

173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223

      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'SumOfScores' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  For ALL_PRESENTED_ITEMS            and ITEM_HAS_VARIABLE(SCORE)
                and 
    (item.metadata.difficulty = basic or item.metadata.difficulty = advanced) 
  do
    begin
      SCORE.min = SCORE.min + item.SCORE.minvalue
      SCORE.max = SCORE.max + item.SCORE.maxvalue
      SCORE = SCORE + item.SCORE
    Enddo
  SCORE.normalized = (SCORE - SCORE.min) / (SCORE.max - SCORE.min)

In the case where all ten Items have been attempted and seven have been answered correctly then the final set of Section variable states will be (the meta-data selections ensure that all Items are selected to contribute to the score aggregation):

SCORE = 7

SCORE.min = 0

SCORE.max = 10

SCORE.normalized = 0.7

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample09.xml'.

4.4 Advanced Example XML Instances

The following set of advanced examples demonstrates how the different use-cases are supported.

4.4.1 Example (Multiple-choice Quiz)

Multiple-choice quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete (single response) questions whose outcomes are a dichotomous variable e.g. multiple-choice questions. For the sake of being definite we assume there are 5 Items and that we have a defined outcome, "CORRECT" for each Item. This takes on the value "True" if the response was correct and "False' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the percentage of total score obtained by a candidate and (3) the percentage of items actually answered which were correct.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S01" ident = "IMS_QTIV1p2_S_OUTADV_01">
    <outcomes_processing scoremodel = "NumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>

23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "SCORE">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "SCORE">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "NumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_Attempted</map_output>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <sectionfeedback ident = "SectionMastery" view = "All">
      <material>
        <mattext>Well done.  You have passed this Topic.</mattext>
      </material>
    </sectionfeedback>
    <sectionfeedback ident = "SectionFail" view = "All">
      <material>
        <mattext>Sorry.  You have failed this topic.</mattext>
      </material>
    </sectionfeedback>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Which one of the following statements is true about the QTI 
                  specification ?</mattext>
          </material>
          <response_lid ident = "LID01" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes" minnumber = "1" maxnumber = "1">
              <flow_label>
                <response_label ident = "A">

83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
11
112
113
114
115
116
117
118
119
120
121
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142

                  <material>
                    <mattext>It is independent of pedagogy</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>It cannot be extended to support proprietary 
                      features</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>It contains many unnecesary features
                    </mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>It provides only one view for different 
                      types of participants</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <qticomment>To detect the correct answer.</qticomment>
          <conditionvar>
            <varequal respident = "LID01">A</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <qticomment>To detect if the incorrect answer is supplied.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "LID01" case = "No">A</varequal>
            </not>
          </conditionvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "InCorrect"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">

143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202

        <material>
          <mattext>Correct answer is A.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext xml:lang="en-US" xml:space="default" texttype= "" charset= "">
            Answers B, C and D are incorrect.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>The QTI specification supports the exchange of which data 
              structure ?</mattext>
          </material>
          <response_lid ident = "LID02" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes" minnumber = "1" maxnumber = "1">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>Response</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>Question</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>Item</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>Test</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar varname = "CORRECT" vartype = "Boolean" defaultval = "False"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">

203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
281
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262

          <qticomment>Correct response detection</qticomment>
          <conditionvar>
            <varequal respident = "LID02">C</varequal>
          </conditionvar>
          <setvar varname = "CORRECT" action = "Set">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
        <respcondition continue = "No">
          <qticomment>Processing for the incorrect answer.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "LID02">C</varequal>
            </not>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct.  Items.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>No.  The correct answer is Item.</mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>What is the maximum number of Assessments can be contained 
              in a single QTI ASI XML instance?</mattext>
          </material>
          <response_lid ident = "TF01" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>0</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>1</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>More than one</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>

263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322

          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <qticomment>Processing for the incorrect answer.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "TF01">B</varequal>
            </not>
          </conditionvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Incorrect"/>
        </respcondition>
        <respcondition continue = "No">
          <qticomment>Processing for the correct answer.</qticomment>
          <conditionvar>
            <varequal respident = "TF01" case = "No">B</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  A QTI-XML instance CAN contain, 
            at most, one Asssessment.</mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "1">
      <presentation>
        <flow>
          <material>
            <mattext>Which one of the following data structures is used to 
              provide hierarchically structured assessments ?</mattext>
          </material>
          <response_lid ident = "LID03" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>Assessment</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>Section</mattext>

323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382

                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>Item</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>Question</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar varname = "CORRECT" vartype = "Boolean" defaultval = "False"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <qticomment>Correct response check</qticomment>
          <conditionvar>
            <varequal respident = "LID03" case = "No">B</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
        <respcondition>
          <conditionvar>
            <not>
              <varequal respident = "LID03" case = "No">B</varequal>
            </not>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>No.  Only Sections support hierachical Assessments.</mattext>
        </material>
      </itemfeedback>
    </item>    
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "1">
      <presentation>
        <flow>
          <material>

383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
403
403
404
405
406
407
408
409
410
411
412
313
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442

            <mattext>Which question types are supported by the QTILite 
              specification ?</mattext>
          </material>
          <response_lid ident = "LID03" rcardinality = "Multiple" rtiming = "No">
            <render_choice shuffle = "Yes" minnumber = "1" maxnumber = "4">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>Multiple choice</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>Multiple response</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>Fill-in-the-blank</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>Drag-and-drop</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar varname = "CORRECT" vartype = "Boolean" defaultval = "False"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
          <qticomment>Correct answer response processing.</qticomment>
          <conditionvar>
            <varequal respident = "LID03">A</varequal>
          </conditionvar>
          <setvar varname = "CORRECT" action = "Set">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
        <respcondition>
          <qticomment>Incorrect answer response processing.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "LID03">A</varequal>
            </not>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>

443
444
445
446
447
448
448
459
460
461
462
463
464
465
466

        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct.  Multple-choice only.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>No.  Only Multiple-choice questions are supported.</mattext>
        </material>
      </itemfeedback>
    </item>
  </section>
</questestinterop>

The actual percentage total score and percentage of items actually answered will be derived values from the SCORE.normalized (SumofScores) and COUNT_Attempted.normalized (NumberCorrectAttempted) variables. If it is assumed that the candidate attempts the first four questions and gets the first three of those correct, then the set of variables are:

SCORE = 3

SCORE.min = 0

SCORE.max = 5

SCORE.normalized = 0.6

PercentageTotalScore = 60%

COUNT = 3

COUNT.min = 0

COUNT.max = 5

COUNT.normalized = 0.6

COUNT_Attempted = 3

COUNT_Attempted.min = 0

COUNT_Attempted.max = 4

COUNT_Attempted.normalized = 0.75

PercentageCorrectAttempted = 75%

This example is given as 'ims_qtiasiv1p2/outcome/advoutexample01.xml'.

4.4.2 Example (True/false Quiz)

True/false quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete true/false. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "isCorrect" for each Item. This takes on the value '1' if the response was correct and "-1' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the weighted total score.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S02" ident = "IMS_QTIV1p2_S_OUTADV_02">
    <outcomes_processing scoremodel = "NumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
          minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
          minvalue = "-10" maxvalue = "10" cutvalue = "0"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterSumofScoresWeighted">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
          minvalue = "-20" maxvalue = "20" cutvalue = "0"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2</objects_parameter>
      </objects_condition>
      <map_output varname="SCORE">SCORE_PWSOS</map_output>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>

61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
11
112
113
114
115
116
117
118
119
120

    <sectionfeedback ident = "SectionMastery" view = "All">
      <material>
        <mattext>Well done.  You have passed this Topic.</mattext>
      </material>
    </sectionfeedback>
    <sectionfeedback ident = "SectionFail" view = "All">
      <material>
        <mattext>Sorry.  You have failed this topic.</mattext>
      </material>
    </sectionfeedback>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Multiple Assessments cannot be contained in a single QTI 
              ASI XML instance.
            </mattext>
          </material>
          <response_lid ident = "TF01" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the incorrect answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF01">T</varequal>
          </conditionvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Incorrect"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF01" case = "No">F</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>

121
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180

          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  A QTI-XML instance CAN contain multiple 
            Assessments.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Time-based responses are supported by the QTI 
              specification.
            </mattext>
          </material>
          <response_lid ident = "TF02" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
        <decvar defaultval = "-1" vartype = "Integer" minvalue="-1" 
          maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
          <conditionvar>
            <varequal respident = "TF02">T</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>

181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
281
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240

        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF02" case = "No">F</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  Time-based reponses ARE supported by 
            the specification.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Items defined according to QTI V1.0 are not compatible with 
              Items from QTI V1.2.
            </mattext>
          </material>
          <response_lid ident = "TF03" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "True" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
          <conditionvar>
            <varequal respident = "TF03">F</varequal>
          </conditionvar>

241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300

          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF03" case = "No">T</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  V1.0 Items ARE compatible with the V1.2 
            specification.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>The QTI specification is incompatible with the 1EdTech Content 
              Packaging specification.
            </mattext>
          </material>
          <response_lid ident = "TF04" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition>

301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360

          <conditionvar>
            <varequal respident = "TF04">F</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF04" case = "No">T</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  The 1EdTech QTI and Content Packaging 
            specifications are compatible.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <presentation>
        <flow class = "Block">
          <material>
            <mattext texttype = "text/plain" charset = "ascii-us">The QTI 
              specification is used within the ADL SCORM.
            </mattext>
          </material>
          <response_lid ident = "TF05" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label class = "Block">
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label class = "Block">
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 

361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392

            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF05" case = "No">F</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF05" case = "No">T</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>Sorry, you are wrong.  The QTI specification is NOT used 
            within ADL SCORM.
          </mattext>
        </material>
      </itemfeedback>
    </item>
  </section>
</questestinterop>

If it is assumed that the candidate attempts the first four questions and gets the first three of those correct, then the set of variables are:

SCORE = 1

SCORE.min = -5

SCORE.max = 5

SCORE.normalized = 0.6

COUNT = 3

COUNT.min = 0

COUNT.max = 5

COUNT.normalized = 0.6

SCORE_Weighted = 2

SCORE_Weighted.min = -10

SCORE_Weighted.max = 10

SCORE_Weighted.normalized = 0.6

This example is given as 'ims_qtiasiv1p2/outcome/advoutexample01.xml'.

4.5 Implementation Guidance

4.5.1 The In-built Scoring Algorithms

Whenever possible the in-built scoring algorithms should be used to ensure interoperability between systems. It should be noted that it is possible to use multiple scoring algorithms on the same set of evaluation objects - this enables multivariate scoring mechanisms. At the current time the in-built scoring algorithms are:

• Number Correct - see Appendix B1;
• Number Correct Attempted - see Appendix B2;
• Weighted Number Correct - see Appendix B3;
• Weighted Number Correct Attempted - see Appendix B4;
• Parameter Weighted Number Correct - see Appendix B5;
• Parameter Weighted Number Correct Attempted - see Appendix B6;
• Sum of Scores - see Appendix B7;
• Sum of Scores Attempted - see Appendix B8;
• Weighted Sum of Scores - see Appendix B9;
• Weighted Sum of Scores Attempted - see Appendix B10;
• Parameter Weighted Sum of Scores - see Appendix B11;
• Parameter Weighted Sum of Scores Attempted - see Appendix B12;
• Best K of N - see Appendix B13;
• Guessing Penalty - see Appendix B14;
• Weighted Guessing Penalty - see Appendix B15;

4.5.2 Defining Proprietary Scoring Algorithms

It is understood that not all of the required scoring algorithms will be supported intrinsically. Proprietary algorithms can be supported by:

• Assigning an appropriate name to the new scoring algorithm and identifying this algorithm in the 'scoremodel' attribute for the <outcome_processing> element. Scoring engines that do not recognize the named scoring algorithm should indicate an error stating that it will not be possible to produce the associated set of aggregated scores;
• Deciding how the default or specially named will be handled by the new scoring algorithm;
• Assigning particular parameter names and passing these to the scoring algorithm using the <objects_parameter> and <processing_parameter> elements.

In some cases it will be necessary to coerce a numeric score into a count of whether or not the object has been answered correctly. The conversion from a SCORE to a CORRECT variable will be achieved using the cutscore value for the variable. The algorithm now becomes:

  if (object.SCORE >= object.SCORE.cutscore)
  then object.CORRECT = True
  else object.CORRECT = False

4.5.3 Using Meta-data

The meta-data of the objects is an important aggregation object selection mechanism. This selection must operate on the 1EdTech QTI-specific meta-data and the 1EdTech Meta-data definitions. There are three meta-data entry techniques used within 1EdTech QTI:

• For the explicitly identified 1EdTech QTI-specific meta-data fields (the elements named using the 'qmd_**...***' scheme) the name of the meta-data element is used as the string name in the 'mdname' attribute of the <outcomes_metadata> element. The actual meta-data field value is that content of the <qmd_***...***> element. This mechanism is deprecated and should not be used;
• For the 1EdTech QTI-specific meta-data fields named using the generic vocabulary mechanism (using the element <qtimetadatafield>) the string name will be that identified using the <qtimetadatafield><fieldlabel> structure. The actual meta-data field value is the content of the structure <qtimetadatafield><fieldentry>;
• For the 1EdTech Meta-data entries the string naming convention will adopt the '.' mechanism to denote the concatenation of nested tags e.g. the string "metametadata.contribute.role.source" is used to denote the 1EdTech Meta-data structure of '<metametadata><contribute><role><source>'. The 'lom:' namespace should be used at the front of the string.

4.5.4 Consideration of Selection & Ordering

It is important to realize that the actual score assigned can only be determined once the evaluation has been presented. Assessments and Sections can consist of a complex of child objects that are selected and ordered according to some set of rules defined within the <selection_ordering> element. The current outcomes processing mechanism is used to define the static rules that are used to create the aggregated scores that result from the selected, ordered, presented and attempted objects. Scores can only be assigned to objects that have been presented but some of the algorithms will provide normalized information that takes into account whether or not a presented object is attempted by the participant.

4.6 Example XML Instances

The full set of example files, as referred to in Sections 4.3 and 4.4 are available as part of the 1EdTech QTI Resource Kit. The set of example XML instances are listed in Table 4.2.

These files can be found in the directory: 'ims_qtiasiv1p2/outcome/'.

4.7 Proprietary Extensions

The proprietary extensions facilities listed in Table 4.3 are supported as elements within the specifications:

Note: These elements are only used if the suppliers of the ASIs require proprietary features that are not supported by the available range of elements. It is recommended that extensions are used sparingly. Backwards compatibility with proprietary extensions will NOT be guaranteed in future versions of these specifications.

4.8 V1.x/V2.0 Issues & Compatibility

The 'Outcomes Processing' functionality is not backwards compatible with early versions of the specification. These functions that have been replaced in earlier specifications are listed in Table 4.4.

V1.0, V1.01 and V1.1 QTI-XML instances that do not use the elements listed within Table 4.4 are backwards compatible because the <outcomes_processing> element within V1.2 is optional.

4.9 1EdTech Harmonization

The issues of harmonization of this part of the 1EdTech QTI specification with other 1EdTech specifications is addressed by:

• 1EdTech Meta-data - the <outcomes_metadata> element should be capable of operating on any form of the entries defined within the 1EdTech Meta-data specification.

Appendix A - Glossary of Terms

A1 - General Terms

A2 - Elements & Attributes

Appendix B - In-Built Scoring Algorithms

B1 - 'Number Correct' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max = parentobject.COUNT.max + 1
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT + 1
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max

parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max = parentobject.COUNT.max + 1
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT + 1
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max

B3 - 'Weighted Number Correct' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max

B4 - 'Weighted Number Correct (Attempted)' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max

B5 - 'Parameter Weighted Number Correct' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max

B6 - 'Parameter Weighted Number Correct (Attempted)' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max

B7 - 'Sum of Scores' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + childobject.SCORE.minvalue
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + childobject.SCORE.maxvalue
        parentobject.SCORE             = parentobject.SCORE 
                    + childobject.SCORE
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max

B8 - 'Sum of Scores (Attempted)' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + childobject.SCORE.minvalue
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + childobject.SCORE.maxvalue
        parentobject.SCORE             = parentobject.SCORE 
                    + childobject.SCORE
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max

B9 - 'Weighted Sum of Scores' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max

B10 - 'Weighted Sum of Scores (Attempted)' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max

B11 - 'Parameter Weighted Sum of Scores' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max

B12 - 'Parameter Weighted Sum of Scores (Attempted)' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max

B13 - 'Best K from N' Scoring Algorithm

parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized=0.0
parentobject.SCORE.max               = value_of_K  * childobject.SCORE.maxvalue
parentobject.SCORE.min               = value_of_K  * childobject.SCORE.minvalue
SORT (ALLPRESENTED_OBJECTS and child.object(HAS_VARIABLE(SCORE) 
                and ORDERED_BY_ DECREASING_SCORE)
for TOP(value_of_K) do              
  begin
    parentobject.SCORE = 
      parentobject.SCORE  + childobject.SCORE
  enddo
parentobject.SCORE.normalized =
  (parentobject.SCORE- parentobject.SCORE.min) /
  (parentobject.SCORE.max- parentobject.SCORE.min)

B14 - 'Guessing Penalty' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.correct = 0
parentobject.COUNT.incorrect = 0
parentobject.COUNT.unattempted = 0
for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
  begin
    if child.object(ATTEMPTED) = True 
    then
      begin
        if childobject.CORRECT = True
        then 
          begin
            parentobject.COUNT.correct = 
              parentobject.COUNT .correct + 1
            parentobject.COUNT = parentobject.COUNT + 1
          end
        else 
          begin
            parentobject.COUNT.incorrect = 
              parentobject.COUNT .incorrect + 1
            parentobject.COUNT = parentobject.COUNT - 
                childobject.penaltyvalue
          end
      endif
    else parentobject.COUNT.unattempted = 
      parentobject.COUNT.unattempted + 1
  enddo

B15 - 'Weighted Guessing Penalty' Scoring Algorithm

parentobject.COUNT = 0
parentobject.COUNT.correct = 0
parentobject.COUNT.incorrect = 0
parentobject.COUNT.unattempted = 0
for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
  begin
    if child.object(ATTEMPTED) = True 
    then
      begin
        if childobject.CORRECT = True
        then 
          begin
            parentobject.COUNT.correct = 
              parentobject.COUNT .correct + 1
            parentobject.COUNT = parentobject.COUNT + 
              parentobject.weighting
          end
        else 
          begin
            parentobject.COUNT.incorrect = 
              parentobject.COUNT .incorrect + 1
            parentobject.COUNT = 
              parentobject.COUNT *childobject.weighting- 
              childobject.penaltyvalue
          end
      endif
    else parentobject.COUNT.unattempted = 
      parentobject.COUNT.unattempted + 1
  enddo

Appendix C - Logic Rules

The elements or_objects, and_objects and not_objects are used within the objects_condition element to select the Items and/or Sections whose scoring variables are to be aggregated. An aggregation rule takes the form such as:

<outcomes_metadata mdname="qmd_topic" mdoperator="EQ">trigonometry</outcomes_metadata>

i.e. select all of the objects that have a meta-data entry of "qmd_topic" that is equal to 'trigonometry'.

The objects_condition element will contain a set of rules, i.e. R1, R2, R3...Rk. The logic operators between these rules are defined as:

• ¬R1 means 'NOT R1' which is represented in QTIASI-XML as <not_objects>R1</not_objects>
• R1 and R2 means 'R1 AND R2' which is represented in QTIASI-XML as <and_objects>R1, R2</and_objects>
• R1 or R2 means 'R1 OR R2' which is represented in QTIASI-XML as <or_objects>R1, R2</or_objects>

Using this initial set of constructions gives rise to the following representations:

• ¬ (R1 OR R2) which is represented in QTIASI-XML as

  <not_objects>
    <or_objects>
        R1
        R2
    </or_objects>
    </not_objects>

• ¬ (R1 AND R2) which is represented in QTIASI-XML as

  <not_objects>
    <and_objects>
        R1
        R2
    </and_objects>
  </not_objects>

• (¬R1 AND ¬R2) which is represented in QTIASI-XML as

  <and_objects>
    <not_objects>R1</not_objects>    
    <not_objects>R2</not_objects>
  </and_objects>

• (¬R1 OR ¬R2) which is represented in QTIASI-XML as

  <or_objects>
    <not_objects>R1</not_objects>    
    <not_objects>R2</not_objects>
  </or_objects>

• ((R1 OR R2) AND (R3 OR R4)) which is represented in QTIASI-XML as

  <and_objects>
    <or_objects>
        R1
        R2
    </or_objects>
    <or_objects>
        R3
        R4
    </or_objects>
  </and_objects>

• ((R1 OR R2 OR R3) AND (R4 OR R5 OR (R6 AND R7 AND R8)) AND ¬(R9 OR R10 OR R11)) which is represented in QTIASI-XML as

  <and_objects>
    <or_objects>
        R1
        R2
        R3
    </or_objects>
    <or_objects>
        R4
        R5 
      <and_objects>
          R6
          R7
          R8
      </and_objects>
    </or_objects>
    <or_objects>
      <not_objects>
        <or_objects>
            R9
            R10
            R11
        </or_objects>
      </not_objects>
    </or_objects>
  </and_objects>

This set of rules gives rise to the following guidelines:

• Only a single <and>, <or> or <not> structure is required at the top-most level of the logic statement;
• Multiple occurrences of the <and>, <or> and <not> elements is permitted within <and> and <or> elements to allow the construction of complex logic statements;
• The <not> element can only contain a single element the result of which is to invert the logic of the test.

About This Document

List of Contributors

The following individuals contributed to the development of this document:

Revision History

Index

A
Administrating Authority 1
Administrator 1, 2, 3
ASI 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
Assessment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
Assessment Elements
     assessfeedback 1, 2
     assessment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
Assessor 1, 2, 3
Attributes
     action 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     case 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
     charset 1, 2
     class 1
     columns 1
     continue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     cutvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     defaultval 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47
     feedbacktype 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     ident 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47
     label 1
     linkrefid 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     maxattempts 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38
     maxnumber 1, 2, 3
     maxvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50
     mdname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     mdoperator 1, 2, 3, 4, 5, 6, 7, 8, 9
     members 1, 2, 3, 4, 5
     minnumber 1, 2, 3
     minvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50
     order_type 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     pname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     rcardinality 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     respident 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     rtiming 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     scoremodel 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     shuffle 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     step 1
     testoperator 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     texttype 1, 2
     title 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
     varname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49
     vartype 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36
     view 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     xmllang 1, 2, 3, 4, 5, 6
     xmlspace 1
Author 1, 2, 3
 

B
Basic 1, 2

C
Candidate 1, 2, 3
Common Elements
     conditionvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     decvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53
     displayfeedback 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     fieldentry 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     interpretvar 1, 2, 3, 4, 5, 6, 7
     material 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     mattext 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     objectives 1
     order 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     other 1, 2, 3, 4, 5, 6
     outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81
     outcomes_processing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25
     qticomment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     qtimetadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     qtimetadatafield 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     reference 1
     selection 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28
     selection_ordering 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     setvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     varequal 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     vocabulary 1, 2, 3, 4, 5, 6, 7