Primer on 1EdTech Assessment Conformance and Certification

 

Version 1.0

Date Issued:            1 May 2012

 

IPR and Distribution Notices

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

1EdTech takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on 1EdTech’s procedures with respect to rights in 1EdTech specifications can be found at the 1EdTech Intellectual Property Rights web page: /imsipr_policyFinal.pdf.

Copyright © 2012 1EdTech Consortium. All Rights Reserved.

Use of this specification to develop products or services is governed by the license with 1EdTech found on the 1EdTech website: /speclicense.html.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by 1EdTech or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

Public contributions, comments and questions can be posted here: APIP Forum.

 

1                  Introduction

1EdTech is establishing well-defined conformance certification programs for Question and Test Interoperability (QTI) v2.1 and Accessible Portable Item Protocol (APIP) v1.0 to ensure high levels of interoperability.  The conformance certification programs test for the ability to support different levels of features.

The purpose of this document is to introduce the specific areas of conformance available to APIP and QTI and to describe the procedures for achieving certification. APIP and QTI are standards created and governed by 1EdTech Consortium (1EdTech), a non-profit learning standards organization. This document is intended to be used, for example, by vendors that wish to certify their systems as officially conforming to APIP v1.0 and QTI v2.1 as supported by 1EdTech. Ultimately, certification of systems will help ensure that diverse end-users (students) receive access to computer-based assessments.

1.1            1EdTech APIP

1EdTech APIP is a technical standard for digital interchange of assessment content (items and tests). APIP also describes digital delivery options for a range of common accessibility needs. Use of APIP enables assessment materials to be exchanged digitally among a wide variety of products, such as item/test authoring products, item banks, and test delivery systems.

APIP addresses two long-standing needs in assessments. First, it allows for the transfer of content between vendors using a standardized XML format. Second, it supplies the necessary accessibility information in that content to support the needs of all students. Using 1EdTech Access For All Personal Needs and Preferences (PNP) profile information, the content can be tailored to meet the access needs of each student.

The APIP standard defines an assessment file exchange that serves all students, not only students who have disabilities or are English language learners. In fact, the default content of the question (item) is based on a well established 1EdTech QTI standard that has been used internationally for over a decade. APIP uses a new profile of QTI v2.1.

Figure 1.1 illustrates the basic idea of APIP. Authors create content, providing extra information when needed to make that content accessible to all students. An assessment rostering system can provide the specific needs of individual students; these are documented in the PNP profile. An APIP certified delivery system takes the needs of a specific student, and ensures that the content is provided to the student with all their needed access supports.

 

Figure 1.1 The Basic Concept of APIP.

The three main areas of APIP are:

·         Content: The accessible content, which has supporting information for different kinds of accessibility needs

·         PNP: The user Personal Needs & Preferences profile, which tells the delivery system what the needs are for specific examinees

·         Delivery: The delivery system, which combines the user needs with the content to ensure the item is accessible for the examinee

1.2            1EdTech QTI

The QTI v2.1 specification describes a data model for the representation of question (assessmentItem) and test (assessmentTest) data and their corresponding results reports. The specification enables the exchange of item, test, and results data between authoring tools, item banks, test constructional tools, learning systems, and assessment delivery systems. The specification has been designed to support both interoperability and innovation through the provision of well-defined extension points.

1.3            1EdTech Assessment Community

To join the 1EdTech developer and conformance certification community focused on APIP and QTI v2.1 please visit the 1EdTech QTI2/APIP Alliance online here: /developers/apipalliance/index.cfm

Public contributions, comments and questions can be posted here: APIP Forum.

2                  Conformance and Certification Overview

2.1            Relationship between APIP and QTI

APIP uses a precisely defined subset of 1EdTech QTI v2.1 with additional features included in some other 1EdTech specifications. The first step in implementing APIP is to implement the APIP defined subset of QTI v2.1. This will allow assessment suppliers to utilize QTI when accessibility is not a requirement and to receive a QTI conformance certification.

APIP and the profiles of QTI used in APIP will stay synchronized by 1EdTech in conjunction with 1EdTech member organizations through the QTI/APIP Accredited Profile Management Group (APMG).  The QTI subset within APIP, such as interaction types (question types), is expected to grow over time as the marketplace for digital assessment evolves.

The additional features of APIP beyond QTI are accessibility metadata (based on 1EdTech Access For All), accessible delivery options (developed on the original APIP grant program) and curriculum standards metadata (based on 1EdTech Content Packaging/Common Cartridge). These features are defined in the APIP specification.

2.2            Conformance Certifications

1EdTech is establishing well-defined conformance certifications for QTI v2.1 and APIP v1.0 to ensure high levels of interoperability.  The conformance certification process involves testing for the ability to support different levels of features. The conformance details are described in Section 3.

2.3            How Conformance Certification is Achieved and Recorded

Conformance certification is granted by passing a series of tests, submitting the results to 1EdTech and agreeing to become part of the support community (the QTI/APIP Alliance /apip/index.html). 1EdTech records all certifications on a public website: Conformance Certification Status. At the time conformance is certified, 1EdTech issues a unique registration number designator for the product name and version for use in proposals. Buyers and users are encouraged to visit the certification website to find certified products and to log any issues with certified products.

The community aspect of conformance is critical to address any issues that may arise in the marketplace as well as the evolution to new versions.  1EdTech requires suppliers to participate in resolving issues that may arise, as well as helping to guide and implement each incremental revision to the conformance certifications.

2.4            Achieving Conformance Certification

The tests required for certification are different depending on the particular QTI/APIP certification sought. The tests are also different for content/authoring tools versus delivery platforms. This is because content/authoring tools are tested to ensure that the content produced and exported is conformant within the scope of item types and tests that a particular tool can generate, while delivery systems are tested to ensure they can appropriately process a range of content alternatives. Conformant delivery platforms must import conformant content and provide conformant results. Conformant authoring tools must export conformant content and may or may not import conformant content, depending on the purpose of the tool.

 

3                  Assessment Conformance Statements

The following conformance matrix lists the current profiles available for QTI and APIP certification as they relate to these categories:
 

QTI Delivery System & Authoring System	The QTI conformance criteria for Delivery Systems and Authoring Systems.
APIP  Authoring System	The APIP conformance criteria for Authoring Systems and APIP packages.
APIP Delivery System	The APIP conformance criteria for Delivery Systems.
APIP Personal Needs & Preferences (PNP) Systems	The APIP PNP conformance criteria for systems supplying or processing PNP instances.

3.1            Conformance Profiles

Named profiles currently defined for conformance are described here as they appear in the matrix below.
 

QTIv2.1 Summative Entry	Conformance for Delivery Systems that process Tests used for 'Summative' assessment (this is equivalent to APIP Entry minus the accessibility features).
QTIv2.1 Formative Entry*	Conformance for Delivery Systems that process Tests based upon QTIv2.1 (this is equivalent to the Common Cartridge profile for QTIv2.1).
QTIv2.1 Core	Conformance for Delivery Systems that process Tests based upon QTIv2.1 (this is equivalent to the Common Cartridge profile but with all of the prohibited features made Elective).
APIP Entry	The entry level APIP conformance (this is equivalent to the Common Cartridge profile). Entry level conformance also permits the set of implementation Electives.
APIP Core	This is the level of conformance that 1EdTech recommends should be adopted by APIP systems.
APIP PNP Entry	This is the set of entry level APIP PNP requirements for a Preferences System producing or processing PNP instances.
APIP PNP Core	This is the level of conformance to which 1EdTech recommends that all APIP PNP systems should aspire.

 

Click here to go to the most up to date version of the QTI/APIP Conformance Matrix: QTI/APIP Conformance Matrix.

Elective access features are those features that could be supported by systems, but are not required for conformance. Elective features will be individually certified, and systems that use elective features must list the specific APIP Elective access features they support. Either Entry or Core certified systems are eligible to be certified in any elective feature. There is no Elective certification without Entry or Core certification.

The certification categories include a differentiation between the ability to import and export APIP Content or PNP files. Since APIP is primarily a transfer format, it will not certify how systems manage or modify data within the system itself. Delivery Systems are expected to combine the information provided by APIP content packages and PNP files. Although they do not need to natively use the APIP format during delivery, certified delivery systems must make use of the data/information supplied by APIP.

Systems that import APIP information must distinguish between valid APIP formats and invalid formats, and provide information about which packages are invalid. Systems will minimally support APIP Entry access features, and possibly support APIP Core and APIP Elective access features.

Systems that export APIP content or PNP files must be able to supply valid APIP formatted instances, minimally supporting APIP Entry access features, and possibly supporting APIP Core and APIP Elective access features.

 

4                  Achieving Certification

Validation for APIP and QTI conformance will consist of the following strategies:

·         Automated Validation. An electronic process to verify conformance using automated tools provided by the 1EdTech APMG, and

·         Test Case Based Validation. Using a standard set of APIP/QTI content packages and PNP files with known characteristics provided by the 1EdTech APMG that can be used as part of a validation process and have known expected results that can be documented and/or demonstrated, and

·         Documented or Demonstrable Validation. Documented statements and/or demonstrable capabilities showing conformance that can be reviewed by the 1EdTech APMG or by the user.

Because the APIP standard is primarily a data transfer format, conformance and certification does not govern how applications manage or modify data within the system itself. For example, delivery systems must be able to combine the information provided by imported content and PNP files, but need not natively use the APIP format during delivery. There are limits to what can be verified in an automated fashion. In the categories listed in the prior section as an example, it makes a clear distinction between import and export. The reason for this is that export functions are much more easily verified using automation. The output from an export process must adhere to the APIP XML specifications and can be verified using automation. Import, on the other hand, is much more difficult to verify because the import process results in a proprietary system consuming the information and representing it in an internal format that does not need to adhere to the XML specification and cannot be verified using automation.  Each of these approaches is discussed in detail below.

4.1            Automated Validation

Capabilities to be validated using automated tools:

·         APIP/QTI Content Package export

·         APIP PNP file export

For the export functional context, the APIPv1 conformance approach focuses on determining if the system can generate an object (PNP file or item) that is non-problematic. 1EdTech provides an online tool available for individuals or organization to submit an object (PNP file, item) and then receive information about the existence (and sometimes nature) of problems with syntax and semantics of the object.

It is expected that the automated validation tool will:

·         Accept Entry, Core, and Elective APIP/QTI conformance content elements identified in the conformance matrix.

·         Accept custom (and profiled) extensions that are agreed to between exchanging parties (interdependent vendors).

·         Identify which Core and Elective access features as well as the QTI v2.1 features are implemented in the supplied instance.

4.2            Test Case Based Validation

Capabilities to be validated using a standard set of test cases:

·         APIP/QTI Content Package import

·         APIP PNP file import

The QTI/APIP Alliance makes available sets of objects (PNP files, items) — some that are problematic and others that are non-problematic — for conducting import validation. Import functionality focuses on determining if the system can (a) flag as problematic objects that are known to have problems in syntax or semantics and (b) recognize as non-problematic the objects that are known to be without problems.

4.3            Documented or Demonstrable Validation

Capabilities to be documented and/or demonstrated by the solution provider:

• Render APIP content, supporting differences in PNP file information

As each test delivery platform may provide a unique user experience and use different technologies to deliver to content, providing automated validation of each implementation is virtually impossible. Therefore, the 1EdTech APMG will rely on the solution provider to document that the required APIP content and PNP features have been implemented and can document and/or demonstrate them based on user expectations. The documents and demo systems can be submitted to the APMG for consideration and review but the APMG will not be able verify the exact system behaviors.

 

About this Document

Title:                                       Primer on 1EdTech Assessment Conformance and Certification

Editors:                                  Colin Smythe (1EdTech) and Mark McKell (1EdTech)

Version:                                  1.0

Version Date:                         1 May 2012

Release:                                  Draft 1.0

Status:                                    Draft

Summary:                              This document introduces the different areas of conformance available to APIP and QTI and describes the procedures for achieving certification.

Revision Information:           First release

Document Location:            /apip/IMSAssessmentPrimerv1p0.html

 

 

 

 

1EdTech Consortium, Inc. (“1EdTech”) is publishing the information contained in this document (“Specification”) for purposes of scientific, experimental, and scholarly collaboration only.

1EdTech makes no warranty or representation regarding the accuracy or completeness of the Specification.

This material is provided on an “As Is” and “As Available” basis.

The Specification is at all times subject to change and revision without notice. 

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

1EdTech would appreciate receiving your comments and suggestions.

Please contact 1EdTech through our website at imsglobal.org.

Please refer to Document Name: Primer on 1EdTech Assessment Conformance and Certification v1.0

Date: 1 May 2012