Digital Course Ware Evaluation Rubric - Tan

5th National eLearning Conference Taking IT to the Next Level: The Team Approach to eLearning Program Development

A Digital Courseware Evaluation Rubric for CICT Content Development Projects Maria Melizza D. Tan HEA, CICT-HCDG Project Coordinator, eSkwela [email protected]

AGENDA


CICT’s Content Development Strategy

• BECTA Quality Principles


Digital Courseware Evaluation Rubric: the eSkwela Experience

Commission on Information and Communications Technology

HCDG’s Programs and Projects





national ICT competency standards + certifications ICT4E: transformative role of ICT in teaching and learning ICT competency training for government personnel eSociety: digital inclusion + digital communities Commission on Information and Communications Technology

Comprehensive Approach Best Practices + Standards Training

Content

Infrastructure

Applications

Community Involvement Coordination + Ownership + Sustainability

Commission on Information and Communications Technology

constant clamor from the field

Digital Content Development “After the first decade of ICT4D, a major lesson learned by the development assistance sector, particularly the UN agencies and the World Bank, is that investments on content and capability building should be significantly larger than investments on infrastructure (Walsh 2006). According to some estimates, for every dollar invested on ICT4D, ten cents should go to infrastructure. Another ten cents should go to software. Still another ten cents should go to training. But the remaining 70 cents should be spent on content development.”

−

DepEd’s Strategic Framework on ICT4E under BESRA

Commission on Information and Communications Technology

Content Development Strategy


Developing digital content that is appropriate, relevant, accessible, and meaningful to Filipinos is a priority of CICT.

• • • • •


Interactive, engaging, motivating High Quality Well-designed Well-produced Easily replicated at low cost for greater access

Focus Areas:

• •

Educational Multimedia Content Digital Cultural Content Commission on Information and Communications Technology

Content Development Strategy 1. 2.

3.

4.

Mapping the content landscape Developing a Content Development Policy Framework Building the necessary technical architecture for comprehensive national access Training: development & use Commission on Information and Communications Technology

Digital Content Devt phase 1










Mapping activities Multi-media Learning Materials for Teachers (Pre & In) Computer-Based Training Modules (CBTs) for English Proficiency Web Competitions Content and Application Development for Rural Communities and Enterprises: Digitization of Replicable Research Outputs of Partner SUCs









Jeepney ni JenJen –Digitization of Tuklas Sining eHealth: Health information for emergencies website Indigenous People’s (IP) Knowledge skoool.ph eSkwela elearning modules Islam in the Philippines

Commission on Information and Communications Technology

QA: BECTA Principles


British Educational Communications and Technology Agency (BECTA) Quality Principles for Digital Learning Resources 1. Core Pedagogic Principles 2. Core Design Principles

http://partners.becta.org.uk/index.php?section=sa&catcode=_sa_cs_cf_03

Commission on Information and Communications Technology

BECTA Principles


Core Pedagogic Principles 1. Match to the curriculum 2. Inclusion and access 3. Learner engagement, challenge, and motivation 4. Effective learning 5. Assessment to support learning 6. Robust summative assessment 7. Innovative approaches 8. Ease of use




Core Design Principles 1. Digital learning resource design 2. Robustness and support 3. Human-computer interaction 4. Quality and Suitability of assets 5. Accessibility 6. Interoperability 7. Testing and verification 8. Effective communication

Commission on Information and Communications Technology

eSkwela





to use ICTs in broadening access to basic education by tapping into appropriate ICT-supported models to improve the quality of learning and enhancing the quality of teaching in the non-formal education setting target: out-of-school youth and adults

Commission on Information and Communications Technology

eSkwela Infrastructure Deployment

Community Support for Program Sustainability

Materials Development

eSkwela Stakeholders’ Training

Commission on Information and Communications Technology

eSkwela Content Development



roots: Smokey Mountain conversion of existing ALS print modules into digital modules

• Print modules: BALS + SEAMEO-INNOTECH • Curriculum Review done by BALS (2005)

Commission on Information and Communications Technology

eSkwela eSkwela elearning modules (A&E) (beta) eSkwela Learning Management System (alpha) BALS Session Guides & other materials

Learner outputs/ portfolios + Teacher-learner interactions + Teacher-teacher & learner-learner interactions

Involvement of Learning Community + Livelihood Programs Commission on Information and Communications Technology

Module Review for eSkwela


assumptions & constraints:

• content of print modules already reviewed • HCDG-EDO-BALS combined effort: devt of • • • •

version 1 English as medium of instruction review process requirement verbal discussions/agreements: problematic search for existing evaluation form Commission on Information and Communications Technology

Module Review Experience






Rubric: a work-in-progress Briefing and discussion with reviewers/ raters

•

Retooling on ICT-enhanced instructional design

•

Clarifications on item definitions/ descriptions

•

Clarifications on developer limitations and scope of work

• i.e. Session Guides also need to be revised! • Targets: at least 3 per rubric item • Not summed up/averaged

Clearer standards/ expectations for developers

• •

Currently in the module enhancement stage (1st 20) Due to time constraints: “minimums” were set; better to have priority levels for detailed recommendations Commission on Information and Communications Technology

Evaluation Rubric


Assumptions:

• •


Acknowledgements:

•


The learners are not impaired in any way (i.e. not disabled or psychologically dysfunctional). The modules are always accessible to the learners. Some rubric definitions were taken from publiclyavailable rubric-makers on Writing and Web Design, located at Rubistar (http://rubistar.4teachers.org).

Initial Review: Dr. Lolit Suplido-Westergaard

Commission on Information and Communications Technology

Evaluation Rubric: Content

Matched to / Appropriate for the Curriculum

4

3

2

1

The entire module is clearly matched to /appropriate to set curricular standards.

Majority of the module content is matched to / appropriate to set curricular standards but may have one or two elements that are not appropriate.

Module content is loosely matched to set curricular standards.

Module content does not match the set curricular standards.

Commission on Information and Communications Technology

Evaluation Rubric: Content

Focus

4

3

The module has a well-stated purpose and theme that is maintained consistently throughout the site.

The module has a clearly stated purpose and theme, but may have one or two elements that do not seem to be related to it.

2 The purpose and theme of the module is somewhat muddy or vague.

Commission on Information and Communications Technology

1 The module lacks a purpose and theme.

Evaluation Rubric: Content

Organization

4

3

Content is well organized using headings or bulleted lists to group related material. Subtopics naturally progress into succeeding ones.

Uses headings or bulleted lists to organize, but the overall organization of topics appears flawed.

2 Content is logically organized for the most part.

Commission on Information and Communications Technology

1 There was no clear or logical organizational structure.

Evaluation Rubric: Content

Thoroughness and Accuracy

4

3

2

Covers topic in-depth with details, extra information, and examples. Subject knowledge is excellent – no factual errors detected.

Includes essential knowledge about the topic. Subject knowledge appears to be good – no factual errors detected.

Includes essential information about the topic but there are 12 factual errors.

Commission on Information and Communications Technology

1 Content is minimal OR there are several factual errors.

Evaluation Rubric: Content

Interest

4

3

2

1

The author has made an exceptional attempt to make the content of this module interesting to the people for whom it is intended.

The author has tried to make the content of this module interesting to the people for whom it is intended.

The author has put lots of information in the module but there is little evidence that the person tried to present the information in an interesting way.

The author has provided only the minimum amount of information and has not transformed the information to make it more interesting to the audience (e.g., has only provided a list of links to the content of others).

Commission on Information and Communications Technology

Evaluation Rubric: Content

Quality of Assessment

4

3

2

1

Forms of assessment (formative and summative) are varied, effective and appropriate to the content.

Forms of assessment (formative and summative) are limited, but effective and appropriate to the content.

Forms of assessment (formative and summative) are few, barely effective or inappropriate to the content.

There is only one form of assessment, and it is ineffective or inappropriate to the content.

Commission on Information and Communications Technology

Evaluation Rubric: Content

Effectiveness

4

3

2

Module consistently makes use of ICT effectively vis-à-vis content to enhance learning. It is a highly effective elearning module.

Module generally makes use of ICT effectively vis-àvis content to enhance learning but is lacking one or two key elements. It is an adequate elearning module.

Module is missing more than two key elements in its use of ICT vis-àvis content to enhance learning. It would make an incomplete elearning module.

Commission on Information and Communications Technology

1 Module is lacking several key elements and has inaccuracies that make it a poor elearning module.

Evaluation Rubric: Content

Sources

4

3

2

1

Source information collected for all graphics, facts, and quotes. All documented in the desired format.

Source information collected for all graphics, facts, and quotes. Most documented in the desired format.

Source information collected for graphics, facts, and quotes, but not documented in the desired format.

Very little or no source information was collected.

Commission on Information and Communications Technology

Evaluation Rubric: Content

Copyright

4

3

2

1

Fair use guidelines are followed with clear, easy-tolocate and accurate citations for all borrowed material. No material is included from sources that state that publication is not allowed unless permission has been obtained from copyright owner.

Fair use guidelines are followed with clear, easy-tolocate and accurate citations for almost all borrowed material. No material is included from sources that state that publication is not allowed unless permission has been obtained from copyright owner.

Fair use guidelines are followed with clear, easy-to-locate and accurate citations for most borrowed material. No material is included from sources that state that publication is not allowed unless permission has been obtained from copyright owner.

Borrowed materials are not properly documented OR material was published without obtaining permission from copyright owner.

Commission on Information and Communications Technology

Evaluation Rubric: Design

Aesthetics

4

3

2

1

Makes excellent use of font, color, graphics, video, animation, etc. to enhance the presentation.

Makes good use of font, color, graphics, video, animation, etc. to enhance to presentation.

Makes use of font, color, graphics, video, animation, etc. but occasionally these detract from the presentation content.

Use of font, color, graphics, video, animation, etc. but these often distract from the presentation content.

Commission on Information and Communications Technology

Evaluation Rubric: Design

Multimedia (Graphics, Sound, Video, etc.)

4

3

2

1

Multimedia elements are relevant and appropriate to the module, are of high quality and enhance the learner's interest or understanding.

Multimedia elements are related and appropriate to the module, are of good quality and enhance the learner's interest or understanding.

Multimedia elements are relevant and appropriate to the module, are of good quality.

Multimedia elements seem to be used randomly, are of poor quality or distract the learner from the content.

Commission on Information and Communications Technology

Evaluation Rubric: Design

Interactivity

4

3

2

1

Module allows learner a high degree of interactivity by manipulating objects in animations/ simulations, models, data sets, or online assessment. Module provides immediate feedback.

Module allows an acceptable degree of interactivity by manipulating objects in animations/ simulations, models, data sets, or online assessment. Module provides immediate feedback.

Module allows a minimum degree of interactivity by manipulating objects in animations/ simulations, models, data sets, or online assessment.

Module does not contain any form of learner interactivity nor features for immediate feedback.

Commission on Information and Communications Technology

Evaluation Rubric: Design 4

Navigation

All buttons and links are clearly labeled, consistently placed, and work properly. A user does not become lost.

3

2

1

Almost all (90%99%) links are clearly labeled, consistently placed, and work properly. A user rarely becomes lost.

Most links (75%-89%) links are clearly labeled, consistently placed, and work properly. Some needed links seem to be missing. A user sometimes gets lost.

Less than 75%of the links are clearly labeled, consistently placed, work properly. A user typically feels lost.

Commission on Information and Communications Technology

Evaluation Rubric: Design

User Support

4

3

2

Help and/or FAQ is easily accessible, understandable for the users and comprehensive.

Help and/ or FAQ is accessible, understandable and mostly comprehensive .

Help and/or FAQ is hard to find. Content is hardly understandabl e or obviously lacking.

Commission on Information and Communications Technology

1 No Help or FAQ function.

http://balsontheweb.wikispaces.com

5th National eLearning Conference Taking IT to the Next Level: The Team Approach to eLearning Program Development

Thank you.

[email protected]