Online Assessment:
Principles and Practices
Online Assessment:
Continuous and Interactive
The Role of the
Instructor
Types of Online
Assessment
Adult, Career, Voc Ed
Resources
Distance Assessment
Principles
Web Resources
Bibliography
More and more teaching and learning are taking place at a distance, especially on the World Wide Web. How do educators measure what distance learners know? This issue of the Practitioner File looks at issues and techniques related to the assessment of online learning for adult, career, and vocational education. We address assessment in the sense of obtaining and interpreting information about what has been learned, rather than evaluating web-based courses and instruction (although we found that the literature tilts more toward course evaluation than learner assessment). Inside you will find discussions of the principles, advantages, and challenges of online assessment; the concepts of continuous assessment and interactivity; the role of the instructor; and descriptions of types, tools, techniques, and resources.
Learner assessment has a foundation of good practice whether it takes place online or face to face (Dirks 1997; Zvacek 1999). Assessment is usually intended to provide both instructors and learners with information on progress and to measure achievement of learning goals. Formative assessments provide ongoing feedback to improve instruction and learning. Summative assessments are made to assign value to what has been learned (Australian National Training Authority 1999; Hopper 1998). Principles of good assessment include validity (does it measure what is intended?), reliability (does it consistently produce the same information?), flexibility (are various methods and approaches used to accommodate diverse needs?), and fairness (is it free of biases?) (ANTA 1999; Juchnowski and Atkins 1999). The quality of test construction matters. Because assessment has multiple purposes, multiple methods are needed.
These principles apply to assessment regardless of the delivery mechanism. There are, of course, differences between online and face-to-face situations. Some of the disadvantages and advantages to conducting assessment at a distance are discussed here, concluding with some principles and practices for online assessment.
"Assessment and measurement become even more critical in the absence of the face-to-face interactions that enable teachers to use informal observation to gauge student response, obtain feedback, and progress toward goals" (Pennsylvania State University 1998, n.p.). This lack of nonverbal cues poses the greatest challenge to online assessment. Other disadvantages include the following ("Assessing Students" 1999; Juchnowski and Atkins 1999)
• Learner isolation and the impersonality of computerized assessment
• Lack of instructor control over assessment conditions
• Uneven access of learners to technology and resources
• Technical problems associated with access—e.g., media-rich learning activities such as video segments may be expensive to produce and slow to download; failed hardware or phoneline outages may prevent access
• Learners’ and instructors' technical skill levels
• Learner anxiety about being disadvantaged by difficulties with the technology
• Instructor time to score tests and provide feedback
• Fatiguelengthy assessment activities may stress the eyes from the use of computer screens
Two issues that may create particular barriers to online assessment are security and the need for programming.
Ensuring that the individuals completing the assessment are who they say are (authentication) “is an issue in all examinations…However, in a computer environment, impersonation may be perceived as a greater risk” (Mogey and Watt 1999, n.p.). As King (1998) notes, “no existing technology can ensure academic honesty” (p. 27). However, some suggested security measures include the following (Dickinson 1997; Gray 1997; Jarmon 1999; King 1998; Zirkle and Ourand 1999):
• Holding proctored examinations at remote sites
• Minimizing objective tests and focusing on methods calling for analysis and application (essays, case studies, etc.)
• Constraining access to the website of a test through pass- words and usernames
• Embedding quizzes into exercises and readings
• Randomizing questions and the order in which they are presented
• Carefully designing Web-based instruction to convey outcomes the learner perceives as useful and desirable
• Considering online exams in the same way as take-home tests. As Nelson (1998) points out, "they will never be asked on the job to sit away from all reference materi- als, forced to solve problems and communicate in isolation" (p. 2)
• Using multiple sources of assessment data
According to Gray (1997), the potential of the Web to create a virtual community can give instructors additional "reality checks": Through e-mail and web-based discussions, instructors become familiar with the writing styles and abilities of individual students, so that an exam written in a radically different style would be a red flag. "Integrating collaborative elements such as joint projects, synchronous and asynchronous discussion forums, and workgroup computing can further reduce the possibility of misrepresentation and increase student accountability" (ibid., n.p.).
Instructors’ lack of programming skill as well as the time involved can be major barriers to online assessment. Some commercial software allows customization without requiring programming, but it is often platform specific or requires certain applications or players (Gray 1999). Web editing software such as HotMetal or FrontPage can be used to create HTML forms and tests. A number of resources listed on page 10 provide free examples of JavaScript and Common Gateway Interface (CGI) programs, and the articles in the bibliography by Dickinson (1997), Doulai and Stace (1998), and Gray (1999) give advice for incorporating these tools in online assessment. Hayes and Huckstadt (2000) describe the use of Java scriptlets and FrontPage to create online continuing education modules for nurse practitioner self-study. Examples are provided at http://www.nursing.twsu.edu/clp.
Online assessment offers a number of advantages ("Assessing Students" 1999; Juchnowski and Atkins 1999; Ravitz 1998):
• Ease of distribution
• Timeliness (when the assessment is available and how long it is available)
• Provision of feedbackscores and explanations may be given immediately; e-mail allows for both individual and group feedback; learner feedback to the instructor can be incorporated immediately into the course
• Links to tracking and management systems
• Interactivity (see next article)
“If WWW-based learning systems simply mirror the pragmatic weaknesses in the assessment process practiced by educators in traditional settings, there will be a lack of emphasis on the special advantages that the computer can provide” (Hopper 1998, p. 332). Assessment at a distance should take advantage of the best features of the online environment. Nelson (1998) describes the Web's potential for real-world, authentic performance assessments. Web-based assessment enables true scaffolding (Hazari and Schnorr 1999); for example, an instructor can develop critical thinking questions based on the content and tenor of online discussion, rather than having a predetermined set of questions. The online environment enables learners to regulate and monitor their own learning (Hazari and Schnorr 1999; Ravitz 1998). Informal quizzes can give learners instant feedback to gauge their understanding of a topic; hyperlinks can lead learners to sources that explain incorrect answers. “With its flexible participation structures, the Internet allows learners to pace their own activities, share information at opportune times, receive feedback and interact with others, all while building a permanent record of their activities and what they have learned” (Ravitz 1998, n.p.).
The unique characteristics of the Web as a hypermedium give online learning environments the potential to facilitate dialoguebetween learners and materials, their instructors, and fellow learners (Wild and Omari 1996). The “learning community” model enables learners to conduct collaborative projects, engage in teamwork, and adopt a variety of rolesall aspects of the contemporary workplace (Marshall 2000). Assessments of qualitative aspects such as coherence and comprehensiveness of projects, team functioning, and interpersonal skills can thus reflect how learners will be evaluated in the world of work.
Integration of content, online tasks, and assessment is an essential element. Hedberg and Corrent-Agostinho (2000) describe how Web-based group project tasks were completed at various intervals during a course rather than at the end of semester. The tasks thus became resources that learners accessed to assist them with the final group project.
Communication works somewhat differently in web-based training and distance education than in the traditional classroom. In the traditional classroom, face-to-face communication allows all parties to send and receive a full range of verbal messages, both written and oral, as well as nonverbal messages like facial expression, tone of voice, and inflection that give additional information. In particular, traditional classroom instructors can monitor nonverbal communication as a check on learners' comprehension; a puzzled look or a pause in notetaking can indicate that learners are not comprehending (Champagne 1998).
Online communication using computer-mediated communication (CMC) mechanisms such as e-mail and computer conferencing is different in two important respects, however (Bigelow 1999). First, online communication is not as rich; it's usually only written text with static graphic images (if any). Since instructors cannot observe learners, there are no nonverbal messages indicating comprehension or lack thereof. Second, online communication is typically slower because you can't type as fast as you can talk. The extra time allowed by that slower rate of communication can lead to more thoughtful interactions, but it can also have an adverse effect on the interpersonal interaction.
Those important differences in communication mean that in distance education, assessment for formative purposes becomes much more important. Since online instructors and learners cannot communicate as quickly, easily, and fully, instructors need to take extra steps to monitor learners' performance and comprehension (Lindner 1998). Continuous assessment allows instructors to familiarize themselves with each learner's work repeatedly to help ensure that understanding occurs (Schrum 1998). In other words, since online instructors can't see their learners, can't observe nonverbal clues, and can't ask quick, impromptu questions to check comprehension, instructors must use other means to ensure that learners are on the right track (Thorpe 1998).
Careful, Systematic, and Planned
The results of continuous assessment can certainly be graded and included in determining a final course grade, but the purpose is more formative than summative. So to be effective, continuous assessment must be more than just a final add-on to instruction for determining a grade; it must be planned carefully and systematically as an ongoing part of instruction. Planning for continuous assessment should take three key issues into account (ibid.):
• Pacing. Continuous assessment should require that learners demonstrate and use what they have learned by completing an independent piece of work (alone or in groups) at a specified point in instruction. The mechanistic constraint of pacing helps learners manage and schedule their own learning, while it also enables constructivist learning by providing scaffolding that helps learners use what they have already learned to construct a coherent understanding of new information.
• Feedback. Instructors provide feedback on the quality of learners' understanding. Again, feedback provides expert scaffolding to help learners construct understanding; timely, critical feedback helps learners improve and refine their early understanding. In addition, positive feedback provides experiences of success that motivate learners to continue, overcoming obstacles if necessary.
• Learning quality. Both pacing and feedback directly affect whether learners study and learn and how effectively they do so. The assigned activities determine what learners learn by directing them to specific content areas and by requiring specific skills (e.g., problem solving, calculation, analysis, synthesis, communication).
Kinds of Activities
A range of activities can be used for continuous assessment to help develop learning, as long as assessment activities are closely aligned and interwoven with study materials (Morgan and O’Reilly 1999):
• Ungraded activities and feedback built into study materials
• Self-assessment quizzes and tests that allow learners to check their own learning
• Formal feedback on assignments from instructors, peers, or workplace colleagues or mentors
• Informal dialogue with instructors, peers, or others
• Ungraded tests that prepare learners for formal, graded assessment
Assignments and informal dialogue offer many opportunities for continuous assessment in particular. For example, learners can be required to participate in online CMC conferences; they might be required to submit a minimum number of messages contributing to the discussion and receive instructor feedback on the quality of those messages; or they might receive feedback on the quality of specific contributions they submit for feedback (Thorpe 1998). Assigned activities and dialogue can also involve group activities or projects in which learners collaborate to solve a problem, create a simulation, design a product, or complete a task (Schrum 1998). Of course, one critical criterion in identifying assessment activities is that they must relate to the learning objectives you have established (Jarmon 1999).
Providing Feedback
The feedback instructors give in continuous assessment should provide the scaffolding that allows learners' understanding to grow and evolve. It should be full and specific enough to tell learners clearly when and how they are on track or off trackfor example, “Good job identifying the primary differences… [between two theories]. Your response is right on target. Should this be an exam question, you might want to elaborate a bit on each difference (e.g., give an example of each of these differences related to a teaching strategy)” (Hazari and Schnorr 1999, p. 36). Inexperienced online learners may be particularly reassured by instructor feedback that confirms they’re on track and doing well (Thorpe 1998).
Feedback should be timely and prompt; full, specific feedback is most useful when learners receive it in time to use it effectively. Furthermore, learners are most likely to use feedback productively when they are oriented beforehand to its purpose and significance, and when feedback is provided in a consistent format (e.g., separate comments on structure and format of learner work; sequence of comments from the general to the specific) (Mann 1998). Finally, feedbackparticularly on graded assessmentsshould always be based on consistent, clearly communicated standards, or rubrics (Nelson 1998).
Online assessment in distance education can make use of Internet technologies that allow greatly increased communication, interactivity, and collaboration compared to the traditional classroom or paper-and-pencil forms of distance education. CMC allows synchronous and asynchronous, one-to-one, one-to-many, and many-to-many communication; its use of text-based communication promotes thoughtful and reflective commentary (Schrum 1998). The interactivity possible in online continuous assessment, either with the online environment or with instructors or other learners, helps engage learners in active application of knowledge, principles, and values and provides them with feedback that allows their understanding to develop (Hazari and Schnorr 1999). Hence, online assessment can play an important part in a constructivist learning experience.
In addition, online assessment can take advantage of the capabilities of other software applications to develop interactive online tests with a wide array of functions (Dickinson 1997):
• Record answers to a file the instructor can review later.
• Compare learner answers with correct answers and give feedback.
• Grade the test and give feedback.
• Recommend remediation.
• Show the learner a web page with more information on a missed question.
• Prevent the learner from proceeding to new instruction until the learner passes the test.
• Control retriesallow any number or a restricted number only.
• Require the learner to enter a name and controlled password.
• Restrict the dates on which learners can take the test, avoiding last-minute test-taking.
• Time the test.
• E-mail test results to the instructor.
Developers can create customized online tests from scratch using programming languages such as Common Gateway Interface (CGI) (ibid.), Active Server Pages (ASP), and JavaScript (Doulai and Stace 1998; Gray 1999) or commercially available packages such as Toolbook II, Macromedia Authorware, Jamba, or CGI*Star (Dickinson 1997). Easiest to use are commercially available template-based instructional management systems, for example, WebCT, Blackboard, TopClass, and Web-Course-in-a-Box (Bigelow 1999). Interactive tests can include multiple choice, true/false, matching, and even essay questions (ibid.), although scoring essay questions is difficult for anything beyond merely looking for specific words or sequences of words in learner answers (Dickinson 1997).
A key factor that distinguishes distance education from the traditional classroom is the dramatic shift in the role of the instructor from being the authoritative source of knowledge to being a facilitator of learner-centered, collaborative, and egalitarian learning (Hutton 1999). Instead, instructors can take advantage of the potentially constructivist online environment to build in opportunities for creativity, so that learners can relate learning to their experience and local context. The governing principle is that learning is an active process in which learners construct rather than acquire knowledge, and the instructor’s role is to support that construction rather than communicate knowledge (Milman 1999).
Being a Mentor, Coach, and Facilitator
That constructivist perspective also applies to the instructor’s role in online assessment. First, instructors may find that they do less questioning, probing, and focusing in continuous assessment. In the online environment, the capabilities of CMC allow learners to take a more active and facilitative role. The instructor assumes a more social role: forming a supportive learning environment, promoting human relationships, maintaining the group as a unit, and helping members work together in a mutual cause (Hutton 1999). In addition, distance education students are often typical adult learnersself-directed, motivated by personal learning goals, willing and able to apply their own experience to learning content (Tillson et al. 1998). As a result, the instructor's role as mentor, coach, and facilitator helps encourage connections and learning made possible by carefully, systematically planned continuous assessment (Elliott 1999).
Furthermore, instructors are often the first point of contact for distance education learners when they need administrative, technological, or affective support (ibid.). Instructors must be able to provide referrals and linkages to technology services, support services, and other institutional resources. Inexperienced learners often rely on assistance from instructors to acquire the technology skills needed to conduct a web search or participate in a chatroom discussion (ibid.). And prompt intervention with affective support and even tips on stress management is well appreciated by distance learners who are uncertain how to proceed.
Communicating Information
In the traditional classroom, instructors are used to providing direction, guidance, and feedback orally as a part of assessment. In online assessment, however, instructors must provide that same direction, guidance, and feedback in writing; thus, instructors must communicate clearly and effectively in a different medium (ibid.). Instructors are likely to find that writing takes more time than talking, and they may also find that, like learners (Hutton 1999), they want to take more time and be more thoughtful when communicating in writing.
One key factor in communicating effectively in writing is the content of feedback provided to distance learners. The content of feedback needed can range from a simple “Yes, that's correct” to a much more complex presentation in a whole new way to aid understanding (Gibson et al. 1995) Similarly, distance learners have a particular need to know clearly and specifically what the expectations are and what constitutes successful outcomes; they should know this before they begin the course. Working on the Web may require more detailed descriptions and definitions of success (Hardy 1999). Weight assigned to different factors in graded assessments (e.g., 30 percent for presentation and organization, 70 percent for content and relevance) is especially important (Mann 1998). Learners typically make more positive comments on instructor feedback on actual assignments when that feedback is fuller and more detailed (Thorpe 1998).
Instructors should also remember that online communication involves more than feedback on the details of learning content (ibid.). Online learners also value advice and guidance to help them start and stay on track. Inexperienced learners may especially appreciate confirmation that they are on trackthey may be glad to have that specific reassurance even though their instructor has said that he or she would step in if they got off track.
Providing Timely Feedback
The timeliness of feedback instructors provide is crucial. There is little point in making extensive comments on learner work if the learner doesn’t receive them before going on to the next assignment; the learner needs to be able to incorporate the feedback into the next assignment (Dirks 1998). Learners usually react favorably to frequent contact and feedback from instructors (Tillson et al. 1998), especially in the initial planning stages of project work.
Although the asynchronous nature of CMC is usually one of its most appealing features, learners may benefit from online office hours at specified times so learners can log on and talk with the instructor in a chatroom (Zirkle and Ourand 1999). Instructors can monitor the effectiveness of their online communication by asking learners about their perceptions and assessment of the quality of that feedback (level, structure, content, relevance, usefulness) in order to improve effectiveness (Mann 1998).
Being a Team Member
Instructors must keep in mind that they may be only one member of the distance learning team, which can also include instructional designers, instructional developers, technology supporters, facilities supporters, materials supporters, distant-site facilitators, and management sponsors (Abernathy 1998). As team members, instructors often find that far more time and attention to detail are required than for face-to-face courses. Teamwork and cooperation among instructors, technical media specialists, and administrators are critical. Instructors may need to complete course plans and materials well in advance so that other team members can do their part (Tillson et al. 1998).
Equally, instructors often receive significant support and assistance from other team members, particularly technology staff (McDougall, Place, and Currie 1999). For example, technology support staff may provide online management systems designed to enable instructors with little or no computer experience to create online tests with multiple choice, short answer, and essay questions incorporating formatted text, images, video, and audio both for learner self-assessment and graded assessment. Systems might randomly create tests from item data banks, grade the results, and post them both to learners and instructors.
Such sophisticated online systems often make their own demands on instructors. If an online system creates randomized tests using an item bank, instructors may need to write additional test items from which to draw. Likewise, if the online system provides content-related hints for learners during the test, instructors need to provide that content information to technology staff. Finally, instructors may need to supply detailed content information in the forms of hints (received by learners during the test, based on their answers to guide them in the correct direction) and key words or other forms of expert answers (Kennedy and Eizenberg 1998).
Institutional technology support might offer professional development opportunities like training sessions in specific technologies, long- or short-term programs that provide the time and resources for instructors to acquire technology skills, targeted workshops and demonstrations, and one-on-one consultation. Assistance may be available not only from full-time technical experts but also from part-time student staff who are familiar with software applications. Instructors can also develop skills by sharing experiences, ideas, and discoveries with one another (Zirkle and Ourand 1999).
A variety of methods may be used online to assess learners in the following areas (Morgan and O'Reilly 1999):
• Critical thinking (essays, reports, reflective journals)
• Problem solving (multimedia or text-based scenarios, simulations using CD-ROM, videoconferencing)
• Demonstrating techniques (videoconferencing, verification by workplace mentor, site monitor)
• Self-management (journal, autobiography, portfolio, learning contract)
• Information access/management (database development, bibliography, problem solving)
• Demonstrating knowledge (written exam with local proctors, quick feedback through multiple choice, true/false matching, short answer tests)
• Designing, creating (portfolios, projects using video or the Web)
• Communicating (debate, role play, PowerPoint presentation, report journal, essay)
• Teamwork and collaboration (e-mail, listserv, or conferencing discussions/debates)
Advantages, disadvantages, and examples of several types are presented in the following chart.
Austin, J. T., and Mahlman, R. A. “Using the Internet for Career-Technical Assessment: A Pilot Project in Ohio.” Journal of Career and Technical Education 16, no. 2 (Spring 2000): 27-37.
Presents results of a study of administrative office technology assessment delivered via the Internet. Discusses strategies for moving career-technical assessment to the Internet.
Australian National Training Authority. “Flexible Learning Toolbox Project.” 1999. http://www.anta.gov.au/toolbox/ <http://www.toolboxcentral.com>
Offers multimedia resources that provide tools for the development, implementation, and evaluation of online vocational training programs. Includes guides for developers, facilitators, project managers, and support staff.
Canale, R., and Duwart, E. “Internet-based Reflective Learning for Cooperative Education Students during Co-op Work Periods.” Journal of Cooperative Education 34, no. 2 (1999): 25-34.
Explains how distance learning links cooperative education students to their institution during work experiences. Describes the use of conferencing software for assignments and reflective discussions.
Gallant, G. M. “Professional Development for Web-based Teaching: Overcoming Innocence and Resistance.” New Directions for Adult and Continuing Education no. 88 (Winter 2000) forthcoming.
Addresses faculty development issues and lists numerous resources to help prepare teachers for web-based instruction.
Hayes, K., and Huckstadt, A. “Developing Interactive Continuing Education on the Web.” Journal of Continuing Education in Nursing 31, no. 5 (September-October 2000): 199-203. Explores the use of Java scriptlets to provide formative evaluation and feedback throughout virtual patient scenarios in online modules for nurse practitioner self-study.
Human, S. E.; Kilbourne, L. M.; Clark, T. D.; Shriberg, A.; and Cunningham, B. “Using Web-Enhanced Instruction in an Interpersonal Skills Course.” Journal of Management Education 23, no. 5 (October 1999): 584-606. (EJ 592 565)
Describes how management students developed criteria they then used to select web-based diagnostic self-assessment tools for a course on interpersonal skills.
Journal of Management Education 23, no. 6 (December 1999). Special Issue: Innovative Uses of Information Technology in Management Education.
Bigelow, J. D. “The Web as an Organizational Behavior Learning Medium.” pp. 635-660. (EJ 596 509)
Addresses the impact of moving learning activities to an online medium, including effects on team projects, role plays, self-assessment, interactive cases and simulations, exams, and feedback.
Chappell, D. S., and Schermerhorn, J. R., Jr. “Using Electronic Student Portfolios in Management Education: A Stakeholder Perspective.” pp. 651-662. (EJ 596 510)
Discusses how a business school is using electronic student portfolios for academic assessment, career development, internship, and job placements. Recommends that they be mandatory, even for students with weaker computer skills, and they should have defined deadlines and feedback mechanisms.
Shrivastava, P. “Management Classes as Online Learning Communities.” pp. 691-702. (EJ 596 512)
Describes the use of eSocrates software for networked communications in the creation of an online learning community in business education.
Meisel, S., and Marx, B. “Screen to Screen versus Face to Face: Experiencing the Differences in Management Education.” pp. 719-731. (EJ 596 514)
Presents an online experiential exercise on power and negotiation to explore differences between online and traditional education.
Niederhauser, V. P.; Bigley, M. B.; Hale, J.; and Harper, D. “Cybercases: An Innovation in Internet Education.” Journal of Nursing Education 38, no. 9 (December 1999): 415-418. (EJ 596 594)
Describes a mix of traditional and technological modes to present online case studies for nurse practitioner continuing education. Addresses security, computer literacy, and evaluation issues.
Schilke, R. A. “Web-based Instruction for Adult Educators—Hi-Touch versus Hi-Tech.” In Proceedings of the 19th Annual Midwest Research to Practice Conference in Adult, Continuing, and Community Education, edited by M. Glowacki-Dudka, pp. 208-212. Madison: University of Wisconsin, 2000.
Asserts that most literature and practice in web-based instruction is behaviorist. Attempts to identify course designs and techniques that are better aligned with adult learning principles and theories.
Zirkle, C., and Ourand, D. B. “Teaching a Course through Multiple Delivery Systems: Some Lessons Learned.” Paper presented at the Indiana Postsecondary Education Conference, Muncie, IN, October 25, 1999. (ED 435 800)
Describes how Indiana State University offers courses for bachelor's and master's degrees in human resources development, technology education, and vocational-technical education using satellite transmission to remote sites, videotape, and the Internet.
Innovations in Distance Education is a Penn State initiative that developed a set of principles and practices to guide the effective use of distance education. The essential principles for assessment of online learning, with examples of ways to put them into practice, are as follows (PSU 1998):
1. Assessment instruments and activities should be congruent with the learning goals and skills required of the learner throughout a distance education program or course.
Representative Practices
a. Design the instructional activities in the same form and methods as will be used to measure mastery of the program goals.
b. At the start of the program, communicate the planned assessment and measurement strategies to the learner. Clearly state the nature, duration, due date, and impact on program grade of all assignments and measurement techniques.
c. Where possible, provide assessment and measurement techniques and options that capitalize on the unique characteristics and situations of the distance learner.
2. Assessment and measurement strategies should be integral parts of the learning experience, enabling learners to assess their progress, to identify areas for review, and to reestablish immediate learning or lesson goals.
Representative Practices
a. Consider a variety of “low-stakes” assessment and measurement strategies that enable learners to gauge their progress without affecting course grade or performance measurement.
b. Create automated systems (e.g., online quiz tools, database programs) that can provide immediate feedback, relevant suggestions, and guided support.
c. Develop techniques and systems that support learner-to-learner interactions for assessment and measurement.
d. Design as many self-check activities as possible within the distance education program, enabling students to adjust their progress within the course.
e. Consider credit for student effort as one component in the assessment process when the skill being taught is complex.
3. Assessment and measurement strategies should accommodate the special needs, characteristics, and situations of the distance learner.
Representative Practices
a. Consider the use of synchronous technologies such as teleconferencing or interactive compressed video for providing learners immediate assessment on work in progress.
b. Consider the use of asynchronous technologies such as e-mail, bulletin boards, voice mail, fax, and/or other technologies to support assessment and measurement activities.
c. Select media carefully when planning an assessment, recognizing that the use of several different media might be appropriate. For example, e-mail might be used to have students describe, online chats or telephone interviews might be used to have students discuss, and videotape might be used to demonstrate oral presentation skills or physical interactions, such as a counseling session.
d. Where possible, use assessment and measurement strategies that use resources local to the distance learner. For example, use projects based on learners' employment as part of the assessment and measurement for the program.
e. If needed, help students make local arrangements to complete course assessments and evaluations.
face="Arial, Helvetica, sans-serif">Reprinted with permission from the Management Team for Innovations in Distance Education, Penn State University
Assessment/Testing/Evaluation Issues (Module 5-Evaluation). http://online.parkland.cc.il.us/ofh/Assessment/Mod5Issues.htm
This checklist of issues that should be considered when assessing online is part of the Online Faculty Handbook from Parkland College's Department of Virtual and Distance Learning.
On-Line Evaluation: Multiple Choice, Discussion Questions, Essay, and Authentic Projects. http://leahi.kcc.hawaii.edu/org/tcon98/paper/nelson.html
Describes an adaptation of traditional assessment to the online environment with specific examples in earth sciences. Emphasis is on authentic assessment of students' communications skills, problem-solving ability, and teamwork.
RMIT University School of Business, Melbourne, Australia. http://www.bf.rmit.edu.au/vet/man/inf401/html/assessment.html
This model of online assessment shows how to present assessment tasks and criteria as well as sequence the learner pathway. Includes task descriptions, links to more information, assessment criteria, deadlines, and the weightings of the different tasks.
Tools and Techniques
Creating Multimedia/Interactive Tests for the Internet in 10 Minutes. http://www.aln.org/alnweb/magazine/issue2/rob.htm
An article from Asynchronous Learning Networks Magazine describes QuizPlease software, which is designed to offer tutoring as well as testing.
Online Educational Delivery Applications: A Webtool for Comparative Analysis. http://www.ctt.bc.ca/landonline/
or
http://www.olin.nf.ca/landonline/
Designed to help educators evaluate and select online delivery software. Describes and compares the most viable applications in use, focusing on technical specifications, instructional design values, media capabilities, tools, ease of use, potential for collaboration and connectivity, and contact information.
QuizCODE-A
Tool for Online Assessment and Feedback.
http://www.enable.evitech.fi/enable99/papers/quizcode/quizcode.html
Describes QuizCODE XML-based assessment software. Includes examples of future uses such as delivering quizzes over personal data assistants and mobile phones.
Rubrics for Web Lessons. http://edweb.sdsu.edu/triton/July/rubrics/Rubrics_for_Web_Lessons.html
This site discusses authentic assessment and the use of rubrics with examples, templates, and links to further reading.
The CASTLE (Computer ASsisted Teaching and LEarning) Toolkit. http://www.le.ac.uk/cc/ltg/castle/index.html
These tools make it possible to create and manage online interactive multiple choice questions. Includes tips and good practice for question and test design.
The
Design, Development and Delivery of Internet Based
Training and Education.
http://teleeducation.nb.ca/content/pdf/design_development_delivery/reportddd.pdf
Contains An Inventory of Design, Development, and Delivery Software for Internet-Based Learning, a comparison of 33 products such as QuestionMark, Toolbook, and FirstClass.
Website Building Tools
CGI Script Archive: http://www.bignosebird.com/carchive/bnbform.shtml
Matt’s Free Perl CGI Scripts: http://www.worldwidemart.com/scripts/
Page Tutor: http://www.pagetutor.com. Free HTML tutorials; tables, frames, and forms tutorials; a MouseOver tutorial; a javascript tutorial; password protection script; and lots of resources for beginning and advanced HTML authors and web designers.
Conferencing Tools
Conferencing Software for the Web: http://thinkofit.com/webconf/. A guide to Web software (free and commercial) for text-based, asynchronous group discussions, maintained by David R. Woolley.
Toolbook II: http://home.click2learn.com/ (formerly Asymetrix) Assistant: e-Learning authoring program; Instructor: high-end authoring tool designed for creating e-Learning courses and applications
Testing Tools
Online Exercises System: http://math.uc.edu/onex/demo.html
Question Mark: http://www.qmark.com/
Self-Test: www.usq.edu.au/users/taylorja/Selftest.htm
Abernathy, D. J. “The WWW of Distance Learning: Who Does What and Where?” Training and Development 52, no. 9 (September 1998): 29-30, 32. (EJ 571 414)
Aschermann, J. R. “Electronic Portfolios: Why? What? How?” In SITE 99: Society for Information Technology & Teacher Education International Conference, edited by J. D. Price et al. Charlottesville, VA: Association for the Advancement of Computing in Education; Society for Information Technology and Teacher Education, 1999. (ED 432 305)
“Assessing Students Online.” University of Newcastle (Australia), 1999. http://www.newcastle.edu.au/discipline/sociolanthrop/staff/kibbymarj/online/assess.html
Australian National Training Authority. “Flexible Learning Toolbox Project.” 1999. http://www.flexiblelearning.net.au/toolbox/
Bigelow, J. D. “The Web as an Organizational Behavior Learning Medium.” Journal of Management Education 23, no. 6 (December 1999): 635, 650. (EJ 596 509)
Champagne, M. V. “Dynamic Evaluation of Distance Education Courses.” In Distance Learning '98: Proceedings of the Annual Conference on Distance Teaching and Learning, pp. 89-96. Madison: University of Wisconsin-Madison, 1998. (ED 422 847)
Dickinson, K. “Distance Learning on the Internet: Testing Students Using Web Forms and the Computer Gateway Interface.” TechTrends 42, no. 2 (March 1997): 43-46. (EJ 543 257)
Dirks, M. “Developing an Appropriate Assessment Strategy: Research and Guidance for Practice.” Paper presented at the Northern Arizona University web.97 conference, June 12-15, 1997. (ED 423 274)
Dirks, M. “How Is Assessment Being Done in Distance Learning?” Paper presented at the Northern Arizona University web.98 conference, May 28-30, 1998. (ED 423 273)
Doulai, P., and Stace, R. “Web-based Surveys and Assessment.” Proceedings of the ASCLITE (Australasian Society for Computers in Learning in Tertiary Education) Conference, Adelaide, Australia, 1998. http://www.ascilite.org.au/conferences/
Elliott, B. “A Systems Approach to Asynchronous Distance Learning: A Community College Model.” In Teaching at a Distance: A Handbook for Instructors, by M. Boaz et al., pp. 65-73. Laguna Hills, CA: League for Innovation in the Community College, 1999. (ED 432 316)
Gibson, E. J.; Brewer, P. W.; Dholakia, A.; Vouk, M. A.; and Bitzer, D. L. A Comparative Analysis of Web-Based Testing and Evaluation Systems. Technical Report. Raleigh: Department of Computer Science, North Carolina State University, 1995. http://renoir.csc.ncsu.edu/MRA/Reports/WebBasedTesting.html
Gray, P. “JavaScript: Convenient Interactivity for the Class Web Page.” In Proceedings of the Mid-South Instructional Technology Conference, March 28-30, 1999. Murfreesboro: Middle Tennessee State University, 1999. (ED 436 117)
Gray, S. “Maintaining Academic Integrity in Web-Based Instruction.” Educational Media Journal 35, no. 3 (1997): 186-188.
Hardy, D. “Fundamentals of Designing a Distance Learning Course: Strategies for Developing an Effective Distance Learning Experience.” In Teaching at a Distance: A Handbook for Instructors, by M. Boaz et al., pp. 1-14. Laguna Hills, CA: League for Innovation in the Community College, 1999. (ED 432 316)
Hayes, K., and Huckstadt, A. “Developing Interactive Continuing Education on the Web.” Journal of Continuing Education in Nursing 31, no. 5 (September-October 2000): 199-203.
Hazari, S., and Schnorr, D. “Leveraging Student Feedback to Improve Teaching in Web-Based Courses.” T.H.E. Journal 26, no. 11 (June 1999):30-32, 34, 36-38. (EJ 589 976)
Hedberg, J. G., and Corrent-Agostinho, S. “Creating a Postgraduate Virtual Community: Assessment Drives Learning.” Educational Media International 37, no. 2 (June 2000): 83-90.
Hopper, M. “Assessment in WWW-Based Learning Systems: Opportunities and Challenges.” Journal of Universal Computer Science 4, no. 4 (1998): 329-347. http://www.jucs.org/jucs_4_4/assessment_in_www_based/paper.html
Hutton, S. “Course Design Strategies: Traditional versus Online. What Transfers? What Doesn’t?” Paper presented at the Annual Meeting of the American Association for Adult and Continuing Education, Phoenix, Arizona, November 1999. (ED 430 115)
Jarmon, C. “Testing and Assessment at a Distance.” In Teaching at a Distance: A Handbook for Instructors, by M. Boaz et al., pp. 55-63. Laguna Hills, CA: League for Innovation in the Community College, 1999. (ED 432 316)
Juchnowski, M., and Atkins, P. “Online Assessment: Let’s Do It.” Swinburne University of Technology, Australia, 1999. http://www.tafe.swin.edu.au/indsci/assess/doit.htm
Kennedy, D. M., and Eizenberg, N. “An Open-Ended, Short Answer, Text Question Tool: Improving Interactivity on the Web.” In ED-MEDIA/ED-TELECOM 98 World Conference on Educational Multimedia and Hypermedia and World Conference on Educational Telecommunications: Proceedings, edited by T. Ottmann and I. Tomek. Charlottesville, VA: Association for the Advancement of Computing in Education, 1998. (ED 428 681)
King, K. P. “Course Development on the World Wide Web.” New Directions for Adult and Continuing Education no. 78 (Summer 1998): 25-32. (EJ 567 058)
Lindner, P. Assessment Tools for Distance Learning: A Review of the Literature. Olympia: Washington State Board for Community and Technical Colleges, 1998. (ED 426 725)
Mann, C. C. “Quality Assurance in Distance Education: The Surrey MA (TESOL) Experience.” Distance Education 19, no. 1 (1998): 7-22. (EJ 572 220)
Marshall, G. “Models, Metaphors and Measures: Issues in Distance Learning.” Educational Media International 37, no. 1 (March 2000): 2-8.
McDougall, G.; Place, C.; and Currie D. “Web-Based Testing: A Form-Based Template for Creating Multimedia Enhanced Tests.” In ED-MEDIA/ED-TELECOM 98 World Conference on Educational Multimedia and Hypermedia and World Conference on Educational Telecommunications: Proceedings, edited by T. Ottmann and I. Tomek. Charlottesville, VA: Association for the Advancement of Computing in Education, 1998. (ED 428 692)
Milman, N. B. “Web-Based Electronic Teaching Portfolios for Preservice Teachers.” In SITE 99: Society for Information Technology and Teacher Education International Conference Proceedings, edited by J. D. Price et al. Charlottesville, VA: Association for the Advancement of Computing in Education; Society for Information Technology and Teacher Education, 1999. (ED 432 273)
Mogey, N., and Watt, H. “The Use of Computers in the Assessment of Student Learning.” In Implementing Learning Technology, edited by G. Stoner. Edinburgh, Scotland: Learning Technology Dissemination Initiative, Heriot-Watt University, 1999. http://www.icbl.hw.ac.uk/ltdi/implementing-it/using.htm
Morgan, C., and O’Reilly, M. Assessing Open and Distance Learners. London: Kogan Page, 1999.
Nelson, G. E. “On-Line Evaluation: Multiple Choice, Discussion Questions, Essay, and Authentic Projects.” Paper presented at the Third Teaching in the Community Colleges Online Conference, Kapiolani Community College, Hawaii, April 7-9, 1998. (ED 430 695)
Niederhauser, V. P.; Bigley, M. B.; and Hale, J. “Cybercases: An Innovation in Internet Education.” Journal of Nursing Education 38, no. 9 (December 1999): 415-418. (EJ 596 594)
Pennsylvania State University. An Emerging Set of Guiding Principles and Practices for the Design and Development of Distance Education. University Park: Innovations in Distance Education, PSU, 1998. http://www.cde.psu.edu/DE/IDE/guiding_principles/
Ravitz, J. “Building Assessment into the Design of Online Projects.” Paper presented at the National Educational Computing Conference, San Diego, CA, June 23, 1998. http://www.gse.uci.edu/Ravitz/Necc98_Assess/1.html
Schacter, J.; Herl, H. E.; Chung, G. K. W. K.; Dennis, R. A.; and O'Neil, H. F., Jr. “Computer-Based Performance Assessments.” Computers in Human Behavior 15, nos. 3-4 (May-July 1999): 403-418. (EJ 593 463)
Schrum, L. “On-Line Education: A Study of Emerging Pedagogy.” New Directions for Adult and Continuing Education no. 78 (Summer 1998): 53-61. (EJ 567 061)
Tarouco, L.; de Vit, A. R.; Hack, L.; and Geller, M. “Supporting Group Learning and Assessment through Internet.” Paper presented at the Trans-European Research and Education Networking Association Conference, Lisbon, Portugal, May 2000. http://www.terena.nl/tnc2000/proceedings/1B/1b2.html
Taylor, J. A. “Self Test: A Flexible Self Assessment Package for Distance and Other Learners.” Computers & Education 31, no. 3 (November 1998): 319-328. (EJ 580 074)
Thorpe, M. “Assessment and 'Third Generation' Distance Education.” Distance Education 19, no. 2 (1998): 265-286. (EJ 582 131)
Tillson, T.; McLean, G. N.; and Warner, J. “An Evaluation of Distance Learning in the University of Minnesota Human Resource Development Program.” In Academy of Human Resource Development Conference Proceedings, Oak Brook, Illinois, March 4-8, 1998, edited by R. J. Torraco, pp. 23-30. Baton Rouge, LA: Academy of Human Resource Development, 1998. (ED 428 225)
Wild, M., and Omari, A. “Developing Educational Content for the Web: Issues and Ideas.” Paper presented at AusWeb96, 2nd Australian World Wide Web Conference, Gold Coast, Queensland, Australia, July 7-9, 1996.
Zirkle, C., and Ourand, D. B. “Teaching a Course through Multiple Delivery Systems: Some Lessons Learned.” Paper presented at the Indiana Postsecondary Education Conference, Muncie, Indiana, October 25, 1999. (ED 435 800)
Zvacek, S. M. “What's My Grade? Assessing Learner Progress.” TechTrends 43, no. 5 (November 1999): 39-43.
This issue of the Practitioner File was written in 2000 by ERIC/ACVE staff members Sandra Kerka and Michael E. Wonacott, with additional information provided by Cheryl Grossman and Judy Wagner.
This project has been funded at least in part with Federal funds from the U.S. Department of Education under Contract No. ED-99-CO-0013. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. The Practitioner File may be freely reproduced and is available at http://ericacve.org/fulltext.asp.