Department of Mathematics and Statistics
University of Nebraska - Lincoln

We started using computers to give automatically-graded basic skills tests in calculus in order to make sure that students in our reform calculus course had strong enough a grasp of the core algebraic skills to satisfy our various constituencies (see our article, "On-line Gateway Exams in Calculus"; Focus on Calculus, 14). Although our focus then was on "Gateway Exams," it was clear to us from the start that the software we were developing (and which is now marketed for calculus classes as "The Wiley Web Tests in Calculus") had the potential to assist a wide variety of styles of assessment. In this article we want to describe the experiences we, and some of our colleagues at other schools, have had using the Wiley Web Tests system in a variety of ways over the past three years.

The term "Gateway Exam" is used to describe a test that students are required to pass at a certain level of proficiency in order to pass the course and proceed to the subsequent courses. A Gateway Exam should focus on the core skills of the course, and provides a certification to follow-on courses and client disciplines that students graduating the course have mastered those skills. In practice, as we use it, success in the Gateway Exam is not an absolute requirement for passing the course. But the weight that the exam carries (about one half a letter grade), together with the core nature of the material covered, make it play a pivotal role in the course, so that success in the Gateway Exam is very highly correlated with success in the course.

We originally developed our Web Testing system to give Gateway Exams in the basic skills of computing derivatives in Calculus I, and integrals in Calculus II. When we first ran on-line tests, in Fall 1996, we had been teaching reformed calculus using the CCH materials for two years, and had become aware of two concerns. The first was that the increased time spent on conceptual problems and applications would "squeeze out" computational skills to the extent that students could no longer be relied upon to compute simple derivatives and integrals. The second was that the wide diversity of assessment tools used in a reform course – homework, exams, quizzes, and projects – might allow a weaker student to scrape through with a C grade and so proceed to follow-on courses without ever really mastering any one part of the course.

A brief experiment with paper Gateway Exams encouraged us to believe that the Gateway concept was sound – over 80% eventually passed – but the burden on our faculty and GTA time was prohibitive. Each year we teach Calculus I and II to about 1500 students and it was clear that the testing had to be automated if Gateway Exams were to succeed.

Using the system we developed for the Wiley Web Tests, students are offered essentially unlimited retakes, together with unlimited opportunity to practice the exam on their own. Because the tests are given over the Web, they can in principle be taken from any computer connected to the Internet. However, because these exams are "high stakes," concerns about honesty make us insist that students take the exam for credit either in our computer lab, or in our Math Resources Center - where they can also get hands-on help from GTAs to prepare for a test or analyze test results. Of course students can practice exams or engage in a study session (see below) from home, the dorms, or labs around campus.

Practice exams are an extremely important part of the on-line assessment system. The expectations in the Gateway Exams are very high - the score required to earn credit for the exam is always over 80% - and initially as few as 30% of the class may pass. But with the help of repeated practice over the Net, over 80% usually pass, over a period of weeks. Our experience is that it usually takes a testing period of four weeks to reach that point. It seems that many students need the exam before them for a few weeks in order to realize that the challenge will not go away, and to learn, from their experiences with practice tests, that they can indeed meet it. Students have told us the knowledge that the practice test they take at home is the same as the test they could have been given for credit, is a very powerful motivator to keep them studying.

Colleagues at other schools report similar experiences. Matthias Kawski at Arizona State University also found that eventually all the students in his Calculus I class who went on to complete the course, passed his on-line Gateway Exam on differentiation and simple integration. Indeed when he tried a second on-line test in that course, for which partial credit was available, he found that students would repeat it anyway until they had reached over 80% correct.

In addition to Calculus I and II, we now give on-line Gateway Exams to two of our college algebra/trigonometry classes using questions we developed for the Wiley Web Tests in Precalculus and College Algebra. In total something like 2000 students take on-line Gateway Exams in our department each year. We cope with what could be an enormous load on our computer resources by giving the first iteration of the test in class on paper. Students who pass that test are filtered out of the system, and for the remaining students, our lab and Resource Center, with a total of 28 machines, are open enough hours in the week to handle the demand, provided the exams in the four courses using on-line tests are spaced through the semester.

Because the stakes are high for our Gateway Exams, students must take the test for credit under a proctor’s supervision. In situations where the consequences are lighter, such as homework assignments, it makes sense to allow the students to take on-line assignments from anywhere they can connect to the Internet. The Wiley Web Tests system supports this kind of test in the "Unproctored" mode.

For example, Professor Bill Ziemer at Indiana University wrote his own bank of test questions (using the LaTeX test-authoring tools described below) which he uses to give his students on-line homework assignments. For him, the advantage of an on-line system is that it enables him to assign homework to large lecture classes and know that all the homework will be graded. He reports that previously, because of the size of his classes, GTAs could only grade a fraction of the students’ total assignment.

At the University of Nebraska, we have used the test bank of questions from the Web Tests in Calculus to give review assignments at the beginning of courses. For example, some instructors have assigned a selection of differentiation questions as review at the start of Calculus II, or assigned integration and differentiation questions for homework in multivariate calculus.

Going further, regular on-line homework assignments offer an exciting possibility of changing the paradigm of how we use classroom time. Students can be given regular reading assignments, together with on-line mastery homework on the basics of the material in their reading, before any classroom discussion takes place. The students’ first contact with the material is in their private study, and they measure their progress by retaking the homework assignment until they can demonstrate mastery of the basic skills of the topic. Only at this point, after all - or at least most - of the class have reached a common level of competency, is there a classroom discussion of the material. Going into class, the instructor knows from the scores in the on-line grade book, that the class has reached a level of familiarity with the basics of the topic, be they names and dates, or definitions and techniques of computation. The classroom time need not be taken up with introducing this material, or with talking about it as to people who are seeing it for the first time. Instead it enables more productive, high-level classroom discussion. Colleagues in our Psychology Department who are developing course material for the Web Tests system using this paradigm report exciting improvements in student results.

As a key feature, the Wiley Web Tests system contains questions that ask the student to enter a formula. The student’s answer is interpreted by the machine as a mathematical expression when it’s graded, so that two equivalent answers such as "(x+1)^2" and "x^2+2x+1" are graded the same. Students enter the formula for their answer in a fairly intuitive syntax, which is compatible with, for example, the TI-86 calculator, and so is something most students have experience of.

Nevertheless, in the first few semesters that that we gave Gateway Exams using this type of "open response" question, we saw quite a number of students who clearly knew how to answer the question correctly, but were graded wrong because of small slips. A typical example would be omitting parentheses - for example, typing "1/x^2+1" in place of "1/(x^2+1)". Although the computer can recognize and warn about syntax that is flat wrong - such as unbalanced parentheses - this type of error is frustrating for the student but difficult to provide help with.

Our solution was to provide capability for the student to preview an answer in standard mathematical notation. In the latest version of the system software (Version 0.4), open response questions come with a "Preview" button beside the answer entry box. When the student has typed an answer, she presses the button and a screen pops up showing her properly typeset answer. If the student really does know the answer, she can spot what she’s done wrong as soon as she sees something like

in place of .

Version 0.4 of the software, which is included with the Wiley Web Tests for Precalculus and College Algebra, is available as an upgrade to users of the Wiley Web Tests for Calculus from the URL at the end of this article.

In addition to testing, the Web Tests system provides students a set of interactive exercises, or Study Sessions. In a Study Session, the system asks the student a series of questions, which the student works one at a time. The machine proves immediate feedback on each question as it’s answered, and offers hints, or complete worked solutions. If the student gets a question wrong, she can return to the question and try again, or go on to a new question. This back and forth dialog continues for as long as the student keeps asking for new questions.

The website at http://www.wiley.com/college/webtests contains links to resources for the Wiley Web Tests system. Licensed users of the Calculus system can download an upgrade to their system software from this site with a password. There are also links to a FAQ page, and software tools to help administer the system.

Users who want to write their own test questions can download a set of LaTeX macros that automate the process of preparing test questions. These macros – which we used to write the test banks for the Calculus and Precalculus systems – make it easy to include mathematical notation, and to organize and proofread large numbers of questions.