DENNIS CLAYSON

Professor, College of Business and Administration

Recently, an article published in the Northern Iowan weighed  the possibility of making student evaluations of teachers (SET) public.  Pros and cons were presented, but the discussion ignored a central and more compelling question: Why have them at all?

Defenders of the SET process typically advance several justifications for the present system.

1. SET is a valid measure of the quality of teaching.

2. The process will improve instruction.

3. SET allow administrators to reward instructors who are performing well in the classroom and who really “care” about their students.

4. Bureaucrats demand it.

None of these (except for bureaucratic complexity) are sustainable under closer scrutiny.

The evaluations are not valid enough to justify the way they are currently utilized. There are many different types of validity.  The pertinent ones depend on how the evaluations are done and, more importantly, their purpose.  Being told that SET is valid, in a universal sense, is like being told that you don’t need AC and only a light jacket to live in Iowa because the yearly average temperature is 50.9 degrees.

The issue of measurement is key. What is the SET instruments designed to measure?  UNI, like most universities, has gone to great lengths not to define this.

Consider a John Deere engineer who creates a new measurement instrument.  It is elaborate and expensive and required the input of numerous individual and groups.  A manager finally asks the pertinent question, “What is this thing supposed to measure?”  The engineer replies, “I don’t have a clue, but it should be adopted by everyone.”

Do instructors who get higher evaluations produce students who have learned more? In other words, are SET scores positively related to learning?  No.  At one time they may have been, but recent meta-analyses of results from higher education have found no relationship between objective learning and the evaluations for the last two decades.

In fact, recent research has shown negative correlations if 1) students are not allowed to select their own sections of a class, and 2) stringent statistical controls are utilized.     

Does instruction improve when SET systems are in place?  As students are well aware, there appears to be no relationship between the quality of teaching and how often instructors are evaluated.  Even defenders of the process and those who make a living pushing the use of SET like Peter Cohen, have stated that there is no improvement unless outside experts are brought in to interpret the process.

Ask some another questions: Since almost no universities used this system in the past and almost all do now, has education improved?  Do students come out of college better prepared? Are they happier?  Has society improved?  Even if you live in a cultural bubble and said “yes” to these questions, it would be almost impossible to connect your opinions with the universal usage of SETs.

If the evaluations don’t measure learning and they don’t necessarily improve the quality of education, what do they do?

Good question.  Other than take up a lot of time and expense, they appear to do three things.

1)  They are a good measure of the average “popularity” of an instructor.  Several research studies by UNI faculty have shown that SET could be replaced with a personality inventory with almost no change in anything at all.  Having popular teachers does have some merit.

Certain administrators live and die based on enrollment numbers.  Having happy students is not an unworthy goal. However, the reasons a teacher may be “popular” are not always positive.  Consider the following:

Suppose a hospital judged their surgeons the way faculty are evaluated.  Surgeons would be promoted, both in the PR and administrative sense, based on their popularity with patients measured with evaluations filled out BEFORE their operation.

2) They allow faculty and administrators to feel good about themselves.  I have been doing this professor gig for over 40 years.  In that time span, I have met exactly three professors who did not believe they were master teachers.  It is the universal conceit of our occupation.

Students are very generous and SETs are highly skewed, which means that, on average, most get high evaluations.  The instructors are saying, “I know I am a master teacher, and my students agree.  SETs are obviously wonderfully valid.”

3)  As one critic put it, the main advantage of the SET process is that it allows administrative sloth and student crowd control.

In addition to other problems, the SET process is filled with legal and social landmines.  They appear to advance discrimination.  A new line of research has shown that gender discrimination has once again raised its ugly head in the evaluation process.

Women in some disciplines get lower evaluations than men.  Black men typically get lower evaluations, as do instructors in harder disciplines.  Math and accounting teachers get lower scores than those in psychology and music.  Better looking instructors get higher evaluations.

Several large-scale studies have concluded that SET give a much better picture of students than it does of instructors.  Yet, universities, including UNI, continue to base merit pay, tenure and promotion on instruments with weak to nonexistent validity, in which campus wide comparisons are made irrespective of discipline, gender, race and other biasing influences.

Administrators say they know what is “really happening” and they will adjust for such things, but that only puts us back to the time when administrators made these important decisions by themselves.  The apparent difference is now they have SETs filed away somewhere in case an administrator later needs to cover their behind.

In summary, the SET process is:

Small nonrandom samples from biased observers with questionable instruments subject to statistical analysis of unwarranted scales is used to determine a hypothetical construct that no one will define.

Is there a SET that would make sense?   Yes.  But only if the goal was better teaching.  Simply ask three questions:

1)   What went well in this class?

2)   What could be done to make the class better?

3)   What is your name?