a.k.a. Grading Professors So WE Get Better Feedback

From Eben: " I think it is time to stop expecting to acquire commitment points by writing extensively about how to improve law school without having "actually" been through it. On the plus side, given the baseline . . . some . . . usual chatterers have laid down, you can get improvement points at this stage by writing even badly about substantive subjects. . . . [A]ll the unnecessary verbiage that has been expended on discussing . . . a hypothetical (and quite ridiculous) initiative to use "course evaluations" to improve the quality of educational evaluation - is still obscure to you, as it is to most of your teachers. That's not important here: you're studying to be a lawyer, not an educational reformer. You might want to take up the task in hand."

@Eben Thank you for your guidance. I would like to hear more about why the below idea is "ridiculous," but understand that you do not want the class to go in that direction.

All right, let's start there. Students do not "grade professors" through course evaluation forms. Grades would be at least as stupid for teachers as they are for students, and even grades for courses are stupid. The utility of course evaluations, in my view, is slightly higher than the utility of letter grades because they do not reduce all qualitative measurements to a single quantity, but they have all the other difficulties I discuss in the EvaluationPolicy. The actual learning about teaching that I do, like the actual learning about law that students do, is done in the relationship between teacher and student. That's no more about anonymous scaling for me than it is for you.

What course evaluations actually do is to regulate elective enrollment. A teacher who wants to maximize the pay he receives while minimizing the work he does in order to receive it will minimize teaching effort, because teaching is hard. Being known to grade exactingly, or to do some other thing students dislike and teachers profess to admire reduces workload without penalty. Indeed, there is essentially no penalty for driving down elective enrollment by any means for any teacher, tenured or untenured, and the skill is widely practiced, with different degrees of self-consciousness, throughout the firm.

"Feedback" questions on an evaluation are essentially, in my evaluation vocabulary, an effort measurement; indeed, they're a good one. A teacher not giving you much feedback may be exerting effort to create good classroom experiences, or to choose material for you wisely, but these are the fixed costs of teaching. The variable costs are in the relationships one has with students, and one cannot have meaningful relationships with students in their learning process without interaction, which we call "feedback," unwisely. So in measuring quality of feedback we are essentially measuring quantity of student-specific effort.

But tradition of a century's standing has determined how much effort is "enough" at law school student-faculty ratios. So a teacher reading low on the feedback scale is actually doing his job. He has no incentive to increase his level of effort, if that would increase his enrollment which would require increasing his effort level (a process that is technically and correctly known as "feedback"). No reputational penalty will be paid for doing enough, rating low on the effort scale (helpfully now named in a euphemistic way "Feedback"), and depressing enrollment in a socially-acceptable way.

So in my analysis from both theoretical and practical perspectives the idea is ridiculous: inappropriate, insufficiently self-critical, half-baked and likely to cause the reverse of its intended effects if it has any effects at all. These are positive qualities in an idea. That's because every idea leads to other ideas, and there's no guarantee that an idea which is "wrong" will have inferior ultimate intellectual effects to an idea which is "right." (I have mentioned this point before.) But although a ridiculous idea is an enormous advantage in a sequence of ideas, it's a disastrous social tactic, that is, a tool for achieving finite objectives given finite resources. Small ridiculous ideas turned into tactics are what result in horrendous casualties for no strategic benefit, while large ridiculous ideas turned into strategies are what result in imperial calamity. When you start turning this notion into an objective of political organization, it goes from being inventive and therefore helpful regardless how naive to being perilous. So I try to head it off by sending a small cavalry force to the pass with instructions to use the flat rather than the edge of the blade.

Now, do we really need to spend a great deal of time on why I think that trying to reform law school when you've not yet seen five-sixths of it once is sort of like offering legal opinions about cases you haven't read?

You actually solemnly give me sentences of argument on why lawyers can be educational reformers, when the whole point is that if you undertake to be lawyers, you won't do educational reform or anything else by thinking you can know the case by reading the press coverage, or know the intricacies of the system by watching television, or know the alphabet by mastering the first 19% from A to E.

I have every confidence, more confidence than you know, in your class's ability to mobilize itself to change law school. If law school has not substantially changed itself by the time you reach the last semester of your third year—which I very much doubt it will have done, or even shown inclination to do—you will have an opportunity that comes only once a generation to make your mark in forcing it to change.

But you're not ready yet. You haven't been trained enough. You don't understand the environment enough. You don't have institutional memory because you've just arrived, and without institutional memory you have no basis to evaluate the effects of your actions. You can know now only why law school doesn't work well for you; over time that will permit generalizations for which I am doing my best to provide you with real intellectual support. But the hasty adoption of an aggressive tactical posture consisting of knowing what should be done and how to force it to happen is puerile. By the time you are a well-trained lawyer I hope you'll have learned to approach situations differently. I hope you'll have come to appreciate the pleasure, as well as the necessity, of learning about social situations in a less bias-cultivating "advocative" fashion, and thus to build strategies maturely, on a sound factual basis, as well as creatively, by incorporating many forms of social knowledge in your eventual arrangements. I hope you'll have learned also to adopt tactics in close underlying relationship to the strategies that determine what should be achieved and how much blood and treasure can be allocated to achieving it. Such strategic wisdom and tactical adroitness would allow each of you to devise the master strategy of your practice, to balance getting the resources that you need (including those that support you and your family) with a mix of professional activities that achieves real, valuable goals for you and your community, "feeding back" sufficient resources to sustain the practice in its next phase. With such knowledge you could do what you all quite rightly say you don't see the way to doing now. On the way out the door you could do something good for the future of legal education, but that's the least of it.

I'm trying to create a place of relationships among us in which the basic parts of that learning can be communicated (this is a first-year course, after all, and you are still at the beginnings of your learning process—not everything could be communicated in fourteen weeks even by a perfect teacher to a perfectly-prepared and ideally-behaved student). We are still far from socially committed to the process however, and week after week we are still clearing away brush and building trust. There is still a prevailing belief that I'm trying to get you to have my ideas about what lawyers should do. And there's still a prevailing idea that in order to learn what you need to learn you'd have to fix law school. Both are wrong.

I am not sure where to go now however. Perhaps you can help. While you make a strict distinction between education reform and lawyering, isn't it largely lawyers who write education policy. Certainly law school policy is created and changed by lawyers. From Robinson I took away that a "real lawyer knows how to take care of a legal problem." Advocating for an administrative change to improve a problem is exactly the kind of thing I hope to do as a lawyer. I thought as an aspiring lawyer it is appropriate for me to learn to take care of legal problem. Am I thinking about being a lawyer incorrectly or Am I thinking about your class incorrectly? [Alex, I changed the order of your I(s) and am(s) in order to make sense with your ?, if this is not what you intended I apologize.-Rob]

Students grade professors through course evaluation forms. Maybe we can use these forms to get better feedback from our professors. (The irony that the feedback we give them is already way more instructive than the feedback we receive is not lost on me.) Anyway, through the evaluation forms we give feedback on many different aspects of the professor's performance, but we don't give feedback on how good their feedback to us is. Maybe if we successfully lobby for a "rate your professor's feedback" box on the evaluation forms, we can begin to establish feedback as an important part of a professor's job.

Changing the forms might be a good first step in changing how professor's think about their responsibility. Getting a "feedback given to student" question on the form does not require the university to expend any resources or change any policies. My hunch is, however, that professors are adverse to getting negative evaluations, and after a semester of getting low ratings in the "feedback given to student" category they will evolve. I hope the evolution will not be in the form of a faculty resolution to strip the new category from the evaluations. (Whether cynics or optimists change the world is a question for a different discussion.)

The first hurdle is that most of the little feedback we get comes from exam grades and, if we are lucky, exam comments one month after we fill out the evaluation forms. I have talked to Dean Schizer and he emphasized that it is important to have students fill out the forms before finals, because the response rate decreases if the forms are given after finals. I suggested to him having two rounds of evaluation:

Round I: Evaluations just like they are now. Neither the timing or questions change.

Round II: After the semester, students can answer three additional questions on the quality of the feedback they got during the semester, the quality of the final exam, and whether they got a grade they expected.

The Dean responded quickly -- and I give him credit for doing so -- writing "I will pass on your thought to colleagues who help to set these policies." I am sure he is very busy, so I do not expect anything to happen unless someone pushes it along.

If other students are interested, maybe we can develop this idea further and try to build momentum.

What do you think?

-- AlexAsen - 12 Feb 2010

Interesting idea, but I question what value it would really have. Is there any evidence professors do anything with the evaluations they get now? That their supervisors look at them or do anything with them? If they don't care about what we're saying now, or at least don't change in response to it, I don't see how rating their feedback level will make a difference any more than any other complaints (or compliments) do. I'm especially skeptical because it seems that all this would do is congratulate those profs who do give feedback, but would not hurt those who don't, because the norm is not to. If the school doesn't require (or even encourage) any feedback other than grades, the school is not going to punish a prof for us telling them what they already know-- that we don't get feedback. Another problem is that profs are hired and kept around based on what they publish/the work they do outside of class, not if they are particularly good teachers, so I don't know if the school really cares how much feedback we're getting in the first place.

On the other hand, it couldn't hurt, and would at least give future students an idea of what to expect. And like you said, it would not involve too much work to implement.

-- RorySkaggs - 13 Feb 2010

@Rob- I think you're confusing the necessity to change a problem with differences regarding how to do it. I also think you are addressing a different problem than Alex did-- my understanding was that Alex was proposing this idea in an effort to change what professors do, not to just let the students know about what happens. And my point is that if we are trying to change what professors do, this seems a rather passive way to do it. As I mentioned, do professors change in response to student evaluations? And more importantly, does administration put any pressure on them to give feedback to us? If we really want to change the system, we have to attack it directly, not just by telling each other what we already know. We need to change how classes, and probably the whole curriculum, are structured. I'm not at all sure how to do that, all I'm saying is adding another box to class evals doesn't seem like the best or most effective way to do it.

Also, I'm not sure how much benefit they would have even for us, especially during first year when we can't pick our classes anyways. Don't we know that we're not going to get feedback just by the nature of law school itself? Until that changes, I'm not sure how much there will be to say. The eval will say 'you take one test and don't get any feedback on it', we will say 'yes I know it's like almost every other class I could have told you that.' So we need to figure out how to change the norm, not just talk about it. But like I said, it also couldn't hurt anything, especially if some professors do change in response to evaluations, so I would support it regardless-- just not as the only way to solve the problem.

-- RorySkaggs - 15 Feb 2010

As practice in reduction and editing, my thoughts on this is it is an easily attainable way to alert students to professors who provide quality feedback. I think that is a good thing. As for changing the structure of the legal education, I doubt it would have an effect. That is all, let's move on to other things now.

-- Main. RobLaser? - 17 Feb 2010

@Rob and @Rory - Thanks for the feedback! I very much appreciate you taking the time to consider this idea.

@Rob I admire your ambition.

Eben recounted the history leading up to Brown v Board of Ed in class. He talked about how equalizing pay for black teachers built up a base of support for future initiatives and, to a lesser extent, got the country comfortable considering that there is a problem.

(Please do not read this as me equating institutional segregation and poor professoring as morally equivalent problems. I am just trying to draw lessons from a successful movement.)

The lessons I draw are (1) start small and (2) take concrete steps the affect actual, if small, change.

I don't think we are ready for protests such as sit-ins. Foremost, I don't think we have the support from the student body required to making protesting work. People are busy and I think we have to let this issue develop naturally. I hope these new evaluation questions may fertilize that natural development. As you pointed out, evaluations will help future students know what to expect and I think it will also help current students realize what is missing.

The second reason I think we have to take a slow approach to this issue, is for two reasons I don't think we can force professors into changing their ways; we have to make them want to. Protests make us adversarial. We are only here for another 2.5 years and will probably only be engaged for 1 more year. The faculty knows they can just wait us out.

Eben quoted an anonymous professor arguing for the existence of grades because giving students feedback is important. My hunch, and hope, is that many professor are genuinely interested in providing the best possible experience for their students, they has just not considered that current system is failing. I think the best way to show that the system is failing is by giving professors feedback on the issue. After all, that is our point -- to improve one first needs feedback on what they are doing wrong, otherwise they will continue to make the same mistakes.

I am happy for people to spend time brainstorming bigger ideas, but I don't want to lose focus on this idea. Perhaps you would like to make a post starting a forum to discuss the big ideas and this page could become a subtopic. What do you think?

One last point, I can think of no better weapon for future protests, if it comes to that, than a mound of data showing that feedback is currently a problem.

I think the next steps are to:

(1) draft what we want the Round II questions to look like.

(2) draft the letter introducing Round II and making it clear we do not want to interfere with the timing of Round I. When I briefly exchange emails with the dean about adding Round II, protecting Round I was his biggest concern.

(3) revise.

(4) figure out who has the power to implement this and contact them. Is anyone involved in the Student Senate? Could it be helpful?

I think we should wait until after class on Tuesday before moving on with this road map. I imagine this wiki is most active on class days and hope more people will add ideas to the discussion before we continue.

-- AlexAsen - 15 Feb 2010

I'll hold off anything substantive for now, but one thought popped into my head which I think needs to be answered-- is this the kind of feedback we're looking for? What we did right or wrong on a test? Isn't that feedback only useful for taking other tests, which is not what we'll be doing once we leave school? It seems to me the feedback we want should be related to how to be a lawyer, not how to take a law school test. That's why I think some more fundamental changes are needed, and also why I find it kind of perplexing how many law school professors have never actually been lawyers.

-- RorySkaggs - 15 Feb 2010