Ennies judge, part 2

Have you ever been a part of a book club?  Where you read a book, and then meet to discuss the book?  Maybe one book a month?  Or remember in college or high school when your entire class read a book, and then would discuss in class what you thought of the book and so forth?

Well, judging for the Ennies is not quite like that.

First, the scale is such that there is no way we could discuss every single submission in detail.  We received 366 submissions – most of which were books that arrived at my doorstep and probably made my mail carrier and UPS driver cry a lot.  There were a lot of e-submissions as well, which arrived in PDF format (searchable – yay!).  And then there were podcasts to download, websites to peruse, and the occasional piece of software.

Second, the judges for the Ennies do not meet like you would with a book club or a class.  I have yet to meet any of the others in person, for one thing.  For another, the vast majority of our year was spent in solitary efforts to read and form opinions about submissions.  It was punctuated with the occasional email with questions or comments.  But 95% of the effort involved is individual and self-motivated.

The Ennies organizers let the judges be – aside from answering the occasional question or helping with problems – to maintain the independence of the judges.  We had an agreed-upon deadline to provide our list of nominations to them, but they in no way stepped in to shape our opinions.

So what does all of this mean?  It means the vast majority of my judging duties was sitting in a quiet place, reading reading reading.  I would use my lunch hours at work to read.  I would read on the weekends.  Not every day, but in May and June pretty darn close to it. 

We kept a spreadsheet where each judge could note his or her personal candidates for nomination in each category.  I changed mine frequently, as I read more things.  I would revisit items another judge ranked highly but I hadn’t cared for particularly.  It was an ever shifting process.

Occasionally, I would have to remind myself that the Ennies were not about what my “favorite” products were, but what the “best” products were.  It didn’t happen too often, but there are still a couple of products that I loved that didn’t make the cut for other judges.  And that’s okay.

So … after lots of preparation and re-evaluation, the actual judging process went fairly smoothly.  It consisted of the judges in a Google Hangout, looking over the list of items each judge would put forward for nomination.  There were many products that overlapped between judges, and that made selection easy.  But once past those initial items, we would essentially debate amongst ourselves – very respectfully and politely I might add – what deserved to be nominated.  It took a few hours, and then after further consideration in the following days we changed two of the 100ish nominations.  It sounds simple because it was simple.  There were some categories where a judge or two would have very strong opinions, and then other judges had strong opinions in other categories.  It worked out because we were all prepared, we listened to each other, and we kept emotions out of it for the most part.

That in and of itself was a revelation to me.  I had been anxious about the judging process because I didn’t know the other judges (you can’t read tone into emails), and I didn’t know how being the only girl on the panel would go.  What I learned was that the judging process in and of itself was all about building consensus.  It was almost like a group of five people mediating themselves.  I will give you a (fake) example.  Say, judges 1-3 really liked Book Lasagna, but judges 4-5 really liked Book Enchiladas.  Well, perhaps the negotiated result of that conflict would be nominating Book Lasagna in this particular category, but recognizing that Book Enchiladas was also a quality product by putting it on the short list for nominee discussion in a different category that would be discussed a bit later.  It worked out rather well.

So after a few hours, we had a list of nominations.  We slept on it, and further refined a couple of nominations, but all in all we are very satisfied with our decisions. 

Ennies nominations will be announced July 13, after which I will give you some details to chew on (though not gossipy who-liked-what stuff – just my opinions or products I want to highlight).