I believe in transparency and think an open community has to be transparent, so I am going for full-disclosure on the inner workings of the judging process for the Romanian Wiki Loves Monuments contest.
The team is small, 2 organizers, 3 jury members, one person shared by both teams, so 4 people in total, the number of submissions had to be made manageable. For this we had each member of the jury go trough all submissions and make an initial selection, as his proposal. Each of us proposed around 100 images.
Then the image lists were merged, sorted, and the duplicates removed, resulting in a final selection of 284 images, those were marked in a special Wikimedia Commons category, have a look at it, is supposed to contain the best images.
The harder part is the next one: all 284 pictures will be reviewed by each member of the jury and each will receive a rating on 3 categories: technical, originality and usefulness, the final note will be the average of all 3 notes, and this will be the winning order.
For this we needed an application, preferably web-based and since I am way to lazy to code a web app for a single use, I went to the darkside and used a proprietary solution: Google Docs. It has 2 godsends: one is the ability to be collaboratively edited at the same time by multiple users, and the other is a formula, "=IMAGE(URL, view)".
So how does my app look? Is a spreadsheet, first column is the image thumbnail, made with "=IMAGE(URL, view)", the second is the URL of the image page, to be easily consulted at full size, then a series of columns for noting follow, the last one is a general purpose comments column. Each jury member has 3 columns for notes (technical, originality, usefulness) and an automatic total, a pondered sum of his notes (technical * 30% + originality * 50% + usefulness * 20%), then the sums are averaged. Quite simple. And you can see a read-only copy of this spreadsheet, reuse the model if you have any use for it.
Now there is a little over a week, until the end of the next week, while the jurors have the time to give their notes and then they will meet face to face for any eventual arbitration and final review.