What's an optimal strategy to generate an accurate ranking for the Prediction phase of a contest?
The problem here is that you're not just ranking the submissions based on your own tastes, nor are you predicting the "taste of the crowd" (a much simpler task) -- your goal is to pull off the even more difficult feat of predicting the taste of a much smaller entity (i.e., a committee of judges), without any prior explicit knowledge of their taste. That's hard!
For instance, for the MatMarket contest's prediction phase, I decided to try something different: I narrowed down the entries to my top 10 and asked a couple friends and a family member to rank them based on the product info and a summary of the contest requirements I'd told them. Basically, I'm attempting my own little second-degree crowdsourcing experiment. :] The rankings I got back, however, were not only wildly different from each other, but all were also orthogonal to my own ranking prediction!
Of course, these are only 3 data points, and the rankings may converge/stabilize if I got a lot more people to rank my top 10. Still, even in this case, the big gotcha is that there's absolutely no guarantee that there's any correlation between the rankings I get back and the rankings the judges will produce! If anything, my collected data would be better predictors for how people would rank the entries *in general*; this would be more useful if contest winners were decided by the entire Tongal community. Alas!
Even with mitigating factors such as explicit contest descriptions, rules, and requirements, it's hard to be a consistently good predictor in such contests, especially since the judges are almost always different people with different tastes and biases. That's exactly why it's very rare to see the same users win across many such prediction contests, and why such people are extremely valuable in many arenas. And if there are any such near-psychic people reading this -- what's your strategy? Do you follow some scientific method or mathematical model? Crystal ball? Do you crowdsource your friends, like I just tried? Is there even a strategy? Or is it all your own insight, intuition, guesswork, and black magic?
I'm not sure how (or even if) Tongal uses our predictions to infer deeper insights about contests, judges, or users themselves, but it would at least be cool to show, for each Prediction phase, The Judges' top 10 vs The Crowd's top 10. It would be very interesting to see, in this way, how "wise" the Tongal crowd actually is. ;]
Anyway, I'm just musing here, but I thought it could be fun to discuss prediction strategies and to hear what other people think about when and how to use crowdsourcing for this kind of prediction where, unlike in March Madness*, the outcomes are based almost entirely on the subjective judgments of an unknown few.
* - Further reading:
http://blog.oddhead.com/2008/12/22/what-is-and-what-good-is-a-combinatorial-prediction-market/ 
