program selection system

We’ve organized a coding contest last year (July 2007) in Linz, Austria (see http://www.catalysts.cc/index.php?id=contest) which targeted the following questions:
* Who is faster? Who is more efficient? Who is more effective? Who is more productive?
* Does “Pair Programming” really make you faster?
* Does “Test-Driven Development” really lead to fewer bugs?
* How many ways are there to solve a problem?
* Is it right that good developers only write 30 lines of code for which mediocre developers need to write 300 lines?

We had more than 50 participants (students and practitioners).
We used a special-purpose test server for distributing the problem statement / the assignment to the participants, for tracking their progress, and for collecting their solutions.
The results are currently being elaborated in a masters thesis.

In the mean-time we have organized several such contests in Nepal and we will organize one at the XP conference in Limerick this year.

What we like about these coding contests is:
* The participants can view it as a game - trying to solve a puzzle as fast as possible, playing through the various levels of the problem statement (just as playing some online game), getting some visibility or even money prizes.
* The participants can compare their progress with that of other particpants - tagging themselves with groups (Java / C# / Ruby, student / practitioner / expert, … all freely chosen by the participants).
* We can set it up as an experiment, dividing the participants into several groups (control group and test group), we can collect the source code snapshots at regular intervals, the progress information about the participants (which level they have mastered already, how many acceptance test errors they have), visualize the evolution of the good and bad solutions.
* We can evaluate the results and draw conclusions.

So it can be seen as “just a contest” but also as an experiment that can help us to underline our hypotheses with some numbers.

Of course, it is possible to remain completely anonymous as a participant.

Responding to the comments below, I’ve taken some screen shots (but remember, the proposal is not about the tool, we never got any feedback “on the tool” from the participants, but a lot of feedback on the contest / experiment):

Entry Screen On the entry screen you see which training games are available and which contests are scheduled. A mindmap explains what you can do.

Contest Details You can have a look at the summary information of the contest, who has registered, and you can have a look at the source code of those who completed the contest.

Playing a Game Once you’ve selected a game / a contest, you are can play through the levels of the game, look at the description, develop the solution, request test cases, submit your results and … kill your bugs or move ahead to the next level.

You can define groups yourself or simply join a group that someone else has defined (like we will predefine groups for “Pairs” and “Individuals”.

You can look at the overall progress or select one of your groups to compare with the other group members (beware, this may create some pressure on your shoulders, since you see - in real-time - how you do compared to the others).

Drawing Conclusions

We’ve had 56 participants last year at our first coding contest in Linz, Austria (18 people under 20 years, 20 students, 18 practitioners). That’s not yet representative, but it’s already a good mix and more than most academical studies can ever get.

We will have 50-100 students participating at our coding contests in February/March in Nepal.

We hope to get a lot participants at our coding contest at the XP conference in Limerick, Ireland.

We will repeat the coding contest in Linz in July.

Last year, we also thought about running the coding contest virtually around the world via SecondLife. We had too few registrants for that, but we will give it another try this year.

In total, that may not be representative for the whole world, but it may help us a little to understand ourselves better.

Running the contest throughout the conference

Gil Broza raised the question why not run the contest throughout the conference. No problem from our side.

Except that it makes a psychological difference when you sit in a room with others working on the problem concurrently, when the first has played through the entire game, etc.

And we will allow the users to upload their source code, so it would be easy to cheat (by using someone elses solution as the basis) - but we’re part of a trustworthy community, aren’t we ;-)