In my last post I described how we are using a Ruby DSL for describing review patterns in PeerPigeon. However, we need to record how each artefact gets distributed between the review participants.
We’ve decided to record this using the notion of transforms: at each review cycle, a participant takes an existing artefact and produces a new artefact based on the existing one. For example, a reviewer takes a paper and transforms it into a review.
In our Ruby DSL, this might look like this:
c.transform :paper_1, 2, :review_1
We define a transform in the cycle, where :paper_1 is transformed into :review_1 by participant 2.
For the simple review scenario from my previous post, each reviewer reviews someone else’s paper. The easiest way to model this is to simply pass each paper along to the next person, for example in a group with person 1, 2 and 3:
- 1’s paper is reviewed by 2
- 2’s paper is reviewed by 3
- 3’s paper is reviewed by 1
In our Ruby DSL:
cycle :review do |c|
c.description 'Write Review'
c.deadline '2 days from now','4 days from now'
c.transform :paper_1, 2, :review_1
c.transform :paper_2, 3, :review_2
c.transform :paper_3, 1, :review_3
end
In our experiments we have seen that this simple mechanism copes with a range of different peer review scenarios. So we will use it to model the distribution of material between participants in the PeerPigeon system.
However, it is crucial that these transform lists do not have to be manually generated. If we now wanted to run this simple review on a group of 100 people instead of 3, we would have to write 100 transforms! So we must have a way of describing these distributions programatically.
Luckily as we are working with a Ruby DSL, we can define a block of Ruby code to do this:
c.distribution :reviewers, lambda { |group,person|
c.transform
group.wrap('paper', person, -1),
person,
group.wrap('review', person, +1)
}
This defines a distribution pattern in the cycle that is applied to the reviewers (:reviewers). The distribution code block takes a group and a person as an argument, and is run on each each reviewer in the group. For each reviewer, you create a transform as before. The input/output of the transform (e.g. :paper_1, :review_1) is calculated using the group.wrap method with either the next (+1) or previous person in the group (-1). For example, in a group [1,2,3], calling group.wrap('paper', 1, -1) will return :paper_1.
Using this technique we can apply the review pattern to an any sized group. We can also define more complex distribution patterns in Ruby code. However, it may be possible to tweak the Ruby DSL to provide a cleaner syntax for these.
