peer-review system to recognize contributors

[Bayle Shanks]

> On Fri, May 28, 2010 at 12:39 AM, Bayle Shanks <bshanks.list at gmail.com> wrote:
> > Hello, in a few months I'm starting a company to help projects fairly
> > recognize or reward all contributors based on peer review. In addition
> > to recognition, contributors could potentially be rewarded with money
> > from a corporate sponsor, or with a greater say in the project.
> 
> I assume you're aware of the various research on the impact of money
> on intrinsic motivation? You might start with
> http://papers.ssrn.com/sol3/papers.cfm?abstract_id=203330 but see also
> http://www.sciencemag.org/cgi/content/abstract/314/5802/1154
> http://ideas.repec.org/p/zur/iewwpx/007.html
> http://www.sciencemag.org/cgi/content/abstract/320/5883/1605
> 
> Luis

Thank you for mentioning that, and for the links. I had seen one of
these articles before, but not the others. Before talking about
whether giving money to open-source contributors would decrease their
motivation, I'd like to note that peer-review could also be used just
to recognize contributors, or alternately to reward them with a
greater say in the project, neither of which involve money.

SUMMARY

In some contexts, extrinsic motivation decreases results, and in
others, it increases them. As the concluding paragraph of (1), page
19, states, "Crowding effects thus are an empirically relevant
phenomenon. But it does, of course, not always prevail over the
traditional relative price effect." I am not an expert in this
literature, but it seems like the sort of thing where even an expert
might frequently be surprised by which context produces which result.
Therefore, it seems to me that, if the cost is not too great, it is
worth trying out various motivations in various contexts and seeing
what happens.


DISCUSSION

Some evidence for my viewpoint, as applied to the present context, may
be found in the papers you linked to. For brevity, I will refer to the
papers you mentioned as (1), (2), (3), (4), numbered in the order that
you gave them.

PAPERS (1), (3), and (4): SOMETIMES ADDITIONAL EXTRINSIC MOTIVATION
DECREASES OVERALL MOTIVATION, BUT OFTEN IT DOES NOT

There are three possibilities.

One possibility is that intrinsic motivation is decreased, but
the increase in external motivation outweighs the decrease in internal
motivation. This possibility is discussed on pg. 5, section (c) of (1)
(by "results", i mean what (1) terms "net benefit"). (4) says,
"...where the effect of incentives is merely blunted rather than
reversed, surprisingly, either greater or lesser use of economic
incentives may be optimal."

On page 1 of (1), it is noted that the relative price
effect, by which a higher reward induces more work, was the only known
effect economic theory for quite sometime (in fact, on page three this
is called "the most important economic law"). In other words, this
situation seems to be the rule, and the other situation (when
increasing reward decreases results) seems to be the exception (i.e.
the less common situation), although it's hard to be sure.

So, in the absence of other evidence, I would guess that in a given
situation, increasing reward would lead to an increase in results.

A second possibility is that intrinsic motivation will be increased by
an increase in external motivation. This possibility is discussed on
page 5, section (a) of (1) as "the crowding-in effect". Various
examples are given in (1) and (4).

Of course, a third possibility is that increasing reward will lead to
decreased results. Indeed, (3) showed that small amounts of pay lead
to less volunteer hours given to political organizations. This is
troubling as it is the most similar context to open source in any of
the studies that cited by Luis (although; why did this paper restrict
their analysis to political organizations, when their data set
included other nonprofits as well? Page 10 of (3) suggests that they
did not expect to find a demotivational result in other nonprofits (in
fact, given that they were able to report the percent of non-political
volunteers who received money, perhaps they may have entered
all the data, found no effect in general, and then redid the analysis
by sector and reported only the one sector which showed an effect).
Perhaps the volunteering for a non-political nonprofit is even more
similar to open-source than political volunteering). As noted below,
there are reasons for thinking that would not be the case here, but I
don't think we can really be sure until we try. Decreasing results due
to tainted intrinsic motivation would be unfortunate; in some of the
studies, once intrinsic motivation was decreased in a particular
actor, it did not increase again even after the extrinsic motivation
was removed. Of course, assuming an open source project did not die,
eventually other untainted contributors would join the project and
replace the demotivated ones.

 Which will happen here?

I note that, insofar as it may be predicted whether crowding-out (when
intrisic motivation goes down) or crowding-in (when it goes up) will
occur, that all of the contexts of

 (a) increased recognition, with no rewards 
 (b) after-the-fact payments to independent volunteers
 (c) increased voice in the project

in open-source projects appear to meet the criteria on page 7 of (1)
when crowding-in (the good one) is expected. Specifically, I expect
that each of these would often be perceived as supportive, not as
controlling, and increase self-determination and self-esteem as
discussed on pages 6 and 7 of (1). This seems clear in (a) and (c); in
(b), I should explain my position further: there is no boss telling
you, "do this or you're fired", rather, you do what you want and then
perhaps the community expresses its thanks (this is assuming that a
community using (b) does not try to command individuals to work on
what is popular rather than what the individual wants to work on; the
peer review would be used by the community to express gratitude after
the fact, not to control). You are not being controlled or belittled;
you are free to choose what you work on, and you are being
appreciated.

It is also worth noting that perhaps different individuals are
motivated in different ways. I would not be surprised if, in a
situation where contributors to open source projects were given
increased recognition or rewards, some people who currently contribute
would contribute less, but that other people who do not currently
contribute at all would join. Which of these factors would dominate
remains to be seen.


PAPER (2): MONEY INDUCES A SELF-SUFFICIENCY ORIENTATION, BUT IT IS
UNCLEAR IF THAT IS HELPFUL OR HARMFUL

Regarding paper (2), it addresses what are the psychological effects
of thinking about money, excluding motivation (it does not address
whether or not output is increased or decreased). It states, "We
tested whether activating the concept of money leads people to behave
self-sufficiently, which we define as an insulated state wherein
people put forth effort to attain personal goals and prefer to be
separate from others. The term as we define it does not imply a value
judgment and encompasses a mixture of desirable and undesirable
qualities, which may help explain the positive and negative
consequences of money." Some examples of what they mean by a
self-sufficiency orientation are working longer on a task before
asking for help, spending more time on one's own task and less helping
someone else, and choosing to spend leisure activities in a non-social
way. It seems to me that it is hard to predict whether a
self-sufficiency orientation would be harmful or beneficial for
open-source projects.

For example, perhaps people would waste time trying to figure out
things alone when they should ask for help, be less helpful, and be
less inclined to contribute to a project at all. On the other hand,
perhaps people would save others' time by asking for help less and
figuring out things on their own, spend more time working and less
time talking, and be more inclined to sit in front of their computer
working on open-source as opposed to other more sociable leisure
activites.

Also, (2) purposefully does not address whether motivation and output
are improved by money. Even if a self-sufficiency orientation was
harmful in itself for a project, this might be outweighed if it was a
small effect, but at the same time the increase in output was large.

MY OWN BIASES AND INTERESTS

I should note that I tend to answer, "try it and see" when something
is not very certain.

Even without the benefit of increasing motivation, I would still be
interested in using a peer-review system in open source projects
because this seems to me to be a necessary prerequisite to something
else I am interested in, namely democratic decisionmaking without
central authority in projects that anyone can join.


CONCLUSION

Until about a decade ago, the consensus assumption apparently was that
raising rewards would raise output. Recently, some situations were
discovered where the reverse holds. We do not have enough data to
conclude that any of [recognizing, paying, giving increased voice to]
contributors of an open-source project via a peer-review evaluation
system will decrease output; and, in the absence of such data, it
seems to me that a reasonable guess is that this situation is not
exceptional and that raising rewards would raise output.


Regards,
  bayle