Reliability on Wikipedia

Fri 28 Oct 2005 11:01 PM

In a paper for MacHack several years ago, I tried to sort out the possible methods for evaluating claims found on the internet. (`Reliability on a Crowded Net' -- The conference still hosts a PDF of it.) I was primarily interested in claims made on web pages and in chat rooms, and I think the analysis extends pretty well to blogs.

Many people now use Wikipedia as their reference of first resort, however, and I wonder whether my previous analysis applies to it. In the paper, I identify four basic methods for evaluating a claim found on the internet and claim that they are exhaustive. Consider each in relation to the Wikipedia:

Appeal to Reliability involves relying on the reputation of the source. If I read an article on the New York Times website, I give it roughly the same weight as I would an article in the print edition.

It is not entirely clear how reliable the Wikipedia really is. It does have a good reputation. There is also a story to be told about the self-correcting nature of collaborative work. Nevertheless, a wiki will only be as reliable as its most persuasive members. I suspect that its reliability depends on the topic area-- because different topic areas will have different contributors.

Appeal to Plausibility involves assessing whether the general claims even sound like they are in the right ball park. This can be done both in terms of content and in terms of style. I think there is some reason to think that the Wikipedia could be deceptive in this regard. Even where it contains false information, contributors may have preened it to make it sound more plausible.

Suppose an entry contains incorrect information. If people wander through the site and make the entry sound better, even though they do not actually have any special expertise on the subject of the entry, then the entry will be written in a more plausible way than it would be if it were just the original falsities on somebody's personal webpage.

Calibration involves checking the facts where you can and extrapolating: If the source gets things right on matters you can check, then that is some reason to believe that it gets things right on matters you can't. Again, the collaborative nature of the Wikipedia makes this harder. If the things that you can check independently are the things that other people could check, then those things will probably be correct-- someone will have corrected any mistakes. The correctness on those points will fail to be evidence for the correctness on the remainder, if the background knowledge of honest and conscientious contributors runs out where yours does.

Sampling involves checking multiple sources and comparing them against one another. Insofar as one just does a quick Wikipedia lookup, one avoids sampling.

So the basic methods for evaluating credibility all get harder with the Wikipedia. The central issue is the degree to which the collective nature of the Wikipedia can be relied on to be self-correcting. How much is this a reasonable expectation, and how much is this an article of faith?