How to Identify a "Good" Scientific Journal

by Brian Dunning
June 15, 2007

Today's episode is about a subject that's as old as debating itself, and it was prompted by the following email:

I would love to look a lot of this up on my own, but am unsure about what sources can be trusted. I know you talked about how scientists are not created equal, but as an average person without the background to fully understand the primary sources or the ability to synthesize a consensus without reading meta-analyses, where can I go for reliable information?

This listener brings up a great point. If you follow any of the conversations on the Skeptalk email discussion list, you've probably heard us banter this back and forth. One guy says "Hey, leprechauns are real, here's an article in a peer reviewed scientific journal that says so," and then someone else replies "No they're not, because there haven't been any such articles in my peer reviewed scientific journals." Almost any debate can degrade into "My peer reviewed scientific journal is better than yours."

Now, a really satisfying answer to this question would be "Here, go to www.legitimate-scientific-journals.com, and you can see at a glance if your source is a credible one." Surely there must be some register like that, right? I will dash your hopes with a simple answer: No. There is no such thing as an authoritative list of reputable scientific journals. There can't be. And the reason is that word "authoritative". Who is qualified to be the authority? No one is. No one must be. The moment that any one group is anointed with the ability to declare a source to be legitimate or not, is the moment that we lose objectivity and impartiality.

It is very important to be aware that there is any number of bodies who do presume to be such an authority. Approach with extreme skepticism! The only reason anyone would compile such a list is to promote an agenda. Someone once commented on one of my episodes, and they tried to shoot me down by pointing out that one of my sources was discredited on a website called sourcewatch.org. Sourcewatch.org, sounds pretty legitimate, sounds like they do good work, sounds like they're out there looking out for our best interests by rubber stamping some sources and discrediting others. But according to critics, Sourcewatch is a two-man operation that endorses only publications following their own narrow political bias. This is a perfect example of what you should expect from any source that attempts to identify itself as a rubber stamping authority. Be skeptical of any group you find whose purpose is to identify reputable journals.

As long as we're throwing around the word reputable, I might as well give the somewhat disappointing answer to the listener's question, and tell where you can find a reliable journal. Scientific journals achieve their status only through long histories and good reputations. To be broadly accepted within the mainstream scientific community, a journal must have established a long history of responsible reporting, good quality articles detailing well performed research, and exhaustive peer review. Long standing reputation among the scientists who matter the most. If you're not one of those scientists, it can be difficult to know which research is good, which editors and referees are good, and which journals have a long history of publishing them in good standing. For this reputation to have any meaning, it must stand on its own and not be supported by appearing on some simple list. Unfortunately, listener, you just have to know; but I will give you a starting point in a moment.

While it is essential that good journals be peer reviewed, you should be aware that almost every publication hoping for prominence describes itself as peer reviewed. When you hear someone defend their source by stating that it's peer reviewed, be skeptical. By itself, that's meaningless. Think back to our old example of the guy writing a UFO newsletter in his basement who has a couple of his UFOlogist buddies endorse his writing. Suddenly he's "peer reviewed". This is not the type of peer review that carries any meaning within the mainstream scientific community, since the peers have a clear agenda and have not established long histories of scientific acumen by the legitimate community at large. This is an extreme example, but it does accurately portray a lot of what's out there. When you don't know anything else about a journal, the fact that it calls itself peer reviewed cannot, must not, be allowed to carry any weight.

One source that a lot of laypeople are turning to these days is Wikipedia. What about Wikipedia? It's new and it's a very different animal from anything previously available, and is something of a paradigm shift. Wikipedia is not perfect, but it is generally very good. Its principle weakness, so often pointed out by critics, is also its principle strength. Critics of Wikipedia love to charge that any old Joe Blow can go in there and edit any article to say whatever the heck he wants. And this is true, to a point; but they do have multiple layers of redundant checks and balances in place. Every topic has editors, and every edit eventually makes it past several sets of eyeballs. Every article is read by untold zillions of eyeballs, and tempered with suggested edits by many of them. Most of these suggestions are good, and some of them are bad. The volunteer editors include some of the foremost authorities on the subject, and they include crank nitwits, and everyone in between. Wikipedia has tens of thousands of regular editors, over a thousand administrators who enforce the behavior of those editors (eventually weeding out the crank nitwits), and even a judicial committee which resolves any disputes that are not otherwise handled by the process. The underlying software provides a powerful nerve center from which the editors and administrators can track history and changes. This open source process leads to an inevitable effect: Many Wikipedia articles end up being the closest thing to an authoritative consensus that we have on a given subject. Each article continually improves over time until it becomes what Wikipedia describes as the "ideal" article: "balanced, neutral and encyclopedic, containing notable, verifiable knowledge."

When Wikipedia was first conceived, it was a brand new idea that had never been tried on such a scale. No doubt, it had plenty of growing pains. But they've had years to improve the system. They've been dragged through the media more than once over high-profile errors resulting from vandalism, and every day since inception, they've worked to address those loopholes. The process still isn't perfect, but it's one of the most amazing compendiums in human history.

So, I'm going to give our listener a simple answer to his simple question. Start with Wikipedia, or any other encyclopedic resource with a good reputation like Britannica or MSN's Encarta. Nearly always, good articles will include links to additional references (especially in Wikipedia), but these links are of tremendously varying quality. Be careful of their external links, and carefully note why each external reference is being cited. Good articles will often include a section on criticism or skepticism of the subject. Read it.

Note that I'm no doubt going to be criticized for pointing laypeople toward Wikipedia as a starting point for research, mainly due to the usual criticisms of Wikipedia. But, as I said before, Wikipedia's weakness is also its strength, and I do stand by this recommendation, especially for laypeople of a given subject who don't otherwise have the experience to choose a good starting point.

What about identifying which scientific journals are reliable? Since we're not all scientists in the chosen field with the education and experience to know which are the most reputable publications in our field, we need some kind of list. But, as we've discussed, lists are bad things when they come from a source with an agenda. So we turn again to our source with no agenda, Wikipedia. Search Wikipedia for "List of Scientific Journals" and you'll find that they have a page listing a few hundred reputable journals in most scientific fields. Generally, this is an excellent list. The fact that it comes from Wikipedia, and is constantly being revised for objectivity and quality by experts in each field, is its strongest recommendation. Some fields are not listed, and most subsections are partial. You can drill down to find more. But beware: the further you drill down, the broader the quality control becomes, and the more journals of lower repute are included, and more non-scientific fields are listed. If you use this list to gauge the reliability of a source that you see referenced, you are in as good of hands as are available to the inexperienced journeyman; but you must use the list wisely. Stay at the top level, or as close to it as you can. With each click that you drill down, reputation for the listed journals is generally lower.

Again, this recommendation will no doubt be criticized, and the criticism is generally valid. But I maintain that for the average guy off the street, this is the best way to gain a "good enough" grasp of a journal's quality, and to find good research that truly represents the current scientific consensus.

By Brian Dunning
Follow @BrianDunning