Many important issues in the news today are built around technical information, particularly in scientific and engineering fields. Subjects such as resource exploitation, energy generation, and climate and weather are of interest to many people – but most of us do not have the technical background to understand their inner workings. So we turn to trusted experts to tell us what is important, and to help us figure out what we should believe and support.
Many publications – reports, articles, and newsletters – aimed at policy-makers and the public use the term “peer reviewed” to indicate that they have been approved by experts and should be accepted by the public. “Peer review” is used as a kind of scientific seal of approval.
But what does “peer review” actually mean? Does it help us distinguish objective truth from overt promotion? Let’s have a look.
“Classical” peer review
Peer review has long been an important component of advancing scientific thought in a reasoned, credible fashion. Ideally, when an article describing new research is submitted for publication, an impartial editor selects independent experts in the field of study to review it.
Reviewers look for clear definition of a problem, reasonable methodology, reference to and context from previous studies, introduction of relevant data, cogent analysis, and clear statement of conclusions. They are charged with identifying problems – unclear goals and definitions, missing references to relevant work, poor methodology, inadequate or erroneous data, inadequate analysis, and unsupported conclusions. They suggest revisions, and may recommend that the paper be accepted with minor or major revisions, or simply rejected if it does not offer scientific value.
The editor compiles reviewer findings and communicates them to the author(s), telling them what changes must be made in order to meet the standards for publication. The exact process varies, but is expected to be impartial, critical, and honest, resulting in a publication that advances knowledge and can be relied upon and referenced. Many reputable journals offer experts not involved in the original peer-review process the opportunity to weigh in with “Discussion” articles, where dissenting opinions may stimulate further thought and discussion.
Peer review in 2024
Today, important scientific research appears in a wide variety of journals, books, special reports, and articles published not only by universities and major scientific publishing houses, but by organizations dedicated to specific fields of research (e.g., Geoscience BC). Geoscience BC and like organizations generally follow the classic model of peer review to ensure quality and value of results from the research they sponsor.
But there are no official standards for peer review. In fact, there is a whole spectrum of organizations publishing articles and reports claiming to be “peer reviewed” – but without making clear who the scientific peers are, what qualifies them to do reviews, and whether they are actually impartial. Even if the reviewers selected are indeed qualified, there is often no editorial process in place to ensure that real peer review has taken place.
Let’s look at one example.
Drilling into the Montney
The David Suzuki Foundation, an advocacy group with the mission statement “to protect nature’s diversity and the well-being of all life” recently published Drilling into the Montney – How LNG Expansion will Accelerate Drilling, Fracking and Environmental Impacts in Northeast B.C. and Adjacent Alberta. The author is David Hughes, an earth scientist (i.e., he has post-secondary training in geology but is not a registered professional geoscientist) with knowledge of some aspects of the coal and oil and gas industries.
DSF describes the piece as a “peer-reviewed report provid[ing] an expert and detailed examination of what’s at stake in the Montney.” They name two contributors and reviewers (an American geologist and an Alberta professional geophysicist), and reference two anonymous reviewers. There is no description of the review process.
So is this a peer-reviewed article in the true (classical) sense of the term? Can industry professionals, policy-makers and the public rely upon it to advance knowledge and understanding in an impartial, scientific fashion?
No. Absolutely not.
Why? I will speak to that as a professional geologist with a PhD in sedimentary geology, more than 40 years of experience working in oil and gas, and more than 15 years working many aspects of the Montney play. I have been on both sides of the peer-review process (author and peer reviewer) many times, and had this report been submitted to a reputable scientific publication, I might well have been asked to peer-review it.
Drilling Into the Montney fails to meet any reasonable standards of impartiality. There is no evidence that reviewers have applied proper peer-review standards to ensure clear definition of a problem, reasonable methodology, reference to and context from previous studies, introduction of relevant data, cogent analysis, or a clear statement of conclusions.
And there are many errors in the article that were not addressed in whatever review process that actually took place.
Let’s look at some examples of what an impartial, expert peer reviewer would identify and recommend (appropriate peer-review comments in italics).
- Goals of the report and issues to be addressed are not laid out until page 10; they focus on specific aspects of Montney field operations and environmental impacts.
Recommendation: Lay out goals and issues at the beginning of the paper with only necessary introductory context. A condensed version of the Introduction on Pages 9 and 10 is more than sufficient for context. - The first six pages of text (Foreword, Summary, and Introduction) before the goals statements contain unscientific, unsubstantiated comments speaking not to context but to political/advocacy bias. Examples include:
- [the Montney is] the largest carbon bomb in Canada
- the liquefied natural gas industry has been spending millions of dollars on ad campaigns, marketing, branding and lobbying.
- the LNG industry flogs the discredited claim that exporting LNG can help countries in Asia get off coal
- a massive oversupply of LNG is now being developed
- the well site to the burner tip can be equal to or worse than coal [in terms of GHG emissions].
Recommendation: Delete all comments not relevant to the stated goals of the paper, including those referencing advocacy positions and policy issues. Refrain from making judgmental statements relating to industry or other organizations. Do not make definitive statements such as the fourth and fifth points above based on selective sampling of literature while ignoring abundant evidence to the contrary.
- Figures based on satellite imagery (Fig. 1, 2, 4, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, and 26) are too small and have insufficient resolution to illustrate the points they are designed to support.
Recommendation: Redesign map figures to be larger and clearer. Consider reducing the number presented to illustrate key points succinctly, as many appear to be repetitive.
- Analysis of Montney production addresses only raw gas (Fig. 5), and from that metric Alberta wells are deemed to be less productive than BC wells. But gas liquids production is the primary economic driver for much of the Montney, so both gas and liquids production must be considered. The Alberta Montney is generally much more liquids-prone than the BC Montney, driven by geological fundamentals.
Recommendation: Present complete (both gas and liquids) production statistics, and address their importance in economic analysis.
- Regarding Montney resource potential (p.15), there is no reference to support the statement that areal extent of the Montney play has been reduced. There have been numerous developments in the play fairway since the 2013 resource report that have not been systematically tabulated in public literature. There is no support for the statement that the ultimate marketable gas potential from the 2013 report may “prove to be overly optimistic”.
The implication is made that Montney reserves should support all LNG projects. No mention is made of vast gas resources tested and proven in the Horn River and Liard basins to the north that may also supply future LNG feedstock.
Recommendation: State Montney reserve and resource values succinctly and with clear references. Avoid unsubstantiated speculation on what resource volume changes might occur in the future, and note that in the context of future LNG supply there are additional large gas resource volumes in the Horn River and Liard basins.
- Well productivity and decline analyses are complex, multifaceted subjects requiring reservoir engineering expertise and a deep understanding of the evolution of well design and execution. The summary presented (Section 4, p.16-35) is incomplete and does not support the conclusion presented that “much of the cheapest gas has been extracted and higher drilling rates will be required to maintain or grow production in the future”. Inadequacies are compounded by failing to consider gas liquids production, as noted previously.
The point of the “Areal Variability” (Section 4.3) discussion is unclear, and is illustrated using illegible figures. The value of cursory review of historical production in different geographical regions is not explained, particularly given that future production for LNG will be drawn from specific lands dedicated to LNG projects by their operators. Much of this acreage has not been intensely developed to date, as the dry gas it will produce is far better suited to LNG than to current domestic markets.
Recommendation: If production analysis is to be undertaken, engage the proper expertise and data to complete it. To address future productivity potential in the Montney fairway, consider how liquids production has influenced drilling and completion strategies to date, and how dry gas-focused production for LNG supply would differ from historical patterns.
Present information and interpretations to address specific questions and objectives, and use them to support clear and specific conclusions.
- Section 5 – Projections of Production and Environmental Impacts through 2050 – is incomplete and simplistic, and relies on a small set of future production scenarios by the Canada Energy Regulator. The CER itself notes that their studies present only a few scenarios, and that many other production scenarios are possible, including contributions from other unconventional plays such as the Horn River and Liard basins.
The faulty production analyses discussed above, particularly in ignoring gas liquids production, invalidates the entire discussion – particularly the suggestion that CER projections are too optimistic for number of wells to be drilled.
Assessment of land surface disturbance is simplistic, failing to consider evolving trends of number of wellbores per drilling pad, and the potential to develop multiple stacked productive zones within the Montney using existing roads, pads and pipelines after depletion of the first zones developed.
Water consumption statistics are presented in a confusing fashion, and do not address current and projected future levels of frac water recycling, particularly in areas that will be developed intensively to supply LNG feedstock.
Recommendation: This section is grossly incomplete, and does not present credible perspectives on the potential range of Montney (and other) development to support LNG production. If such a section is to be included in this report, it must be developed by engineers familiar with Montney development and using complete datasets.
- Section 6 on Implications for LNG fails on the inadequacies of the preceding sections, which do not provide sufficient background to support the conclusions presented. As well, it addresses GHG emissions of Canadian LNG in the context of Canadian emissions scenarios only, ignoring global energy supply and global emissions perspectives.
Recommendation: This entire section should be omitted. It relies on incomplete and in places misleading analyses in earlier sections of the report, and fails to consider global energy security imperatives that are rapidly emerging as equally or more important than net-zero emissions aspirations.
Conclusions
As scientific investigations and findings generate more headlines and attract more interest among the general public, people want to know whether they can trust articles presented as scientifically valid studies – and whether they should believe the conclusions.
Peer review is a time-honoured mechanism within the scientific community to support accurate, impartial reporting that builds appropriately on existing knowledge. It remains important today in assuring quality of work in scientific journals and publications by reputable research organizations.
But the term “peer review” is being co-opted by organizations looking to promote their particular viewpoints over impartial science. They slap the label “peer reviewed” on their publications as a badge of quality and validity without actually undertaking appropriate peer-review processes. That is a big problem for everyone involved.
I have subjected the latest David Suzuki Foundation report on the Montney Formation and LNG production in Canada to a true peer review, albeit rather cursory compared to what would be expected if I were reviewing it formally for a scientific journal. It fails true peer-review standards on almost every front – lack of impartiality, incomplete datasets, inadequate analysis, and unsupported conclusions.
If I were reviewing this for a journal, I would recommend it be rejected without the opportunity for revision, as it does very little to advance real knowledge.
And that is the only test that counts.
