At the end of October, uBiome put out a press release describing their product, SmartGut. The press release sounds a bit too good to be true. For example, “SmartGut empowers patients to take steps to understand their gut health and assess actionable information about their gut microbiome with their doctors”. The manuscript has been submitted to a peer-reviewed journal and conveniently, posted to bioRxiv as a pre-print. Nothing in the pre-print supports this and it is unclear how the test changes or actually helps the diagnosis of the “dozens” of microbial populations (N=3 dozen). They also claim that “SmartGut is covered by US health insurance for the majority of patients.” This is surprising since it does not appear to have FDA approval. Considering they were nice enough to make the manuscript public and I was not asked to be a formal reviewer for a journal, I figured I’d post a review. I am doing this for a few reasons…

  1. I would like to improve the quality of the paper and ultimate product. As written, I don’t think the paper, the method, or the press release help advance microbiome science.
  2. I’ve never posted a comment on a pre-print and feel the need to help lead the field field in this regard. If this paper sits in bioRxiv without any comment then I worry that might be taken as approval.
  3. I’m interested in getting feedback on my approach as a peer-reviewer.

This review is also posted in the Disqus comment box over at bioRxiv.

To be clear, I was not asked to review this manuscript by a journal and have no connection to uBiome. This review has been cross posted at http://www.academichermit.com/2016/12/06/Pre-print-review-of-SmartGut.html…

Almonacid and colleagues describe the use of 16S rRNA gene sequencing as a clinical diagnostic tool for detecting the presence of bacteria and archaea commonly associated with fecal samples in health and disease. On the whole, the method is not novel in that many people have been doing 16S rRNA gene sequencing of samples for many years now. The potential novelty of the manuscript is that it attempts to place the value of this technology in a clinical diagnostics rather than exploratory setting. The potential impact of this paper is reduced because it is more of a proof of concept rather than a comparative demonstration relative to other methods. Overall, the methods are poorly described and there are a number of overly generalized claims that are not supported by the literature or their data. The most glaring problem is that the authors assume that the presence of a V4 sequence that is identical to that of a pathogen is proof for evidence of the organism.

Major comments

  1. L16-18, 43-51. I’m curious whether the authors actually have citations to back up the primacy of manual culture-based methods in clinical diagnostic laboratories or their limitations. My understanding is the much of clinical diagnostics is highly automated and while it may use some amount of cultivation, the actual analyses are quite modern. The authors at least need to recognize the high levels of automation and use of qPCR, ELISA, and mass spectroscopy-based approaches in most diagnostic labs. In fact, the authors later use one of these methods, Luminex‘s xTAG Gastrointestinal Pathogen Panel to help develop the panel of organisms used in their own method. The authors’ new method may be novel, but they should portray its novelty using a relative modern comparison rather than a straw man. The manuscript would be considerably strengthened by comparing the Luminex method (or any other method) to the current method.

  2. The authors have tested whether they are able to distinguish distantly related pathogens, but have not done due diligence in determining whether the approach can distinguish pathogenic and non-pathogenic organisms. As an example, they state that “the pathogen Peptoclostridium difficile is found in ~2% of the healthy cohort which shows that asymptomatic P. difficile colonization is not uncommon in healthy individuals (L211).” This statement is emblematic of a number of problems with the authors’ analysis. First, the presence of P.difficile/C.difficile does not mean that it is in fact pathogen as there are many non-toxigenic and, thus non-pathogenic, strains of this organism - the V4 region is simply not a virulence factor. Second, there is already a toxin-based assay for toxin-producing strains that is likely more sensitive and specific than this sequence-based approach and much cheaper for this and other pathogens. Third, the V4 region is only about 250 nt in length. There is always the risk that closely related, but different organisms may have the same sequence and that the same organism may generate different sequences because there is intra-genomic variation. When I used blastn to compare the region of the P. difficile sequence in Table S2 that would be amplified by their primers to NCBI’s reference 16S rRNA gene sequences, it returned two additional P. difficile strains (JCM 1296 and ATCC 9689) that are identical to each other but 1 nt different than the sequence in Table S2. It is interesting that none of the sequences in the NCBI reference were an exact match as required by the current method. When I performed a similar analysis using the author’s E. coli/Shigella sequence, it matched multiple Escherichia and Shigella strains, most of which were not pathogenic. Based on all of this, I am not sure how much utility a clinical diagnostic laboratory would gain from using this method over others. None of these points are considered in the authors’ discussion.

  3. The authors lay out a “healthy reference range” for each of their 28 targets (L199-210). I worry about such a claim, when really the authors are likely only defining an operational healthy range so that they can optimize the sensitivity and specificity of pathogen detection. Claiming a healthy range as they have assumes that the subjects are truly healthy (there is no indication of whether the subjects were honest in self-reporting) and that the microbial communities did not change between collection and analysis. To this second point, the Methods are poorly described and validated. Specifically, I am unclear what “specifications” were laid out by the NIH Human Microbiome Project that would be relevant for this method (L100-102). Furthermore, what is the composition of the lysis and stabilization buffer that allows samples to be stored at ambient temperatures. The authors need to either provide data or a reference to support this claim including evidence that the community composition does not change. All this is necessary to report for others hoping to repeat the authors’ work and for improving the clarity of the writing.

  4. I am impressed by the authors’ ability to quantify the relative abundance of these strains using PCR and sequencing. This runs a bit counter to the prevailing wisdom that there are PCR biases at work that would skew the representation of taxa such that the final proportions are not representative of the initial proportions. I’m a bit confused by the description of the experiment. Namely, what was the diluent DNA that is mentioned in the Methods (L142)? Although the quantitative results are impressive, I am a bit concerned that the authors used DNA fragments that overlap the V4 region of the 16S rRNA gene rather than genomic DNA.

  5. Similar to the previously described concerns regarding the methods description, the list of accessions in the curated database that is described should be made publicly available since this is a critical component to the method (L171-185). More details are needed that describe how this database was created. The manuscript states “After optimizing the confusion matrices for all preliminary targets…”, but it is unclear what “optimizing” means and what was altered to generate better performance. Furthermore, I am curious whether uBiome paid for a license to use the SILVA reference. Unlike many other references, this is not a database that is free for non-academic usage (https://www.arb-silva.de/silva-license-information). Considering they are a for-profit company and are likely to commercialize this, they may want to consider a database that is more public. That being said, I don’t know why the authors would need to use the SILVA reference since they are not making use of the alignment, taxonomy, or metadata features contained within the database.

Minor comments:

  1. L78-80. “Regularly evaluating the microbiome to monitor overall health is therefore gaining traction in contemporary medicine and needs to be part of modern diagnostics.”

  2. L102-109 include no citations. Although these may be “standard protocols”, specific protocols should still be cited as there are no standards and to give credit to those that developed the protocols.

  3. L112-125. The authors present a method for denoising and building contigs from their sequence data that uses Swarm. As far as I know, this approach to denoising the data is novel and has not been validated in this paper or others. Alas, I’m not sure why they bothered with the Swarm clustering since they take the contigs and map them against the SILVA reference database for exact matches. The justification for these two steps is not clear and needs to be clarified.

  4. L154. “Two out of 35 control samples did not pass our sequencing quality thresholds”. If I am right in assuming that this is previously mentioned 10,000 sequence threshold (L129), then the authors should be specific in stating that here. If there are other thresholds, then those should be stated at some point in the manuscript.

  5. “dysbiosis” is used throughout the manuscript. This is a trendy piece of jargon that is pretty meaningless. Furthermore, their method does not really address the whole community, which is usually done when describing a dysbiotic state. This manuscript describes the quantification of single strains.

  6. I do not believe that Peptoclostridium difficile is a valid name for Clostridium difficile. At this point, it appears that the most recent valid name is Clostridioides difficile (http://www.sciencedirect.com/science/article/pii/S1075996416300762).