Benchmarking

Benchmarking exercises are important to assess the performance, and reliability of the available bioinformatics tools which have different complexity in design and function. The purpose of a benchmarking exercise is to assess the ability of the bioinformatics tool to provide reliable analysis of AMR gene content.

An inter-EURL working group has prepared a guidance document presenting how to benchmark the different steps of WGS. This EURL guidance document of WGS benchmarking is currently a draft version. It will be available for download here after approval.

An inter-EURL working group has prepared an opinion paper on the challenges of designing a benchmark strategy. The specific objective of this benchmark strategy is to challenge the bioinformatics step of a workflow to identify antimicrobial resistance in samples by using WGS. This paper is available here:

• The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies

As part of the ENGAGE project, several benchmarking exercises of tools for sequence analysis was performed. This includes benchmarking for assembly tools, serotype prediction, genotypic detection of AMR and phylogeny. The benchmarking reports are available for download:

• ENGAGE Benchmarking for Campylobacter coli phylogeny (PDF document, 800 KB)

• ENGAGE Benchmarking for Salmonella Enteritidis phylogeny (PDF document, 500 KB)
  
  
• ENGAGE Benchmarking of de novo assembly tools: SPAdes 3.9 vs Velvet 1.2 (PDF document, 300 KB)

• ENGAGE Benchmarking of genotypic detection of antimicrobial resistance (AMR) genes (PDF document, 200 KB)

• ENGAGE Benchmarking of genotypic Salmonella serotype prediction (general) (PDF document, 200 KB)

• ENGAGE Benchmarking of genotypic Salmonella serotype prediction complying to the Draft International Standard ISO 16140-6 (ISO/DIS 16140-6:2017 Microbiology of the food chain – Method validation – Part 6: Protocol for the validation of alternative (proprietary) methods for microbiological confirmation and typing procedures) (PDF document, 300 KB)

In the study below, a novel mapping method was developed and benchmarked to two different methods in current use for identification of antibiotic resistance genes in bacterial WGS data:

• Benchmarking of methods for identification of antimicrobial resistance genes in bacterial whole genome data