News

Oct. 1, 2013
Details about the CASMI 2013 Special Issue and dates are now available!

Sept. 24, 2013
The rules and challenge data pages have been updated.

Sept. 2, 2013
The CASMI 2013 Challenges have been officially released!

August 29, 2013
The challenges for CASMI2013 will be released on Monday, September 2nd!

August 29, 2013
The CASMI 2012 poster will be presented in Langenau in November 2013


Results in Category 2

Summary of Rank by Challenge and Participant

For each challenge, the rank of the winner(s) is highlighted in bold. If the submission did not contain the correct candidate this is denoted as "-". If someone did not participate in a challenge, nothing is shown.

birmingham cruttkies hshen metfusion oberacher schymane
challenge1 1 45 5 1 1
challenge2 - - 1 30
challenge3 - 21 - -
challenge4 - - - 195
challenge5 4 275 5 386
challenge6 - - 11 25
challenge10 - 302 - 307 - 63
challenge11 - - - 3
challenge12 - - -
challenge13 - 5 - 64 1 3
challenge14 12 39 - 1 - 22
challenge15 - 316 - 1 1 26
challenge16 2585 - 1562
challenge17 - 21 - 40 -

Disclaimer: The cruttkies, metfusion and schymane submissions come from the organizer's labs and can't be counted as real participants, although we tried to approach the challenges in an unbiased way.


Participant information and abstracts

ParticipantID: Dunn and Birmingham
Category: Category1 and category 2
Authors:Members of Dunn and Viant groups at University of Birmingham, UK
Affiliations:University of Birmingham, UK
Automatic pipeline:No
Spectral libraries:No

Abstract

The group automatically applied workflow 2 of PUTMEDID-LCMS to
calculate one or multiple molecular formula that matched the accurate
mass of the neutral metabolite. The group then automatically or
manually searched MMD, KEGG and ChemSpider in this order to define
potential metabolite structures, which were manually filtered in
relation to isotopes present or absent and 12C/13C ratios for
instruments where an accurate ratio can be calculated. The structures
were applied in MetFrag to calculate matches between in-silico
fragmentation and experimental data; these data were manually assessed
to remove biologically unreasonable metabolites.

We processed only the LC-MS challenges as follows:
1,2,3,4,5,6,10,13,14,15,17. The challenge data were converted to
molecular formula(s), searched against MMD, KEGG and Chemspider and
comparison of experimental and in-silico fragmentation data were
compared in MetFrag v0.9.

ParticipantID:        cruttkie
Category:             category2
Authors:              Christoph ruttkies
Affiliations:         (1) IPB Halle, Dept. of Stress and 
                      Developmental Biology, Halle, Germany
Automatic pipeline:   yes
Spectral libraries:   no

Abstract

The challenge data was converted to MetFrag query files, and processed with
MetFrag v0.9. Afterwards an Inchi-Key filter was applied which removed all
duplicates out of the candidate lists. To receive the final score for
challenge 1-6 the metabolite likeness score (Peironcely JE et al. 2011) was
added to the MetFrag score. The resulting candidate SDF was converted to TXT
as submission.
ParticipantID:        hshen
Category:	      category 1 and 2
Authors:              Huibin, Shen(1) and Nicola, Zamboni(2) and Markus, 
                      Heinonen(3) and Juho, Rousu(1)
Affiliations:         (1) Helsinki Institute for Information Technology; 
                      Department of Information and Computer Science, 
                      Aalto University, Finland (2) Institute of 
                      Molecular Systems Biology, ETH Zurich, Switzerland. 
                      (3) IBISC, Université d’Evry-Val d’Essonne, France

Automatic pipeline:   yes
Spectral libraries:   yes (MassBank)

Abstract

We processed only the LC-MS challenges. We predict the molecular
fingerprints of the challenge data using FingerID and use them to
search the Kegg compound database.
ParticipantID:        mgerlich
Category:             category2
Authors:              Michael Gerlich
Affiliations:         IPB Halle, Dept. of Stress and Developmental Biology,
Halle, Germany
Automatic pipeline:   yes
Spectral libraries:   yes

Abstract

All category2 challenges were converted into MetFusion specific query
files, containing the exact mass of the precursor ion as well as a
merged peaklist from the spectra with varying collision energies (where
applicable). The use of merged spectra is recommended by the MassBank
spectral library, which was used as reference library for spectra. The
instrument filter was set to use only ESI instruments, thus retrieving
no spectra from EI ionization or other ionization types.
The resulting candidate lists were treated with an InChIKey-based filter
which removes duplicate structures based on connectivity information.
The newly ranked candidates were stored in an SDF file which was
converted to a text file containing the corresponding SMILES and scoring
information as submission.
The challenge data was processed with the command line version of MetFusion.
ParticipantID:        oberacher
Category:	      category2
Authors:              Oberacher, Herbert
Affiliations:         Institute of Legal Medicine and Core Facility 
                      Metabolomics, Innsbruck Medical University
Automatic pipeline:   yes
Spectral libraries:   yes

Abstract

We processed only the LC-MS challenges. The challenge data was used as
input for automated library search in 4 libraries.
(a) MassBank (Date: 7.1.2013, Spectrum Search, Tolerance m/z: 0.3
units, Cutoff Threshold: 5, MS Type: All, Positive and Negative)
(b) Metlin (Date: 9.1.2013, MSMS Spectrum Search, Tolerance MSMS
0.01-0.1 Da, Tolerance precursor: 100 ppm, Positive and Negative)
(c) NIST (NIST 2012 - May 2012, NIST MS Search Program 2.0g - MSMS
Search - Identity Search, m/z tolerance: 1.6 for precursor ions and
0.8 for product ions, Positive and Negative)
(d) Wiley Registry of Tandem Mass Spectral Data, MSforID (Ed. 1,
MSforID Search, m/z tolerance: 0.01-0.1, Intensity threshold: 0.05,
Positive and Negative)
ParticipantID:        schymane
Category:             category2
Authors:              Schymanski, Emma(1) and Meringer, Markus (2)
Affiliations:         (1) Eawag: Swiss Federal Institute of Aquatic Science and 
                      Technology, Überlandstrasse 133, CH-8600 Dübendorf, 
                      Switzerland (2) DLR: German Aerospace Centre,
                      Münchnerstrasse 20, D-82234 Oberpfaffenhofen-Wessling,
                      Germany

Automatic pipeline:   no
Spectral libraries:   no (or very limited)

Abstract:

There is no specific MOLGEN for LC-MS/MS yet.
Thus, Category 2 challenges only have answers where the formula and
specific substructure information was clear.  There, we used structure
generation with either MOLGEN 3.5 or 5.0, adding the substructures
from spectral interpretation manually.  We then came up with consensus
scores using the steric energy of the candidates calculated with
MOLGEN-QSPR and scores from in silico fragmentation with MetFrag.  
The MOLGEN 5.0 file went through an additional conversion with OpenBabel
so that MetFrag could handle the aromaticity properly.  Challenge 16
MetFrag results were calculated with the command line version and has
slightly different scoring to the rest. For the same challenge, the
minimum of 3 QSPR calculations was also used.

Challenges 1-6, 12, 16: no submissions.
MOLGEN 3.5: Challenges 10, 11, 13, 15, 17.
MOLGEN 5.0: Challenge 14.

Details per Challenge and Participant. See legend at bottom for more details

The table is also available as CSV download

participant category challenge rank tc bc wc ec rrp p wbc wwc wec wrrp
birmingham category2 challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
cruttkies category2 challenge1 45 1423 21 1378 24 0.98 0.01 0.15 0.69 0.16 0.70
hshen category2 challenge1 5 6 4 1 1 0.20 0.07 0.87 0.06 0.00 0.13
metfusion category2 challenge1 1 1356 0 1355 1 1.00 0.00 0.00 1.00 0.00 1.00
oberacher category2 challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
birmingham category2 challenge2 - 7 - - - - - - - - -
cruttkies category2 challenge2 - 250 - - - - - - - - -
hshen category2 challenge2 1 6 0 5 1 1.00 0.28 0.00 0.72 0.00 1.00
metfusion category2 challenge2 30 543 29 513 1 0.95 0.00 0.09 0.91 0.00 0.91
birmingham category2 challenge3 - 1 - - - - - - - - -
cruttkies category2 challenge3 21 1312 20 1291 1 0.98 0.00 0.20 0.80 0.00 0.80
hshen category2 challenge3 - 8 - - - - - - - - -
metfusion category2 challenge3 - 1246 - - - - - - - - -
birmingham category2 challenge4 - 2 - - - - - - - - -
cruttkies category2 challenge4 - 1092 - - - - - - - - -
hshen category2 challenge4 - 14 - - - - - - - - -
metfusion category2 challenge4 195 3023 194 2828 1 0.94 0.00 0.08 0.92 0.00 0.92
birmingham category2 challenge5 4 4 0 0 4 0.50 0.25 0.00 0.00 0.75 0.25
cruttkies category2 challenge5 275 3978 270 3703 5 0.93 0.00 0.16 0.83 0.00 0.83
hshen category2 challenge5 5 17 3 12 2 0.78 0.09 0.29 0.54 0.09 0.62
metfusion category2 challenge5 386 3760 385 3374 1 0.90 0.00 0.12 0.88 0.00 0.88
birmingham category2 challenge6 - 3 - - - - - - - - -
cruttkies category2 challenge6 - 4566 - - - - - - - - -
hshen category2 challenge6 11 20 10 9 1 0.47 0.06 0.67 0.28 0.00 0.33
metfusion category2 challenge6 25 3254 24 3229 1 0.99 0.00 0.01 0.99 0.00 0.99
birmingham category2 challenge10 - 3 - - - - - - - - -
cruttkies category2 challenge10 302 536 282 234 20 0.46 0.00 0.98 0.01 0.01 0.01
hshen category2 challenge10 - 15 - - - - - - - - -
metfusion category2 challenge10 307 515 306 208 1 0.40 0.00 0.65 0.35 0.00 0.35
oberacher category2 challenge10 - 2 - - - - - - - - -
schymane category2 challenge10 63 171 62 108 1 0.64 0.01 0.51 0.48 0.00 0.49
cruttkies category2 challenge11 - 2156 - - - - - - - - -
hshen category2 challenge11 - 32 - - - - - - - - -
metfusion category2 challenge11 - 346 - - - - - - - - -
schymane category2 challenge11 3 8 2 5 1 0.71 0.13 0.26 0.61 0.00 0.74
cruttkies category2 challenge12 - 386 - - - - - - - - -
hshen category2 challenge12 - 47 - - - - - - - - -
metfusion category2 challenge12 - 699 - - - - - - - - -
birmingham category2 challenge13 - 1 - - - - - - - - -
cruttkies category2 challenge13 5 1307 1 1302 4 1.00 0.01 0.01 0.97 0.02 0.98
hshen category2 challenge13 - 38 - - - - - - - - -
metfusion category2 challenge13 64 1119 63 1055 1 0.94 0.00 0.08 0.92 0.00 0.92
oberacher category2 challenge13 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
schymane category2 challenge13 3 4 2 1 1 0.33 0.25 0.50 0.25 0.00 0.50
birmingham category2 challenge14 12 28 10 16 2 0.61 0.04 0.39 0.53 0.04 0.57
cruttkies category2 challenge14 39 123 35 84 4 0.70 0.01 0.74 0.22 0.03 0.23
hshen category2 challenge14 - 28 - - - - - - - - -
metfusion category2 challenge14 1 243 0 242 1 1.00 0.01 0.00 0.99 0.00 1.00
oberacher category2 challenge14 - 1 - - - - - - - - -
schymane category2 challenge14 22 41 21 19 1 0.47 0.03 0.64 0.34 0.00 0.36
birmingham category2 challenge15 - 1 - - - - - - - - -
cruttkies category2 challenge15 316 2053 312 1737 4 0.85 0.00 0.50 0.49 0.00 0.49
hshen category2 challenge15 - 10 - - - - - - - - -
metfusion category2 challenge15 1 1757 0 1756 1 1.00 0.00 0.00 1.00 0.00 1.00
oberacher category2 challenge15 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
schymane category2 challenge15 26 32 25 6 1 0.19 0.03 0.80 0.17 0.00 0.20
cruttkies category2 challenge16 2585 2585 286 0 2299 0.44 0.00 1.00 0.00 0.00 0.00
hshen category2 challenge16 - 13 - - - - - - - - -
metfusion category2 challenge16 1562 4351 1561 2789 1 0.64 0.00 0.38 0.62 0.00 0.62
birmingham category2 challenge17 - 1 - - - - - - - - -
cruttkies category2 challenge17 21 898 19 877 2 0.98 0.01 0.12 0.87 0.01 0.87
hshen category2 challenge17 - 18 - - - - - - - - -
metfusion category2 challenge17 40 625 39 585 1 0.94 0.00 0.08 0.92 0.00 0.92
schymane category2 challenge17 - 1590 - - - - - - - - -

Table legend:

rank
Absolute rank of correct solution
tc
Total number of candidates
bc
Number of candidates with a score better than correct solution
wc
Number of candidates with a score worse than correct solution
ec
Number of candidates with same score as the correct solution
rrp
Relative ranking position (1.0 is good, 0.0 is not)
p
Score of correct solution
wbc
Sum of scores better than correct solution
wwc
Sum of scores worse than correct solution
wec
Sum of scores equal to correct solution
wrrp
RRP weighted by the scores (1 is good)