News

Oct. 1, 2013
Details about the CASMI 2013 Special Issue and dates are now available!

Sept. 24, 2013
The rules and challenge data pages have been updated.

Sept. 2, 2013
The CASMI 2013 Challenges have been officially released!

August 29, 2013
The challenges for CASMI2013 will be released on Monday, September 2nd!

August 29, 2013
The CASMI 2012 poster will be presented in Langenau in November 2013


Resubmitted results in Category 1
These results are based on submissions we received after we published the solutions. The main reason was that some of the published challenge data (2, 4 and 6) had mass deviations outside the promised specification, and that the peaklist for challenge 12 included an unrelated peak which easily lead to using a wrong precursor mass.

We also did not exclude other bugfixes in the participants' methods, and any changes are mentioned in the respective "Participant information and abstracts" below. The resubmitted results can be discussed by the participants in their respective papers in the CASMI proceedings. Future submissions can also be included, we will process them on a best-effort basis.

Summary of Rank by Challenge and Participant

For each challenge, the rank of the winner(s) is highlighted in bold. If the submission did not contain the correct candidate this is denoted as "-". If someone did not participate in a challenge, nothing is shown.

birmingham hshen kduehrkop rdisop rdisop_resubmitted schymane_resubmit schymane kduehrkop_resubmit
challenge1 1 4 3 18 18 11 23
challenge2 - 1 - - 1 1 - 2
challenge3 1 - 8 - 19 14 1
challenge4 1 3 - 5 2 1 - 2
challenge5 1 4 2 1 3 1 8 29
challenge6 2 4 - 9 1 1 - 2
challenge10 1 1 1 1 1 1
challenge11 - - - - 4
challenge12 - - - 8 4 18
challenge13 - - 1 1 1 1
challenge14 1 1 1 1 1 1
challenge15 1 5 1 2 2 1
challenge16 - - - 4
challenge17 1 - 1 1 1 1

Disclaimer: The rdisop and schymane submissions come from the organizer's labs and can't be counted as real participants, although we tried to approach the challenges in an unbiased way.


Participant information and abstracts

Resubmissions:

ParticipantID:        schymane
Category:             category1
Authors:              Schymanski, Emma(1) and Meringer, Markus (2)
Affiliations:         (1) Eawag: Swiss Federal Institute of Aquatic Science and 
                      Technology, Überlandstrasse 133, CH-8600 Dübendorf, 
                      Switzerland (2) DLR: German Aerospace Centre,
                      Münchnerstrasse 20, D-82234 Oberpfaffenhofen-Wessling,
                      Germany

Automatic pipeline:   yes
Spectral libraries:   no


Abstract:

We processed Category 1 challenges 1-6 again with MOLGEN-MS/MS, using
CHNOPS, as the errors used before were too optimistic.  We used MS
accuracy 5 ppm and MS/MS accuracy 60 ppm, but no other parameters were
adjusted.  For Challenges 2, 4 and 6 we used the recalibrated data,
for 1, 3 and 5 the original data (as Challenge 5 was outside the 5 ppm
range in the recalibrated data).

We submitted the formulas generated with these settings, using the
combined match value as the score.

Where multiple MS/MS files were available we combined them to get the
maximum number of peaks, using the peak of highest intensity where the
same peak occurred more than once. As the score was not weighted by
intensity, this should make no difference to the outcome.  For the
revised entries of Chal 1-6, we also removed isotope peaks from the
MSMS, where these were recognisable as such. If in doubt, these were
left in.

ParticipantID:        kduehrkop
Category:	      category1
Authors:              Kai Dührkop, Kerstin Scheubert, Sebastian Böcker
Affiliations:         Chair for Bioinformatics, Friedrich-Schiller-University, 
                      Jena, Germany
Automatic pipeline:   yes
Spectral libraries:   no

Abstract

We processed only the LC-MS challenges. The spectral data was analyzed
by the sirius command line tool.  We choosed the parameters of our
tool according to the challenge description.  We combined the scores
of the isotopic patterns and the fragmentation trees computed from
ms/ms spectra.  The pubchem molecular formula search was used to
assign adducts to the ions.  If no formula was found in pubchem, we
assumed a protonation.  For intrinsic charged formulas, we added a
protonated variant with a slightly reduced score.

In the resubmission we processed the calibrated data for Challenges 2
and 4-6, and used the correct precursor ion for Challenge 12.
Original:

ParticipantID: Dunn and Birmingham
Category: Category1 and category 2
Authors:Members of Dunn and Viant groups at University of Birmingham, UK
Affiliations:University of Birmingham, UK
Automatic pipeline:No
Spectral libraries:No

Abstract

The group automatically applied workflow 2 of PUTMEDID-LCMS to
calculate one or multiple molecular formula that matched the accurate
mass of the neutral metabolite. The group then automatically or
manually searched MMD, KEGG and ChemSpider in this order to define
potential metabolite structures, which were manually filtered in
relation to isotopes present or absent and 12C/13C ratios for
instruments where an accurate ratio can be calculated. The structures
were applied in MetFrag to calculate matches between in-silico
fragmentation and experimental data; these data were manually assessed
to remove biologically unreasonable metabolites.

We processed only the LC-MS challenges as follows:
1,2,3,4,5,6,10,13,14,15,17. The challenge data were converted to
molecular formula(s), searched against MMD, KEGG and Chemspider and
comparison of experimental and in-silico fragmentation data were
compared in MetFrag v0.9.

ParticipantID:        hshen
Category:	      category 1 and 2
Authors:              Huibin, Shen(1) and Nicola, Zamboni(2) and Markus, 
                      Heinonen(3) and Juho, Rousu(1)
Affiliations:         (1) Helsinki Institute for Information Technology; 
                      Department of Information and Computer Science, 
                      Aalto University, Finland (2) Institute of 
                      Molecular Systems Biology, ETH Zurich, Switzerland. 
                      (3) IBISC, Université d’Evry-Val d’Essonne, France
Automatic pipeline:   yes
Spectral libraries:   yes (MassBank)

Abstract

We processed only the LC-MS challenges. We predict the molecular
fingerprints of the challenge data using FingerID and use them to
search the Kegg compound database.
ParticipantID:        rdisop
Category:             category1
Authors:              Neumann, Steffen
Affiliations:         Leibniz Institute for Plant Biochemistry, Dept. of Stress-
                      and Developmental Biology, Halle, Germany
Automatic pipeline:   yes
Spectral libraries:   no

Abstract:

I create a small script extracting the isotope pattern 
from the MS peaklist files, and processed them with the 
Bioconductor package rdisop (which in turn uses the disop library 
developed by the Boecker group). No efforts were made to 
cleverly detect the [M+H]+ or adducts. 



Details per Challenge and Participant. See legend at bottom for more details

The table is also available as CSV download

participant category challenge rank tc bc wc ec rrp p wbc wwc wec wrrp
birmingham category1 challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge1 4 5 3 1 1 0.25 0.08 0.85 0.06 0.00 0.15
kduehrkop category1 challenge1 3 136 2 133 1 0.99 0.00 1.00 0.00 0.00 0.00
rdisop category1 challenge1 18 1466 17 1448 1 0.99 0.00 1.00 0.00 0.00 0.00
rdisop_resubmitted category1 challenge1 18 4334 17 4316 1 1.00 0.00 1.00 0.00 0.00 0.00
schymane_resubmit category1 challenge1 11 54 10 43 1 0.81 0.02 0.21 0.77 0.00 0.79
schymane category1 challenge1 23 54 22 31 1 0.58 0.02 0.61 0.36 0.00 0.39
birmingham category1 challenge2 - 3 - - - - - - - - -
hshen category1 challenge2 1 4 0 3 1 1.00 0.40 0.00 0.60 0.00 1.00
kduehrkop category1 challenge2 - 926 - - - - - - - - -
kduehrkop_resubmit category1 challenge2 2 1025 1 1023 1 1.00 0.04 0.94 0.01 0.00 0.06
rdisop category1 challenge2 - 4430 - - - - - - - - -
rdisop_resubmitted category1 challenge2 1 13375 0 13374 1 1.00 0.46 0.00 0.54 0.00 1.00
schymane_resubmit category1 challenge2 1 249 0 248 1 1.00 0.00 0.00 1.00 0.00 1.00
schymane category1 challenge2 - 226 - - - - - - - - -
birmingham category1 challenge3 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge3 - 6 - - - - - - - - -
kduehrkop category1 challenge3 8 425 7 417 1 0.98 0.00 1.00 0.00 0.00 0.00
rdisop category1 challenge3 - 1180 - - - - - - - - -
rdisop_resubmitted category1 challenge3 19 3488 18 3469 1 0.99 0.00 1.00 0.00 0.00 0.00
schymane_resubmit category1 challenge3 14 90 13 76 1 0.85 0.01 0.18 0.81 0.00 0.82
schymane category1 challenge3 1 90 0 89 1 1.00 0.01 0.00 0.99 0.00 1.00
birmingham category1 challenge4 1 2 0 1 1 1.00 0.67 0.00 0.33 0.00 1.00
hshen category1 challenge4 3 12 2 9 1 0.82 0.12 0.28 0.60 0.00 0.72
kduehrkop category1 challenge4 - 39 - - - - - - - - -
kduehrkop_resubmit category1 challenge4 2 38 1 36 1 0.97 0.00 1.00 0.00 0.00 0.00
rdisop category1 challenge4 5 217 4 212 1 0.98 0.00 1.00 0.00 0.00 0.00
rdisop_resubmitted category1 challenge4 2 651 1 649 1 1.00 0.01 0.99 0.00 0.00 0.01
schymane_resubmit category1 challenge4 1 15 0 14 1 1.00 0.08 0.00 0.92 0.00 1.00
schymane category1 challenge4 - 13 - - - - - - - - -
birmingham category1 challenge5 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge5 4 11 3 7 1 0.70 0.14 0.47 0.39 0.00 0.53
kduehrkop category1 challenge5 2 30 1 28 1 0.97 0.00 1.00 0.00 0.00 0.00
kduehrkop_resubmit category1 challenge5 29 29 27 0 2 0.02 0.00 1.00 0.00 0.00 0.00
rdisop category1 challenge5 1 226 0 225 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop_resubmitted category1 challenge5 3 663 2 660 1 1.00 0.00 1.00 0.00 0.00 0.00
schymane_resubmit category1 challenge5 1 10 0 9 1 1.00 0.11 0.00 0.89 0.00 1.00
schymane category1 challenge5 8 10 7 2 1 0.22 0.08 0.82 0.10 0.00 0.18
birmingham category1 challenge6 2 2 1 0 1 0.00 0.33 0.67 0.00 0.00 0.33
hshen category1 challenge6 4 15 3 11 1 0.79 0.10 0.32 0.58 0.00 0.68
kduehrkop category1 challenge6 - 78 - - - - - - - - -
kduehrkop_resubmit category1 challenge6 2 83 1 81 1 0.99 0.02 0.98 0.00 0.00 0.02
rdisop category1 challenge6 9 498 8 489 1 0.98 0.00 1.00 0.00 0.00 0.00
rdisop_resubmitted category1 challenge6 1 1464 0 1463 1 1.00 0.69 0.00 0.31 0.00 1.00
schymane_resubmit category1 challenge6 1 31 0 30 1 1.00 0.04 0.00 0.96 0.00 1.00
schymane category1 challenge6 - 25 - - - - - - - - -
birmingham category1 challenge10 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge10 1 11 0 10 1 1.00 0.18 0.00 0.82 0.00 1.00
kduehrkop category1 challenge10 1 7 0 6 1 1.00 0.93 0.00 0.07 0.00 1.00
rdisop category1 challenge10 1 44 0 43 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop_resubmitted category1 challenge10 1 125 0 124 1 1.00 1.00 0.00 0.00 0.00 1.00
schymane category1 challenge10 1 3 0 2 1 1.00 0.48 0.00 0.52 0.00 1.00
hshen category1 challenge11 - 15 - - - - - - - - -
kduehrkop category1 challenge11 - 15 - - - - - - - - -
rdisop category1 challenge11 - 29 - - - - - - - - -
rdisop_resubmitted category1 challenge11 - 177 - - - - - - - - -
schymane category1 challenge11 4 6 0 2 4 0.70 0.20 0.00 0.20 0.60 0.40
hshen category1 challenge12 - 21 - - - - - - - - -
kduehrkop category1 challenge12 - 68 - - - - - - - - -
kduehrkop_resubmit category1 challenge12 18 49 17 31 1 0.65 0.00 1.00 0.00 0.00 0.00
rdisop category1 challenge12 - 8 - - - - - - - - -
rdisop_resubmitted category1 challenge12 8 830 7 822 1 0.99 0.00 0.99 0.00 0.00 0.01
schymane category1 challenge12 4 18 0 14 4 0.91 0.12 0.00 0.51 0.37 0.63
birmingham category1 challenge13 - 1 - - - - - - - - -
hshen category1 challenge13 - 15 - - - - - - - - -
kduehrkop category1 challenge13 1 20 0 19 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop category1 challenge13 1 141 0 140 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop_resubmitted category1 challenge13 1 403 0 402 1 1.00 1.00 0.00 0.00 0.00 1.00
schymane category1 challenge13 1 10 0 9 1 1.00 0.14 0.00 0.86 0.00 1.00
birmingham category1 challenge14 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge14 1 12 0 11 1 1.00 0.17 0.00 0.83 0.00 1.00
kduehrkop category1 challenge14 1 4 0 3 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop category1 challenge14 1 12 0 11 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop_resubmitted category1 challenge14 1 35 0 34 1 1.00 1.00 0.00 0.00 0.00 1.00
schymane category1 challenge14 1 2 0 1 1 1.00 0.83 0.00 0.17 0.00 1.00
birmingham category1 challenge15 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge15 5 6 4 1 1 0.20 0.15 0.71 0.14 0.00 0.29
kduehrkop category1 challenge15 1 4 0 3 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop category1 challenge15 2 27 1 25 1 0.96 0.05 0.92 0.03 0.00 0.08
rdisop_resubmitted category1 challenge15 2 77 1 75 1 0.99 0.05 0.92 0.03 0.00 0.08
schymane category1 challenge15 1 2 0 1 1 1.00 0.50 0.00 0.50 0.00 1.00
hshen category1 challenge16 - 12 - - - - - - - - -
kduehrkop category1 challenge16 - 46 - - - - - - - - -
rdisop_resubmitted category1 challenge16 - 1101 - - - - - - - - -
schymane category1 challenge16 4 20 1 16 3 0.89 0.08 0.09 0.67 0.16 0.75
birmingham category1 challenge17 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
hshen category1 challenge17 - 11 - - - - - - - - -
kduehrkop category1 challenge17 1 5 0 4 1 1.00 0.89 0.00 0.11 0.00 1.00
rdisop category1 challenge17 1 30 0 29 1 1.00 1.00 0.00 0.00 0.00 1.00
rdisop_resubmitted category1 challenge17 1 87 0 86 1 1.00 1.00 0.00 0.00 0.00 1.00
schymane category1 challenge17 1 3 0 2 1 1.00 0.54 0.00 0.46 0.00 1.00

Table legend:

rank
Absolute rank of correct solution
tc
Total number of candidates
bc
Number of candidates with a score better than correct solution
wc
Number of candidates with a score worse than correct solution
ec
Number of candidates with same score as the correct solution
rrp
Relative ranking position (1.0 is good, 0.0 is not)
p
Score of correct solution
wbc
Sum of scores better than correct solution
wwc
Sum of scores worse than correct solution
wec
Sum of scores equal to correct solution
wrrp
RRP weighted by the scores (1 is good)