News

Oct. 1, 2013
Details about the CASMI 2013 Special Issue and dates are now available!

Sept. 24, 2013
The rules and challenge data pages have been updated.

Sept. 2, 2013
The CASMI 2013 Challenges have been officially released!

August 29, 2013
The challenges for CASMI2013 will be released on Monday, September 2nd!

August 29, 2013
The CASMI 2012 poster will be presented in Langenau in November 2013


Resubmitted Results in Category 2
These results are based on submissions we received after we published the solutions. The main reason was that some of the published challenge data (2, 4 and 6) had mass deviations outside the promised specification, and that the peaklist for challenge 12 included an unrelated peak which easily lead to using a wrong precursor mass.

We also did not exclude other bugfixes in the participants' methods, and any changes are mentioned in the respective "Participant information and abstracts" below. The resubmitted results can be discussed by the participants in their respective papers in the CASMI proceedings. Future submissions can also be included, we will process them on a best-effort basis.

Summary of Rank by Challenge and Participant

For each challenge, the rank of the winner(s) is highlighted in bold. If the submission did not contain the correct candidate this is denoted as "-". If someone did not participate in a challenge, nothing is shown.

birmingham cruttkies cruttkies_r hshen metfusion metfusion_r oberacher schymane schymane_resubmit
challenge1 1 45 4 5 1 1 1
challenge2 - - 3 1 30 4
challenge3 - 21 9 - - 14
challenge4 - - 454 - 195 74
challenge5 4 275 1238 5 386 1426
challenge6 - - 281 11 25 25
challenge10 - 302 260 - 307 980 - 63
challenge11 - 23 - - 170 3
challenge12 - 36 - - 136
challenge13 - 5 5 - 64 1 1 3
challenge14 12 39 27 - 1 3 - 22
challenge15 - 316 173 - 1 1 1 26
challenge16 2585 1948 - 1562 1351
challenge17 - 21 15 - 40 88 - 58

Disclaimer: The cruttkies, metfusion and schymane submissions come from the organizer's labs and can't be counted as real participants, although we tried to approach the challenges in an unbiased way.


Participant information and abstracts

Resubmissions:

ParticipantID:        schymane
Category:             category2
Authors:              Schymanski, Emma(1) and Meringer, Markus (2)
Affiliations:         (1) Eawag: Swiss Federal Institute of Aquatic Science and 
                      Technology, Überlandstrasse 133, CH-8600 Dübendorf, 
                      Switzerland (2) DLR: German Aerospace Centre,
                      Münchnerstrasse 20, D-82234 Oberpfaffenhofen-Wessling,
                      Germany

Automatic pipeline:   no
Spectral libraries:   no (or very limited)

Abstract:

There is no specific MOLGEN for LC-MS/MS yet.  Thus, Category 2
challenges only have answers where the formula and specific
substructure information was clear.  For Challenge 17, we used the
wrong substructure information in the original submissions.  In this
resubmission, the correct substructures were used with MOLGEN 3.5.
Steric energy was calculated with MOLGEN-QSPR and the minumum of 3
runs was taken.  In silico fragmentation was performed with the
command line version of MetFrag (thanks Christoph) as the SDF was too
large for the web interface.  A consensus score was made combining the
two results.  SMILES from OpenBabel were used in the submission file.
[an error occurred while processing this directive]

Original:

ParticipantID: Dunn and Birmingham
Category: Category1 and category 2
Authors:Members of Dunn and Viant groups at University of Birmingham, UK
Affiliations:University of Birmingham, UK
Automatic pipeline:No
Spectral libraries:No

Abstract

The group automatically applied workflow 2 of PUTMEDID-LCMS to
calculate one or multiple molecular formula that matched the accurate
mass of the neutral metabolite. The group then automatically or
manually searched MMD, KEGG and ChemSpider in this order to define
potential metabolite structures, which were manually filtered in
relation to isotopes present or absent and 12C/13C ratios for
instruments where an accurate ratio can be calculated. The structures
were applied in MetFrag to calculate matches between in-silico
fragmentation and experimental data; these data were manually assessed
to remove biologically unreasonable metabolites.

We processed only the LC-MS challenges as follows:
1,2,3,4,5,6,10,13,14,15,17. The challenge data were converted to
molecular formula(s), searched against MMD, KEGG and Chemspider and
comparison of experimental and in-silico fragmentation data were
compared in MetFrag v0.9.

ParticipantID:        hshen
Category:	      category 1 and 2
Authors:              Huibin, Shen(1) and Nicola, Zamboni(2) and Markus, 
                      Heinonen(3) and Juho, Rousu(1)
Affiliations:         (1) Helsinki Institute for Information Technology; 
                      Department of Information and Computer Science, 
                      Aalto University, Finland (2) Institute of 
                      Molecular Systems Biology, ETH Zurich, Switzerland. 
                      (3) IBISC, Université d’Evry-Val d’Essonne, France

Automatic pipeline:   yes
Spectral libraries:   yes (MassBank)

Abstract

We processed only the LC-MS challenges. We predict the molecular
fingerprints of the challenge data using FingerID and use them to
search the Kegg compound database.
ParticipantID:        mgerlich
Category:             category2
Authors:              Michael Gerlich
Affiliations:         IPB Halle, Dept. of Stress and Developmental Biology,
Halle, Germany
Automatic pipeline:   yes
Spectral libraries:   yes

Abstract

All category2 challenges were converted into MetFusion specific query
files, containing the exact mass of the precursor ion as well as a
merged peaklist from the spectra with varying collision energies (where
applicable). The use of merged spectra is recommended by the MassBank
spectral library, which was used as reference library for spectra. The
instrument filter was set to use only ESI instruments, thus retrieving
no spectra from EI ionization or other ionization types.
The resulting candidate lists were treated with an InChIKey-based filter
which removes duplicate structures based on connectivity information.
The newly ranked candidates were stored in an SDF file which was
converted to a text file containing the corresponding SMILES and scoring
information as submission.
The challenge data was processed with the command line version of MetFusion.
ParticipantID:        oberacher
Category:	      category2
Authors:              Oberacher, Herbert
Affiliations:         Institute of Legal Medicine and Core Facility 
                      Metabolomics, Innsbruck Medical University
Automatic pipeline:   yes
Spectral libraries:   yes

Abstract

We processed only the LC-MS challenges. The challenge data was used as
input for automated library search in 4 libraries.
(a) MassBank (Date: 7.1.2013, Spectrum Search, Tolerance m/z: 0.3
units, Cutoff Threshold: 5, MS Type: All, Positive and Negative)
(b) Metlin (Date: 9.1.2013, MSMS Spectrum Search, Tolerance MSMS
0.01-0.1 Da, Tolerance precursor: 100 ppm, Positive and Negative)
(c) NIST (NIST 2012 - May 2012, NIST MS Search Program 2.0g - MSMS
Search - Identity Search, m/z tolerance: 1.6 for precursor ions and
0.8 for product ions, Positive and Negative)
(d) Wiley Registry of Tandem Mass Spectral Data, MSforID (Ed. 1,
MSforID Search, m/z tolerance: 0.01-0.1, Intensity threshold: 0.05,
Positive and Negative)

Details per Challenge and Participant. See legend at bottom for more details

The table is also available as CSV download

participant category challenge rank tc bc wc ec rrp p wbc wwc wec wrrp
birmingham category2 challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
cruttkies category2 challenge1 45 1423 21 1378 24 0.98 0.01 0.15 0.69 0.16 0.70
cruttkies_r category2 challenge1 4 994 3 990 1 1.00 0.01 0.04 0.95 0.00 0.96
hshen category2 challenge1 5 6 4 1 1 0.20 0.07 0.87 0.06 0.00 0.13
metfusion category2 challenge1 1 1356 0 1355 1 1.00 0.00 0.00 1.00 0.00 1.00
metfusion_r category2 challenge1 1 2229 0 2228 1 1.00 0.00 0.00 1.00 0.00 1.00
oberacher category2 challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
birmingham category2 challenge2 - 7 - - - - - - - - -
cruttkies category2 challenge2 - 250 - - - - - - - - -
cruttkies_r category2 challenge2 3 248 2 245 1 0.99 0.02 0.07 0.91 0.00 0.93
hshen category2 challenge2 1 6 0 5 1 1.00 0.28 0.00 0.72 0.00 1.00
metfusion category2 challenge2 30 543 29 513 1 0.95 0.00 0.09 0.91 0.00 0.91
metfusion_r category2 challenge2 4 625 3 621 1 1.00 0.00 0.01 0.99 0.00 0.99
birmingham category2 challenge3 - 1 - - - - - - - - -
cruttkies category2 challenge3 21 1312 20 1291 1 0.98 0.00 0.20 0.80 0.00 0.80
cruttkies_r category2 challenge3 9 1094 8 1085 1 0.99 0.01 0.13 0.87 0.00 0.87
hshen category2 challenge3 - 8 - - - - - - - - -
metfusion category2 challenge3 - 1246 - - - - - - - - -
metfusion_r category2 challenge3 14 2944 13 2930 1 1.00 0.00 0.01 0.99 0.00 0.99
birmingham category2 challenge4 - 2 - - - - - - - - -
cruttkies category2 challenge4 - 1092 - - - - - - - - -
cruttkies_r category2 challenge4 454 2234 453 1780 1 0.80 0.00 0.50 0.50 0.00 0.50
hshen category2 challenge4 - 14 - - - - - - - - -
metfusion category2 challenge4 195 3023 194 2828 1 0.94 0.00 0.08 0.92 0.00 0.92
metfusion_r category2 challenge4 74 4219 73 4145 1 0.98 0.00 0.02 0.98 0.00 0.98
birmingham category2 challenge5 4 4 0 0 4 0.50 0.25 0.00 0.00 0.75 0.25
cruttkies category2 challenge5 275 3978 270 3703 5 0.93 0.00 0.16 0.83 0.00 0.83
cruttkies_r category2 challenge5 1238 2891 1229 1653 9 0.57 0.00 0.60 0.39 0.00 0.39
hshen category2 challenge5 5 17 3 12 2 0.78 0.09 0.29 0.54 0.09 0.62
metfusion category2 challenge5 386 3760 385 3374 1 0.90 0.00 0.12 0.88 0.00 0.88
metfusion_r category2 challenge5 1426 4279 1425 2853 1 0.67 0.00 0.38 0.62 0.00 0.62
birmingham category2 challenge6 - 3 - - - - - - - - -
cruttkies category2 challenge6 - 4566 - - - - - - - - -
cruttkies_r category2 challenge6 281 1860 279 1579 2 0.85 0.00 0.40 0.59 0.00 0.60
hshen category2 challenge6 11 20 10 9 1 0.47 0.06 0.67 0.28 0.00 0.33
metfusion category2 challenge6 25 3254 24 3229 1 0.99 0.00 0.01 0.99 0.00 0.99
metfusion_r category2 challenge6 25 6175 24 6150 1 1.00 0.00 0.01 0.99 0.00 0.99
birmingham category2 challenge10 - 3 - - - - - - - - -
cruttkies category2 challenge10 302 536 282 234 20 0.46 0.00 0.98 0.01 0.01 0.01
cruttkies_r category2 challenge10 260 447 240 187 20 0.44 0.00 1.00 0.00 0.00 0.00
hshen category2 challenge10 - 15 - - - - - - - - -
metfusion category2 challenge10 307 515 306 208 1 0.40 0.00 0.65 0.35 0.00 0.35
metfusion_r category2 challenge10 980 1084 979 104 1 0.10 0.00 0.92 0.08 0.00 0.08
oberacher category2 challenge10 - 2 - - - - - - - - -
schymane category2 challenge10 63 171 62 108 1 0.64 0.01 0.51 0.48 0.00 0.49
cruttkies category2 challenge11 - 2156 - - - - - - - - -
cruttkies_r category2 challenge11 23 465 0 442 23 0.98 0.01 0.00 0.86 0.13 0.87
hshen category2 challenge11 - 32 - - - - - - - - -
metfusion category2 challenge11 - 346 - - - - - - - - -
metfusion_r category2 challenge11 170 1444 168 1274 2 0.88 0.00 0.15 0.84 0.00 0.84
schymane category2 challenge11 3 8 2 5 1 0.71 0.13 0.26 0.61 0.00 0.74
cruttkies category2 challenge12 - 386 - - - - - - - - -
cruttkies_r category2 challenge12 36 1531 32 1495 4 0.98 0.01 0.29 0.67 0.03 0.68
hshen category2 challenge12 - 47 - - - - - - - - -
metfusion category2 challenge12 - 699 - - - - - - - - -
metfusion_r category2 challenge12 136 3764 135 3628 1 0.96 0.00 0.05 0.95 0.00 0.95
birmingham category2 challenge13 - 1 - - - - - - - - -
cruttkies category2 challenge13 5 1307 1 1302 4 1.00 0.01 0.01 0.97 0.02 0.98
cruttkies_r category2 challenge13 5 1031 1 1026 4 1.00 0.01 0.01 0.95 0.03 0.96
hshen category2 challenge13 - 38 - - - - - - - - -
metfusion category2 challenge13 64 1119 63 1055 1 0.94 0.00 0.08 0.92 0.00 0.92
metfusion_r category2 challenge13 1 3344 0 3343 1 1.00 0.00 0.00 1.00 0.00 1.00
oberacher category2 challenge13 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
schymane category2 challenge13 3 4 2 1 1 0.33 0.25 0.50 0.25 0.00 0.50
birmingham category2 challenge14 12 28 10 16 2 0.61 0.04 0.39 0.53 0.04 0.57
cruttkies category2 challenge14 39 123 35 84 4 0.70 0.01 0.74 0.22 0.03 0.23
cruttkies_r category2 challenge14 27 125 21 98 6 0.81 0.02 0.57 0.32 0.09 0.34
hshen category2 challenge14 - 28 - - - - - - - - -
metfusion category2 challenge14 1 243 0 242 1 1.00 0.01 0.00 0.99 0.00 1.00
metfusion_r category2 challenge14 3 507 2 504 1 1.00 0.01 0.01 0.98 0.00 0.99
oberacher category2 challenge14 - 1 - - - - - - - - -
schymane category2 challenge14 22 41 21 19 1 0.47 0.03 0.64 0.34 0.00 0.36
birmingham category2 challenge15 - 1 - - - - - - - - -
cruttkies category2 challenge15 316 2053 312 1737 4 0.85 0.00 0.50 0.49 0.00 0.49
cruttkies_r category2 challenge15 173 1825 169 1652 4 0.91 0.00 0.35 0.65 0.01 0.65
hshen category2 challenge15 - 10 - - - - - - - - -
metfusion category2 challenge15 1 1757 0 1756 1 1.00 0.00 0.00 1.00 0.00 1.00
metfusion_r category2 challenge15 1 3392 0 3391 1 1.00 0.00 0.00 1.00 0.00 1.00
oberacher category2 challenge15 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
schymane category2 challenge15 26 32 25 6 1 0.19 0.03 0.80 0.17 0.00 0.20
cruttkies category2 challenge16 2585 2585 286 0 2299 0.44 0.00 1.00 0.00 0.00 0.00
cruttkies_r category2 challenge16 1948 1948 182 0 1766 0.45 0.00 1.00 0.00 0.00 0.00
hshen category2 challenge16 - 13 - - - - - - - - -
metfusion category2 challenge16 1562 4351 1561 2789 1 0.64 0.00 0.38 0.62 0.00 0.62
metfusion_r category2 challenge16 1351 4427 1350 3076 1 0.69 0.00 0.32 0.68 0.00 0.68
birmingham category2 challenge17 - 1 - - - - - - - - -
cruttkies category2 challenge17 21 898 19 877 2 0.98 0.01 0.12 0.87 0.01 0.87
cruttkies_r category2 challenge17 15 475 14 460 1 0.97 0.01 0.13 0.86 0.00 0.87
hshen category2 challenge17 - 18 - - - - - - - - -
metfusion category2 challenge17 40 625 39 585 1 0.94 0.00 0.08 0.92 0.00 0.92
metfusion_r category2 challenge17 88 1848 87 1760 1 0.95 0.00 0.06 0.94 0.00 0.94
schymane_resubmit category2 challenge17 58 1295 57 1237 1 0.96 0.00 0.07 0.92 0.00 0.93
schymane category2 challenge17 - 1590 - - - - - - - - -

Table legend:

rank
Absolute rank of correct solution
tc
Total number of candidates
bc
Number of candidates with a score better than correct solution
wc
Number of candidates with a score worse than correct solution
ec
Number of candidates with same score as the correct solution
rrp
Relative ranking position (1.0 is good, 0.0 is not)
p
Score of correct solution
wbc
Sum of scores better than correct solution
wwc
Sum of scores worse than correct solution
wec
Sum of scores equal to correct solution
wrrp
RRP weighted by the scores (1 is good)