News

March 18th, 2015
The results and winners of CASMI-2014 are now up.

March 3rd, 2015
The solutions for CASMI-2014 are now available on the website.

January 15th, 2015
The CASMI 2014 Challenges submissions deadline has been extended to February 15th, 2015!

September 15th, 2014
The CASMI 2014 Challenges are now officially available !

September 2nd, 2014
All articles of the CASMI 2013 proceedings are now online.

April 30, 2014
First announcement at the Analytical Tools for Cutting-edge Metabolomics meeting.


Results in Category 1

Summary of Rank by Challenge and Participant

For each challenge, the rank of the winner(s) is highlighted in bold. If the submission did not contain the correct candidate this is denoted as "-". If someone did not participate in a challenge, nothing is shown.

avaniya bertrandsamuel felicityallen fitch lridder tpluskal
challenge1 1 - 1 1 1 1
challenge2 1 1 1 1 1 1
challenge3 1 1 1 1 1 1
challenge4 1 1 1 - 1 1
challenge5 1 1 1 1 1 1
challenge6 1 1 1 1 1 1
challenge7 1 2 1 1 1 1
challenge8 1 1 - - 10
challenge9 1 2 1 1 1 10
challenge10 1 - 1 1 1 10
challenge11 1 3 1 - 1 1
challenge12 - 1 1 - 1 1
challenge13 1 4 1 1 1 2
challenge14 1 1 1 1 1 1
challenge15 1 1 1 1 1 1
challenge16 1 4 1 1 1 1
challenge17 1 1 1 1 1 1
challenge18 1 1 1 1 1 1
challenge19 1 2 1 1 2 2
challenge20 1 1 1 1 1 1
challenge21 1 4 1 1 1 1
challenge22 1 1 1 1 1 3
challenge23 - - - - - 3
challenge24 1 1 1 1 1 9
challenge25 1 - 1 - - -
challenge26 - 15 - - - 12
challenge27 1 3 1 1 2 2
challenge28 1 1 1 - 1 2
challenge29 - 2 - - - -
challenge30 1 1 1 1 1 1
challenge31 - - - - - -
challenge32 1 1 1 1 1 1
challenge33 1 28 1 1 1 5
challenge34 1 1 1 1 1 1
challenge35 - 1 1 1 1 5
challenge36 - - - 1 1 1
challenge37 1 - - 1 1 -
challenge38 1 1 1 1 1 1
challenge39 1 1 - 1 1 1
challenge40 1 1 1 1 1 1
challenge41 - - 1 - 1 2
challenge42 1 1 1 1 1 1
challenge43 - 2 7 1 1 5
challenge44 1 1 1 1 1 1
challenge45 - 1 1 1 1 1
challenge46 - 1 1 1 1 1
challenge47 1 1 1 1 2
challenge48 - - - 1 2
This summary is also available as CSV download.


Participant information and abstracts

ParticipantID:        BertrandSamuel 
Category:             category1 
Authors: 	      Bertrand, Samuel(1) and Guitton, Yann (1,2,3,4,5) 
Affiliations:         (1) Groupe Mer, Molécules, Santé-EA 2160, UFR des Sciences 
                      Pharmaceutiques et Biologiques, Université de Nantes, France 
                      (2) Université de Rennes 1, IRISA UMR 6074, Rennes, France 
                      (3) CNRS, IRISA UMR 6074, Rennes, France 
                      (4) Centre Rennes-Bretagne-Atlantique, Projet Dyliss, INRIA, 
                      Rennes Cedex, France 
                      (5) Université de Nantes, LINA UMR 6241, Nantes, France 
Automatic methods:    yes 

Abstract 
The challenge data were automatically treated using R, Rdisop, RMassBank as follow: 
1- retrieving MS Data 
2- searching for NeutralLosses, adducts according to MS1 data. When no information 
were found ions were considered as [M]+, [M+H]+, [M+Na]+, [M+K]+, [M+H+solvent]+, 
[M+NH4]+ (if ammonia was added in the eluant), [M-H]-, [M+Fa]- (if Fa was added) and 
[M+AceticAcid]- (if AA was added); some manual interventions were done to 
simplify some complex detected ion relationship 
3- for each adduct possibility of each challenge molecular formula were deduced by 
Rdisop using CxHyO27N25P9F34Fe (modified according to neutral losses and adduct 
assigned) as maximum allowed atoms in the very strict limit of the MS accuracy. Each
possible formula should pass the seven golden rules to be saved.
4- each possible formula were scored according to MS accuracy, Spectral Accuracy, MF 
redundancy (multiply detected adducts from one MS spectra), type of adduct, the 
possibility to find Molecular formula related to MS2 ions from the MF itself 
5- only formula found in chemSpider/chebi/KnApSaCK were reported for challenges 1-42 
(no filtering according to existence were done for challenges 43-48)

ParticipantID:        lridder
Category:             category2
Authors:              Ridder, Lars
Affiliations:         Wageningen University, Laboratory of Biochemistry,
                      Wageningen, The Netherlands
Automatic methods:    yes

Abstract
The challenge peak lists were converted to MAGMa input files, and processed
with MAGMa (http://www.emetabolomics.org/magma), as described in the
metadata file for category 2.
Submissions for category 1 consists of the lists of unique molecular formula's
obtained in category 2. The provided scores correspond to the highest scoring
candidate (in category 2) with the given molecular formula.
This submission is supported by the observation in Ridder et al. (2012) and Ridder
et al. (2013) that, even if the top scoring candidate structure in MAGMa is
not correct, the molecular formula often is.

References:
Ridder, L.; van der Hooft, J. J. J.; Verhoeven, S.; de Vos, R. C. H.; van Schaik, R.;
Vervoort (2012) J. Rapid Commun. Mass Spectrom. 26, 2461-2471.
Ridder, L.; van der Hooft, J. J. J.; Verhoeven, S.; de Vos, R. C. H.; Bino, R. J.;
Vervoort, J.(2013) Anal. Chem. 85, 6033-6040.

ParticipantID:        tpluskal
Category:             category1
Authors:              Pluskal, Tomáš(1)
Affiliations:         (1) Okinawa Institute of Science and Technology
                      (OIST), Onna, Okinawa, Japan
Automatic methods:    no

Abstract
I imported the challenge spectra into MZmine version 2.13.1 through a small 
custom-made import module. Then I generated chemical formulas using the MZmine’s 
chemical formula prediction module. I did not alter the resulting lists of chemical 
formulas generated by MZmine. However, I had to manually adjust the parameters for 
individual challenges, therefore I do not claim this method is fully automatic.

MZmine requires 7 parameters to be set for prediction of chemical formulas:

1) Neutral mass
To calculate the neutral mass from the monoisotopic m/z, I set the ion type to [M+H]+
for positive mode spectra or [M-H]- for negative mode spectra. The only exceptions 
were challenges #8, #23 and #26, which I believe represented [M+Na]+ ions, based on 
manual observation of the spectra, or empirical knowledge. Charge was assumed to be 1
in all cases.

2) m/z tolerance was set to twice the expected m/z tolerance for the instrument, as 
stated in the challenge description (if the challenge said "The expected mass 
accuracy of the instrument is <5ppm", I set the m/z tolerance to 10 ppm).

3) Element counts
For challenges #1 to #31, I allowed C,H,N,O,P,S elements (0-1000 each). For 
challenges #32 to #48 (synthetic and unknown compounds) I also added elements F,Cl,
Br,I (0-100 each).

4) Element count heuristics
All restrictions available in MZmine were activated (H/C ratio, NOPS/C ratios, 
multiple element counts). These are originally based on the Seven Golden Rules paper 
by Kind and Fiehn, 2007.

5) RDBE restrictions
Generated formulas were restricted to those having integer RDBE values in the range 
of 0-40.

6) Isotope pattern filter
Isotope pattern m/z tolerance was fixed to 0.02 m/z in all cases, because input data 
did not contain any fine isotopic peaks. Minimum absolute intensity for generated 
patterns was set to 0.1. Minimum isotope pattern similarity score was set to 90% 
except challenges #24 (50%, due to declared bias in isotope intensity), #27 (80%, due 
to no candidates returned at 90%) and #35 (80%, due to no isotopes observed for 
209.0453 m/z).

7) MS/MS filter
MS/MS m/z tolerance was set to 0.005 m/z in all cases. MZmine uses this tolerance to 
interpret neutral losses (e.g., 18.0106 = H2O). Minimum MS/MS score was set to 90%, 
except challenges #6, #8, and #18, which contained MS/MS peaks that I was not able to 
interpret (even manually). For these three challenges I disabled the MS/MS filter 
completely.
ParticipantID:        felicityallen
Category:             Category 1 and Category 2
Authors:              Felicity Allen, Russ Greiner, David Wishart
Affiliations:         Department of Computing Science University of Alberta, Canada
Automatic methods:    yes

Abstract
A list of candidate molecular formulae was obtained by querying PubChem
for all structures within the stated MS tolerances of the precursor 
mass (as obtained by combining the integer precursor mass stated in the
experimental details file with the MS1 file). In cases where multiple
precursor masses were listed, candidates and molecular formula were included
for all listed masses. All unique molecular formulae were collated from these 
structures.

The MS1 spectra were then predicted for each candidate molecular formula using
the emass program by A. Rockwood and P. Haimi [1].  These predicted spectra
were compared to the provided MS1 spectra (restricted to within 10 Da of the 
monoisotopic mass of the molecular formula), and an MS1_SCORE was produced 
for each molecular formula based on the closeness of this match. The scoring
metric used was:  
MS1_SCORE = ( (WP + WR + DP)_3ppm + (WP + WR + DP)_5ppm + (WP + WR + DP)_10ppm )/10
where 
WP = intensity weighted precision (0-100)
WR = intensity weighted recall (0-100)
DP = dot product (0-1) x 100

[1] Rockwood A. and Haimi P., "Efficient calculation of accurate masses of isotopic 
peaks.", Journal of the American Society for Mass Spectrometry, 17:3 p415-9 2006.	

A list of candidate structures was obtained by querying all of the following
databases for all candidates within the required mass ranges (determined as above):
HMDB  http://www.hmdb.ca/
ChEBI http://www.ebi.ac.uk/chebi/
ChEMBL https://www.ebi.ac.uk/chembl/
Metlin http://metlin.scripps.edu/
FOODB http://foodb.ca/
T3DB http://www.t3db.ca/
DrugBank http://www.drugbank.ca/
ECMDB http://www.ecmdb.ca/
YMDB http://www.ymdb.ca/
PubChem https://pubchem.ncbi.nlm.nih.gov
PlantDB Privately held list of 200,000 plant and plant-derived compounds.

For all candidate structures with an MS1_SCORE > 28, CFM-ID [2,3] was used
to predict an MS2 spectrum for all candidates at both a 10V and 20V energy level. 
The MS2_SCORE was computed for each to reflect the closeness of this match:

MS2_SCORE = (DP + 2JA + WP + WR)_10V + (DP + 2JA + WP + WR)_20V
where WP, WR and DP are as above, and
JA = Jaccard Score (0-1) x 100

[2] Allen F., Pon A., Wilson M., Greiner R., Wishart D., "CFM-ID: A web server for 
annotation, spectrum prediction and metabolite identification from tandem mass 
spectra", Nucleic Acids Research, Web Server Edition 2014.

[3] Allen F., Greiner R., Wishart D., "Competitive Fragmentatation Modeling of 
ESI-MS/MS spectra for putative metabolite identification", Metabolomics, 11:1, 
p98-110, 2015.

For all candidates, a DB_SCORE was produced according to which of the above databases 
it was found in, to make use of the provided information about metabolic origin. 
Consider the database names below as indicators, i.e. add the amount if in that 
database):

DB_SCORE (Challenge 1-15) = HMDB (+200) + METLIN (+100) + ChEBI (+20) + FooDB (+20) +
DrugBank (+20) + ECMDB (+10) + YMDB (+10) + ChEMBL (+10).
DB_SCORE (Challenge 6-31) = ChEBI (+100) + ECMDB (+100) + YMDB (+100) + PlantDB 
(+100) + Metlin (+50) + FooDB (+50) + ChEMBL (+10).
DB_SCORE (Challenge 32-42) = HMDB (+50) + ChEMBL (+100) + T3DB (+100) + METLIN (+100)
+ DrugBank(+100)
DB_SCORE (Challenge 43-48) = 0

An additional manual offset of +/-50 was also made in some cases to reorder the top 
couple of candidates. This was done where descriptions of compounds in the database 
provided additional information for or against a match e.g. where a particular 
species or biofluid was provided in the experimental data.

For Category 2, the results were ranked according to the sum of the above three 
scores: TOTAL_SCORE = MS2_SCORE + DB_SCORE + MS1_SCORE

For Category 1, the molecular formula was computed for each structure
from Category 2 and kept in the same order. The list was then processed
to remove duplicate entries, keeping only the highest ranked listing 
for each unique molecular formula.
ParticipantID:        avaniya
Category:	          Category 1
Authors:              Vaniya, Arpana, Yan Ma, Dominique Ardura, Zijuan Lai, 
                      Mine Palazoglu, Sajjan Singh Mehta, Gert Wohlgemuth, 
                      Tobias Kind, Oliver Fiehn.
Affiliations:	      Genome Center, University of California, Davis 
Automatic methods:    Yes

Abstract:	
The challenge peak lists were converted to MSP files, and processed with Seven Golden
Rules and Formula Finder. The results were extracted from the Excel sheet. The 
isotope pattern of each potential formula was simulated and compared to the MS1 
spectrum of each challenge. The similarty was compared using Mass Frontier 7.0 where 
only the highest scoring pattern was be selected as the the best molecular formula.   
Databases and MS/MS mass spectral library search was performed to further verify the 
molecular formula. The Human Metabolome Database (HMDB) and MS/MS library search 
using NIST 14, MassBank, METLIN, ReSpect were used in this category. The scores from 
Seven Golden Rules were used for submission to CASMI.
ParticipantID:        Fitch 
Category:             Category 1 and Category 2
Authors:              William Fitch (1)
Affiliations:         (1) Stanford University (bfitch@stanford.edu)
Automatic methods:    No 

Abstract
I used manual methods to generate all structures and formulas. I did not have a good 
formula generator that used isotopic ratios. I did use ChemCalc sometimes and 
Xcalibur’s spectrum simulator. MS/MS interpretation was purely manual with a few 
database searches for confirmation. The dereplication choices among isomers were 
highly subjective based on experience and likely abundance in stated samples.

MS data is not often adequate to select among isomers.  We need LC retention. You 
gave me that data but I didn’t know how to use it. I recently published a 
metabolomics paper (Rapid Comm Mass Spec 2013, 27, 2091) in which I hinted at methods 
for making interlab retention data reproducible using sets of internal standards and 
well identified components. Unfortunately I have not found funding to pursue this 
avenue further.

Two other ideas for assisting these structure determinations, especially the unknowns 
would be deuterium exchange and retention at 2 ph values. Determining the number of 
exchangeable hydrogens is simple to perform, can help select among dereplication 
isomers and gives extra clues with true unknowns. The retention at 2 pH experiment 
requires some standardization but has the potential to indicate acid base character 
of the unknown.

Details per Challenge and Participant. See legend at bottom for more details

The table is also available as CSV download. The individual submissions are also available for download.

participant challenge rank tc bc wc ec rrp p wbc wwc wec wrrp
avaniya challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge1 - 18 - - - - - - - - -
felicityallen challenge1 1 14 0 13 1 1.00 0.22 0.00 0.78 0.00 1.00
fitch challenge1 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge1 1 4 0 3 1 1.00 0.28 0.00 0.72 0.00 1.00
tpluskal challenge1 1 13 0 12 1 1.00 0.08 0.00 0.92 0.00 1.00
avaniya challenge2 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge2 1 10 0 9 1 1.00 0.16 0.00 0.84 0.00 1.00
felicityallen challenge2 1 6 0 5 1 1.00 0.49 0.00 0.51 0.00 1.00
fitch challenge2 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge2 1 3 0 2 1 1.00 0.34 0.00 0.66 0.00 1.00
tpluskal challenge2 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
avaniya challenge3 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge3 1 6 0 5 1 1.00 0.29 0.00 0.71 0.00 1.00
felicityallen challenge3 1 9 0 8 1 1.00 0.34 0.00 0.66 0.00 1.00
fitch challenge3 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge3 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge3 1 3 0 2 1 1.00 0.35 0.00 0.65 0.00 1.00
avaniya challenge4 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge4 1 12 0 11 1 1.00 0.19 0.00 0.81 0.00 1.00
felicityallen challenge4 1 20 0 19 1 1.00 0.21 0.00 0.79 0.00 1.00
fitch challenge4 - 1 - - - - - - - - -
lridder challenge4 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge4 1 3 0 2 1 1.00 0.34 0.00 0.66 0.00 1.00
avaniya challenge5 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge5 1 6 0 5 1 1.00 0.18 0.00 0.82 0.00 1.00
felicityallen challenge5 1 9 0 8 1 1.00 0.46 0.00 0.54 0.00 1.00
fitch challenge5 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge5 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge5 1 2 0 1 1 1.00 0.51 0.00 0.49 0.00 1.00
avaniya challenge6 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge6 1 4 0 3 1 1.00 0.30 0.00 0.70 0.00 1.00
felicityallen challenge6 1 8 0 7 1 1.00 0.65 0.00 0.35 0.00 1.00
fitch challenge6 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge6 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge6 1 3 0 2 1 1.00 0.34 0.00 0.66 0.00 1.00
avaniya challenge7 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge7 2 4 1 2 1 0.67 0.27 0.28 0.45 0.00 0.72
felicityallen challenge7 1 3 0 2 1 1.00 0.55 0.00 0.45 0.00 1.00
fitch challenge7 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge7 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge7 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
avaniya challenge8 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge8 1 51 0 50 1 1.00 0.03 0.00 0.97 0.00 1.00
felicityallen challenge8 - 74 - - - - - - - - -
lridder challenge8 - 10 - - - - - - - - -
tpluskal challenge8 10 23 9 13 1 0.59 0.04 0.40 0.56 0.00 0.60
avaniya challenge9 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge9 2 83 1 81 1 0.99 0.02 0.02 0.96 0.00 0.98
felicityallen challenge9 1 30 0 29 1 1.00 0.23 0.00 0.77 0.00 1.00
fitch challenge9 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge9 1 5 0 4 1 1.00 0.39 0.00 0.61 0.00 1.00
tpluskal challenge9 10 14 9 4 1 0.31 0.07 0.65 0.28 0.00 0.35
avaniya challenge10 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge10 - 51 - - - - - - - - -
felicityallen challenge10 1 37 0 36 1 1.00 0.28 0.00 0.72 0.00 1.00
fitch challenge10 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge10 1 7 0 6 1 1.00 0.30 0.00 0.70 0.00 1.00
tpluskal challenge10 10 24 9 14 1 0.61 0.04 0.39 0.57 0.00 0.61
avaniya challenge11 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge11 3 7 2 4 1 0.67 0.20 0.42 0.38 0.00 0.58
felicityallen challenge11 1 9 0 8 1 1.00 0.42 0.00 0.58 0.00 1.00
fitch challenge11 - 1 - - - - - - - - -
lridder challenge11 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge11 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
avaniya challenge12 - 1 - - - - - - - - -
bertrandsamuel challenge12 1 10 0 9 1 1.00 0.16 0.00 0.84 0.00 1.00
felicityallen challenge12 1 10 0 9 1 1.00 0.23 0.00 0.77 0.00 1.00
fitch challenge12 - 1 - - - - - - - - -
lridder challenge12 1 2 0 1 1 1.00 0.64 0.00 0.36 0.00 1.00
tpluskal challenge12 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
avaniya challenge13 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge13 4 12 3 8 1 0.73 0.12 0.41 0.47 0.00 0.59
felicityallen challenge13 1 14 0 13 1 1.00 0.30 0.00 0.70 0.00 1.00
fitch challenge13 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge13 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge13 2 5 1 3 1 0.75 0.20 0.20 0.59 0.00 0.80
avaniya challenge14 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge14 1 13 0 12 1 1.00 0.13 0.00 0.87 0.00 1.00
felicityallen challenge14 1 12 0 11 1 1.00 0.18 0.00 0.82 0.00 1.00
fitch challenge14 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge14 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge14 1 3 0 2 1 1.00 0.35 0.00 0.65 0.00 1.00
avaniya challenge15 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge15 1 10 0 9 1 1.00 0.18 0.00 0.82 0.00 1.00
felicityallen challenge15 1 7 0 6 1 1.00 0.36 0.00 0.64 0.00 1.00
fitch challenge15 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge15 1 2 0 1 1 1.00 1.00 0.00 0.00 0.00 1.00
tpluskal challenge15 1 2 0 1 1 1.00 0.51 0.00 0.49 0.00 1.00
avaniya challenge16 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge16 4 15 3 11 1 0.79 0.08 0.24 0.69 0.00 0.76
felicityallen challenge16 1 17 0 16 1 1.00 0.10 0.00 0.90 0.00 1.00
fitch challenge16 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge16 1 3 0 2 1 1.00 0.45 0.00 0.55 0.00 1.00
tpluskal challenge16 1 5 0 4 1 1.00 0.21 0.00 0.79 0.00 1.00
avaniya challenge17 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge17 1 26 0 25 1 1.00 0.07 0.00 0.93 0.00 1.00
felicityallen challenge17 1 19 0 18 1 1.00 0.14 0.00 0.86 0.00 1.00
fitch challenge17 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge17 1 3 0 2 1 1.00 0.65 0.00 0.35 0.00 1.00
tpluskal challenge17 1 2 0 1 1 1.00 0.52 0.00 0.48 0.00 1.00
avaniya challenge18 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge18 1 34 0 33 1 1.00 0.06 0.00 0.94 0.00 1.00
felicityallen challenge18 1 20 0 19 1 1.00 0.09 0.00 0.91 0.00 1.00
fitch challenge18 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge18 1 8 0 7 1 1.00 0.42 0.00 0.58 0.00 1.00
tpluskal challenge18 1 7 0 6 1 1.00 0.15 0.00 0.85 0.00 1.00
avaniya challenge19 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge19 2 33 1 31 1 0.97 0.04 0.04 0.93 0.00 0.96
felicityallen challenge19 1 23 0 22 1 1.00 0.07 0.00 0.93 0.00 1.00
fitch challenge19 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge19 2 3 1 1 1 0.50 0.41 0.42 0.17 0.00 0.58
tpluskal challenge19 2 7 1 5 1 0.83 0.15 0.15 0.70 0.00 0.85
avaniya challenge20 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge20 1 28 0 27 1 1.00 0.05 0.00 0.95 0.00 1.00
felicityallen challenge20 1 38 0 37 1 1.00 0.07 0.00 0.93 0.00 1.00
fitch challenge20 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge20 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge20 1 5 0 4 1 1.00 0.21 0.00 0.79 0.00 1.00
avaniya challenge21 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge21 4 34 3 30 1 0.91 0.04 0.12 0.84 0.00 0.88
felicityallen challenge21 1 21 0 20 1 1.00 0.15 0.00 0.85 0.00 1.00
fitch challenge21 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge21 1 3 0 2 1 1.00 0.55 0.00 0.45 0.00 1.00
tpluskal challenge21 1 3 0 2 1 1.00 0.34 0.00 0.66 0.00 1.00
avaniya challenge22 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge22 1 28 0 27 1 1.00 0.05 0.00 0.95 0.00 1.00
felicityallen challenge22 1 27 0 26 1 1.00 0.13 0.00 0.87 0.00 1.00
fitch challenge22 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge22 1 2 0 1 1 1.00 0.88 0.00 0.13 0.00 1.00
tpluskal challenge22 3 3 2 0 1 0.00 0.33 0.67 0.00 0.00 0.33
avaniya challenge23 - 1 - - - - - - - - -
bertrandsamuel challenge23 - 26 - - - - - - - - -
felicityallen challenge23 - 43 - - - - - - - - -
fitch challenge23 - 1 - - - - - - - - -
lridder challenge23 - 1 - - - - - - - - -
tpluskal challenge23 3 4 2 1 1 0.33 0.25 0.50 0.25 0.00 0.50
avaniya challenge24 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge24 1 11 0 10 1 1.00 0.21 0.00 0.79 0.00 1.00
felicityallen challenge24 1 11 0 10 1 1.00 0.42 0.00 0.58 0.00 1.00
fitch challenge24 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge24 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge24 9 19 8 10 1 0.56 0.05 0.45 0.49 0.00 0.55
avaniya challenge25 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge25 - 4 - - - - - - - - -
felicityallen challenge25 1 22 0 21 1 1.00 0.23 0.00 0.77 0.00 1.00
fitch challenge25 - 1 - - - - - - - - -
lridder challenge25 - 9 - - - - - - - - -
tpluskal challenge25 - 21 - - - - - - - - -
avaniya challenge26 - 1 - - - - - - - - -
bertrandsamuel challenge26 15 26 14 11 1 0.44 0.04 0.67 0.29 0.00 0.33
felicityallen challenge26 - 107 - - - - - - - - -
fitch challenge26 - 1 - - - - - - - - -
lridder challenge26 - 8 - - - - - - - - -
tpluskal challenge26 12 25 11 13 1 0.54 0.04 0.45 0.51 0.00 0.55
avaniya challenge27 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge27 3 10 2 7 1 0.78 0.17 0.34 0.49 0.00 0.66
felicityallen challenge27 1 16 0 15 1 1.00 0.18 0.00 0.82 0.00 1.00
fitch challenge27 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge27 2 4 1 2 1 0.67 0.28 0.28 0.44 0.00 0.72
tpluskal challenge27 2 8 1 6 1 0.86 0.13 0.13 0.74 0.00 0.87
avaniya challenge28 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge28 1 35 0 34 1 1.00 0.04 0.00 0.96 0.00 1.00
felicityallen challenge28 1 46 0 45 1 1.00 0.08 0.00 0.92 0.00 1.00
fitch challenge28 - 1 - - - - - - - - -
lridder challenge28 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge28 2 8 1 6 1 0.86 0.13 0.13 0.74 0.00 0.87
avaniya challenge29 - 1 - - - - - - - - -
bertrandsamuel challenge29 2 41 1 39 1 0.97 0.03 0.03 0.93 0.00 0.97
felicityallen challenge29 - 25 - - - - - - - - -
fitch challenge29 - 1 - - - - - - - - -
lridder challenge29 - 2 - - - - - - - - -
tpluskal challenge29 - 5 - - - - - - - - -
avaniya challenge30 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge30 1 10 0 9 1 1.00 0.16 0.00 0.84 0.00 1.00
felicityallen challenge30 1 6 0 5 1 1.00 0.45 0.00 0.55 0.00 1.00
fitch challenge30 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge30 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge30 1 4 0 3 1 1.00 0.26 0.00 0.74 0.00 1.00
avaniya challenge31 - 1 - - - - - - - - -
bertrandsamuel challenge31 - 30 - - - - - - - - -
felicityallen challenge31 - 39 - - - - - - - - -
fitch challenge31 - 1 - - - - - - - - -
lridder challenge31 - 6 - - - - - - - - -
tpluskal challenge31 - 10 - - - - - - - - -
avaniya challenge32 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge32 1 7 0 6 1 1.00 0.23 0.00 0.77 0.00 1.00
felicityallen challenge32 1 8 0 7 1 1.00 0.65 0.00 0.35 0.00 1.00
fitch challenge32 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge32 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge32 1 3 0 2 1 1.00 0.34 0.00 0.66 0.00 1.00
avaniya challenge33 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge33 28 94 27 66 1 0.71 0.01 0.42 0.56 0.00 0.58
felicityallen challenge33 1 55 0 54 1 1.00 0.08 0.00 0.92 0.00 1.00
fitch challenge33 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge33 1 10 0 9 1 1.00 0.13 0.00 0.87 0.00 1.00
tpluskal challenge33 5 37 4 32 1 0.89 0.03 0.11 0.86 0.00 0.89
avaniya challenge34 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge34 1 52 0 51 1 1.00 0.02 0.00 0.98 0.00 1.00
felicityallen challenge34 1 29 0 28 1 1.00 0.11 0.00 0.89 0.00 1.00
fitch challenge34 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge34 1 10 0 9 1 1.00 0.17 0.00 0.83 0.00 1.00
tpluskal challenge34 1 46 0 45 1 1.00 0.02 0.00 0.98 0.00 1.00
avaniya challenge35 - 1 - - - - - - - - -
bertrandsamuel challenge35 1 2 0 1 1 1.00 0.51 0.00 0.49 0.00 1.00
felicityallen challenge35 1 12 0 11 1 1.00 0.35 0.00 0.65 0.00 1.00
fitch challenge35 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge35 1 5 0 4 1 1.00 0.72 0.00 0.28 0.00 1.00
tpluskal challenge35 5 6 4 1 1 0.20 0.16 0.68 0.16 0.00 0.32
avaniya challenge36 - 1 - - - - - - - - -
bertrandsamuel challenge36 - 6 - - - - - - - - -
felicityallen challenge36 - 7 - - - - - - - - -
fitch challenge36 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge36 1 3 0 2 1 1.00 0.41 0.00 0.59 0.00 1.00
tpluskal challenge36 1 9 0 8 1 1.00 0.12 0.00 0.88 0.00 1.00
avaniya challenge37 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge37 - 16 - - - - - - - - -
felicityallen challenge37 - 37 - - - - - - - - -
fitch challenge37 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge37 1 2 0 1 1 1.00 0.67 0.00 0.33 0.00 1.00
tpluskal challenge37 - 1 - - - - - - - - -
avaniya challenge38 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge38 1 41 0 40 1 1.00 0.03 0.00 0.97 0.00 1.00
felicityallen challenge38 1 26 0 25 1 1.00 0.06 0.00 0.94 0.00 1.00
fitch challenge38 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge38 1 10 0 9 1 1.00 0.19 0.00 0.81 0.00 1.00
tpluskal challenge38 1 9 0 8 1 1.00 0.12 0.00 0.88 0.00 1.00
avaniya challenge39 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge39 1 65 0 64 1 1.00 0.02 0.00 0.98 0.00 1.00
felicityallen challenge39 - 42 - - - - - - - - -
fitch challenge39 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge39 1 36 0 35 1 1.00 0.08 0.00 0.92 0.00 1.00
tpluskal challenge39 1 9 0 8 1 1.00 0.12 0.00 0.88 0.00 1.00
avaniya challenge40 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge40 1 8 0 7 1 1.00 0.19 0.00 0.81 0.00 1.00
felicityallen challenge40 1 10 0 9 1 1.00 0.20 0.00 0.80 0.00 1.00
fitch challenge40 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge40 1 4 0 3 1 1.00 0.43 0.00 0.57 0.00 1.00
tpluskal challenge40 1 6 0 5 1 1.00 0.17 0.00 0.83 0.00 1.00
avaniya challenge41 - 1 - - - - - - - - -
bertrandsamuel challenge41 - 30 - - - - - - - - -
felicityallen challenge41 1 78 0 77 1 1.00 0.04 0.00 0.96 0.00 1.00
fitch challenge41 - 1 - - - - - - - - -
lridder challenge41 1 8 0 7 1 1.00 0.20 0.00 0.80 0.00 1.00
tpluskal challenge41 2 2 1 0 1 0.00 0.50 0.50 0.00 0.00 0.50
avaniya challenge42 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge42 1 22 0 21 1 1.00 0.07 0.00 0.93 0.00 1.00
felicityallen challenge42 1 14 0 13 1 1.00 0.16 0.00 0.84 0.00 1.00
fitch challenge42 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge42 1 4 0 3 1 1.00 0.31 0.00 0.69 0.00 1.00
tpluskal challenge42 1 8 0 7 1 1.00 0.13 0.00 0.87 0.00 1.00
avaniya challenge43 - 1 - - - - - - - - -
bertrandsamuel challenge43 2 309 1 307 1 1.00 0.01 0.01 0.99 0.00 0.99
felicityallen challenge43 7 43 6 36 1 0.86 0.03 0.20 0.76 0.00 0.80
fitch challenge43 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge43 1 18 0 17 1 1.00 0.11 0.00 0.89 0.00 1.00
tpluskal challenge43 5 58 4 53 1 0.93 0.02 0.07 0.91 0.00 0.93
avaniya challenge44 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge44 1 44 0 43 1 1.00 0.04 0.00 0.96 0.00 1.00
felicityallen challenge44 1 13 0 12 1 1.00 0.18 0.00 0.82 0.00 1.00
fitch challenge44 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge44 1 3 0 2 1 1.00 1.00 0.00 0.00 0.00 1.00
tpluskal challenge44 1 6 0 5 1 1.00 0.17 0.00 0.83 0.00 1.00
avaniya challenge45 - 1 - - - - - - - - -
bertrandsamuel challenge45 1 9 0 8 1 1.00 0.15 0.00 0.85 0.00 1.00
felicityallen challenge45 1 2 0 1 1 1.00 0.70 0.00 0.30 0.00 1.00
fitch challenge45 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge45 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge45 1 2 0 1 1 1.00 0.51 0.00 0.49 0.00 1.00
avaniya challenge46 - 1 - - - - - - - - -
bertrandsamuel challenge46 1 121 0 120 1 1.00 0.01 0.00 0.99 0.00 1.00
felicityallen challenge46 1 29 0 28 1 1.00 0.08 0.00 0.92 0.00 1.00
fitch challenge46 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
lridder challenge46 1 13 0 12 1 1.00 0.22 0.00 0.78 0.00 1.00
tpluskal challenge46 1 17 0 16 1 1.00 0.06 0.00 0.94 0.00 1.00
avaniya challenge47 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
bertrandsamuel challenge47 1 170 0 169 1 1.00 0.01 0.00 0.99 0.00 1.00
felicityallen challenge47 1 5 0 4 1 1.00 0.24 0.00 0.76 0.00 1.00
lridder challenge47 1 5 0 4 1 1.00 0.30 0.00 0.70 0.00 1.00
tpluskal challenge47 2 14 1 12 1 0.92 0.07 0.07 0.85 0.00 0.93
avaniya challenge48 - 1 - - - - - - - - -
bertrandsamuel challenge48 - 79 - - - - - - - - -
felicityallen challenge48 - 7 - - - - - - - - -
lridder challenge48 1 1 0 0 1 - 1.00 0.00 0.00 0.00 1.00
tpluskal challenge48 2 17 1 15 1 0.94 0.06 0.06 0.88 0.00 0.94

Table legend:

rank
Absolute rank of correct solution
tc
Total number of candidates
bc
Number of candidates with a score better than correct solution
wc
Number of candidates with a score worse than correct solution
ec
Number of candidates with same score as the correct solution
rrp
Relative ranking position (1.0 is good, 0.0 is not)
p
Score of correct solution
wbc
Sum of scores better than correct solution
wwc
Sum of scores worse than correct solution
wec
Sum of scores equal to correct solution
wrrp
RRP weighted by the scores (1 is good)