Background

Skin cancer is the most common cancer globally, with melanoma being the most deadly form. Dermoscopy is a skin imaging modality that has demonstrated improvement for diagnosis of skin cancer compared to unaided visual inspection. However, clinicians should receive adequate training for those improvements to be realized. In order to make expertise more widely available, the International Skin Imaging Collaboration (ISIC) has developed the ISIC Archive, an international repository of dermoscopic images, for both the purposes of clinical training, and for supporting technical research toward automated algorithmic analysis by hosting the ISIC Challenges.

Task

The goal for ISIC 2019 is classify dermoscopic images among nine different diagnostic categories:

  1. Melanoma
  2. Melanocytic nevus
  3. Basal cell carcinoma
  4. Actinic keratosis
  5. Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis)
  6. Dermatofibroma
  7. Vascular lesion
  8. Squamous cell carcinoma
  9. None of the others

25,331 images are available for training across 8 different categories. Additionally, the test dataset (planned release August 2nd) will contain an additional outlier class not represented in the training data, which developed systems must be able to identify.

Two tasks will be available for participation: 1) classify dermoscopic images without meta-data, and 2) classify images with additional available meta-data. Task 1's deadline will be August 16th. Task 2 will be August 23th, after release of test meta-data on August 16th. Participants of Task 2 must submit to Task 1 as well, though participants can submit to Task 1 alone.

In addition to submitting predictions, each competitor is required to submit a link to a manuscript describing the methods used to generate predictions.

Submission

Submissions are made to the ISIC Challenge submission system, which provides automated format validation, pre-scoring, metadata editing capabilties.

More details about the submission process may be found on the submission details page.

Evaluation

Evaluation of algorithms is entirely automated, and rankings are determined by balanced multi-class accuracy. At the conclusion of the ISIC Challenge, all scores are posted to a public leaderboard, providing official rankings and more detailed metrics for every submission.

More details about the evaluation procedure may be found on the evaluation details page.

Awards

Cash prizes of $4,000, $2,000, and $1,000 will be awarded to the first, second, and third place participants of image-only and meta-data tasks. USD will be awarded to winners of each of the tasks. The monetary prizes for the winners of the challenge will be awarded after the MICCAI Workshop in Shenzhen, China, in October 2019. The prizes are being provided by Canfield Scientific, Inc., a US company, and are subject to any restrictions incumbent on the sponsor. Winners will be asked to identify a recipient individual or entity who will be required to provide tax documentation (U.S. citizens- IRS form W-9, non-U.S. citizens Form W-8 BEN).

Timeline

  • Training data release May 3, 2019
  • Test images release August 5, 2019
  • Submission deadline: images-only test August 16, 2019
  • Extended Test metadata release August 16, 2019
  • Extended Submission deadline: images and metadata test August 23, 2019
  • Extended Winners announced, and speaker invitations sent August 30, 2019
  • MICCAI2019 Workshop October 13 or 17, 2019

Sponsors

  • Canfield Scientific
  • IBM
  • MetaOptima

Clinical Chairs

  • Josep Malvehy, M.D.
    University Hospital Clinic of Barcelona, Barcelona, Spain
  • Allan Halpern, M.D.
    Memorial Sloan Kettering Cancer Center, New York City, NY, USA

Computer Vision Chairs

  • Noel C. F. Codella, Ph.D.
    IBM Research, Yorktown Heights, NY, USA

Challenge Co-Chairs

  • M. Emre Celebi, Ph.D.
    University of Central Arkansas, Conway, AR, USA
  • Marc Combalia, M.S.
    Fundació Clínic per a la Recerca Biomèdica, Barcelona, Spain
  • David Gutman, M.D., Ph.D.
    Emory University, Atlanta, GA, USA
  • Brian Helba
    Kitware, Inc., Clifton Park, NY, USA
  • Harald Kittler, M.D.
    Medical University of Vienna, Vienna, Austria
  • Veronica Rotemberg, M.D., Ph.D.
    Memorial Sloan Kettering Cancer Center, New York City, NY, USA
  • Philipp Tschandl, M.D., Ph.D.
    Medical University of Vienna, Vienna, Austria

Note: Any organizations/companies affiliated with members of the organizing committee are not excluded from participation in the Challenge, but must assure that their submissions are completely independent of the members of the organizing committee.