• Automation framework: CK
  
• Development repository: ck-env
  
• Source: GitHub
  
• Tags: dataset,imagenet,calibration,cal
  
• Support for host OS: any
  
• Support for target OS: any
  
• Software detection plugin: dataset.imagenet.calibration
  
• How to get the stable version from this portal via the CK client:
  
  pip install cbench
  cb download package:dataset-imagenet-calibration --version=2.5.0 --all
  ck install package:dataset-imagenet-calibration
  ck help package
  
• How to get the development version:
  
  pip install ck
  ck pull repo:ck-env
  ck install package:dataset-imagenet-calibration
  ck help package
  
• Useful CK automation actions (CLI with the Python CK API and JSON IO):
  
  ck show env
  ck find env:{ID from above}
  ck virtual env:{ID from above}
  
• CLI and Python API: module:package
  
• Variations:
  • all.500
  • first.500
  • first.1
  • first.5
  • first.1.dup.5
  • mlperf.option1
  • mlperf.option2

ImageNet calibration datasets

This package takes an ImageNet 2012 validation dataset with original JPEG images (the full one with 50,000 images or a reduced one with the first 500 images of the full one - tagged "min") and creates a calibration dataset according to one of the below options (CK variations). A val_map.txt file gets generated too, matching the file names and the class labels on the same line.

Accuracy with OpenVINO "pre-release"

NB: For official MLPerf Inference submissions on 50,000 images, mlperf.option2 should be used for MobileNet and mlperf.option1 should be used for ResNet. Surprisingly, first.500 is never the worst, and is actually the best for ResNet on 50,000 images.

Variations

All 500 images of the "min" dataset

$ ck install package --tags=dataset,imagenet,all.500

First 500 images (default)

$ ck install package --tags=dataset,imagenet,first.500

NB: Equivalent to all.500 for the "min" dataset.

MLPerf Inference option 1

$ ck install package --tags=dataset,imagenet,mlperf.option1

NB: This option was used for MLPerf Inference v0.5 by Intel and others. In fact, we use Intel's file, because the official file does not have class labels.

MLPerf Inference option 2

$ ck install package --tags=dataset,imagenet,mlperf.option2

NB: The official file does have class labels in the second column (despite appearing to have just the increasing numbering).

Unit-tests

$ ck install package --tags=dataset,imagenet,first.1
$ ck install package --tags=dataset,imagenet,first.5
$ ck install package --tags=dataset,imagenet,first.1.dup.5

NB: first.1 and first.5 use a file list (with the first and the first 5 images, respectively) with a file name but without a class label. first.1.dup.5 duplicates the same file name 5 times.

Accuracy with OpenVINO "pre-release"

• 2.5.0 (2020-09-04)

• .cm/desc.json (3 bytes)
• .cm/info.json (314 bytes)
• .cm/meta.json (3644 bytes)
• README.md (4137 bytes)
• cal_image_list_first_1.txt (29 bytes)
• cal_image_list_first_1_dup_5.txt (145 bytes)
• cal_image_list_first_5.txt (145 bytes)
• cal_image_list_option_1.txt (16441 bytes)
• cal_image_list_option_2.txt (16390 bytes)
• install.sh (1118 bytes)