Using models generated by NVIDIA for MLPerf v0.5 submission: IMAGE CLASSIFICATION: ===================== Detect preprocessed ImageNet: ```bash ck detect soft:dataset.imagenet.preprocessed \ --full_path=/datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224/ILSVRC2012_val_00000001.rgb8 \ --extra_tags=preprocessed,using-opencv,universal,crop.875,full,inter.linear,side.224 \ --cus.version=opencv-linear ``` (If generated on this machine:) Detect ResNet50 model for GPU: ```bash ck detect soft:model.tensorrt --full_path=/datasets/inference_results_v0.5/closed/NVIDIA/build/engines/Xavier/resnet/MultiStream/resnet-MultiStream-gpu-b70-int8.plan \ --extra_tags=int8,linear,gpu,maxbatch.70,resnet,resnet50,image-classification,converted-by-nvidia \ --cus.version=resnet_gpu_b70_int8 \ --ienv.ML_MODEL_MAX_BATCH_SIZE=70 \ --ienv.ML_MODEL_COLOUR_CHANNELS_BGR=NO \ --ienv.ML_MODEL_IMAGE_HEIGHT=224 \ --ienv.ML_MODEL_IMAGE_WIDTH=224 \ --ienv.ML_MODEL_INPUT_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_LAYOUT=NCHW \ --ienv.ML_MODEL_NORMALIZE_DATA=NO \ --ienv.ML_MODEL_SUBTRACT_MEAN=YES \ --ienv.ML_MODEL_GIVEN_CHANNEL_MEANS="123.68 116.78 103.94" ``` (Otherwise:) Install ResNet50 model: ```bash ck install package --tags=downloaded,tensorrt,model,resnet ``` (If generated on this machine:) Detect MobileNet model for GPU: ```bash ck detect soft:model.tensorrt --full_path=/datasets/inference_results_v0.5/closed/NVIDIA/build/engines/Xavier/mobilenet/MultiStream/mobilenet-MultiStream-gpu-b250-int8.plan \ --extra_tags=maxbatch.250,int8,linear,mobilenet,gpu,image-classification,converted-by-nvidia \ --cus.version=mobilenet_gpu_b250_int8 \ --ienv.ML_MODEL_MAX_BATCH_SIZE=250 \ --ienv.ML_MODEL_COLOUR_CHANNELS_BGR=NO \ --ienv.ML_MODEL_IMAGE_HEIGHT=224 \ --ienv.ML_MODEL_IMAGE_WIDTH=224 \ --ienv.ML_MODEL_INPUT_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_LAYOUT=NCHW \ --ienv.ML_MODEL_NORMALIZE_DATA=NO \ --ienv.ML_MODEL_SUBTRACT_MEAN=YES \ --ienv.ML_MODEL_GIVEN_CHANNEL_MEANS="128 128 128" ck detect soft:model.tensorrt --full_path=/datasets/xavier-original-zenodo/mobilenet-Offline-gpu-b512-int8-chw4.plan \ --extra_tags=maxbatch.512,int8,chw4,mobilenet,gpu,image-classification,converted-by-nvidia \ --cus.version=mobilenet_gpu_b512_int8_chw4 \ --ienv.ML_MODEL_MAX_BATCH_SIZE=512 \ --ienv.ML_MODEL_COLOUR_CHANNELS_BGR=NO \ --ienv.ML_MODEL_IMAGE_HEIGHT=224 \ --ienv.ML_MODEL_IMAGE_WIDTH=224 \ --ienv.ML_MODEL_INPUT_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_LAYOUT=CHW4 \ --ienv.ML_MODEL_USE_DLA=NO \ --ienv.ML_MODEL_NORMALIZE_DATA=NO \ --ienv.ML_MODEL_SUBTRACT_MEAN=YES \ --ienv.ML_MODEL_GIVEN_CHANNEL_MEANS="128 128 128" ck detect soft:model.tensorrt --full_path=/datasets/xavier-original-zenodo/mobilenet-Offline-dla-b32-int8-chw4.plan \ --extra_tags=maxbatch.32,int8,chw4,mobilenet,dla,image-classification,converted-by-nvidia \ --cus.version=mobilenet_dla_b32_int8_chw4 \ --ienv.ML_MODEL_MAX_BATCH_SIZE=32 \ --ienv.ML_MODEL_COLOUR_CHANNELS_BGR=NO \ --ienv.ML_MODEL_IMAGE_HEIGHT=224 \ --ienv.ML_MODEL_IMAGE_WIDTH=224 \ --ienv.ML_MODEL_INPUT_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_LAYOUT=CHW4 \ --ienv.ML_MODEL_USE_DLA=YES \ --ienv.ML_MODEL_NORMALIZE_DATA=NO \ --ienv.ML_MODEL_SUBTRACT_MEAN=YES \ --ienv.ML_MODEL_GIVEN_CHANNEL_MEANS="128 128 128" ``` (Otherwise:) Install MobileNet model: ```bash ck install package --tags=downloaded,tensorrt,model,mobilenet ``` Standalone MobileNet without Loadgen: ```bash time ck run program:image-classification-tensorrt-py --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.weights=gpu,mobilenet --env.CK_BATCH_SIZE=250 --env.CK_BATCH_COUNT=200 ``` Standalone MobileNet with Loadgen: ```bash time ck run program:image-classification-tensorrt-loadgen-py --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.weights=mobilenet,gpu --env.CK_LOADGEN_SCENARIO=MultiStream --env.CK_LOADGEN_MODE=AccuracyOnly \ --env.CK_LOADGEN_BUFFER_SIZE=500 --env.CK_LOADGEN_DATASET_SIZE=50000 \ --env.CK_LOADGEN_MULTISTREAMNESS=250 --env.CK_BATCH_SIZE=250 ``` OBJECT DETECTION: ================= Flat labels: ```bash [sudo] cp `ck find package:model-onnx-mlperf-ssd-mobilenet`/flatlabels.txt /datasets/coco_flatlabels.txt ``` Detect unpreprocessed COCO-2017: ```bash ck detect soft:dataset.coco.2017.val \ --full_path=/datasets/inference_results_v0.5/closed/NVIDIA/build/data/coco/val2017/000000000139.jpg \ --cus.version=coco_2017_val \ --ienv.VAL_DIR=val2017 \ --ienv.LABELS_DIR=annotations \ --cus.instances_val_file=instances_val2017.json ``` Detect OpenCV python package: ```bash ck detect soft --tags=python-package,cv2 --full_path=/usr/lib/python3.6/dist-packages/cv2/__init__.py ``` Preprocess all 5k coco-2017 images using-opencv for SSD-MobileNet: ```bash ck install package --tags=dataset,preprocessed,using-opencv,coco.2017,full,side.300 ``` SSD-MobileNet ------------- (If generated on this machine:) Detect SSD-MobileNet model for GPU: ```bash ck detect soft:model.tensorrt --full_path=/datasets/inference_results_v0.5/closed/NVIDIA/build/engines/Xavier/ssd-small/Offline/ssd-small-Offline-gpu-b128-int8.plan \ --extra_tags=maxbatch.128,int8,linear,ssd-mobilenet,gpu,object-detection,converted-by-nvidia,side.300 \ --cus.version=ssd-mobilenet_nvidia_int8_linear \ --ienv.ML_MODEL_TENSORRT_PLUGIN=/datasets/inference_results_v0.5/closed/NVIDIA/build/plugins/NMSOptPlugin/libnmsoptplugin.so \ --ienv.ML_MODEL_CLASS_LABELS=/datasets/coco_flatlabels.txt \ --ienv.ML_MODEL_MAX_BATCH_SIZE=128 \ --ienv.ML_MODEL_COLOUR_CHANNELS_BGR=NO \ --ienv.ML_MODEL_IMAGE_HEIGHT=300 \ --ienv.ML_MODEL_IMAGE_WIDTH=300 \ --ienv.ML_MODEL_INPUT_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_LAYOUT=NCHW \ --ienv.ML_MODEL_NORMALIZE_DATA=NO \ --ienv.ML_MODEL_SUBTRACT_MEAN=YES \ --ienv.ML_MODEL_MAX_PREDICTIONS=100 \ --ienv.ML_MODEL_GIVEN_CHANNEL_MEANS="128 128 128" ``` (Otherwise:) Install SSD-MobileNet model: ```bash ck install package --tags=downloaded,tensorrt,model,ssd-mobilenet ``` Standalone SSD-MobileNet without LoadGen: ```bash time ck run program:object-detection-tensorrt-py \ --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.weights=ssd-mobilenet,int8,linear --dep_add_tags.dataset=side.300 \ --env.CK_BATCH_SIZE=100 --env.CK_BATCH_COUNT=50 ``` Standalone SSD-MobileNet with LoadGen: ```bash time ck run program:object-detection-tensorrt-loadgen-py --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.weights=ssd-mobilenet,int8,linear --dep_add_tags.dataset=side.300 \ --env.CK_LOADGEN_SCENARIO=MultiStream --env.CK_LOADGEN_MODE=AccuracyOnly \ --env.CK_LOADGEN_BUFFER_SIZE=500 --env.CK_LOADGEN_DATASET_SIZE=5000 \ --env.CK_LOADGEN_MULTISTREAMNESS=100 --env.CK_BATCH_SIZE=100 ``` ZeroMQ SSD-MobileNet without LoadGen: ```bash Worker$ ck run program:zpp-worker-tensorrt-py --env.CK_WORKER_OUTPUT_FORMAT=direct_return \ --dep_add_tags.weights=ssd-mobilenet,int8,linear Hub$ time ck run program:object-detection-zpp-hub-py \ --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.dataset=side.300 --dep_add_tags.weights=ssd-mobilenet,int8,linear \ --env.CK_BATCH_SIZE=125 --env.CK_BATCH_COUNT=40 ``` ZeroMQ SSD-MobileNet with LoadGen: ```bash Worker$ ck run program:zpp-worker-tensorrt-py --env.CK_WORKER_OUTPUT_FORMAT=direct_return \ --dep_add_tags.weights=ssd-mobilenet,int8,linear Hub$ time ck run program:object-detection-zpp-hub-loadgen-py \ --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.dataset=side.300 --dep_add_tags.weights=ssd-mobilenet,int8,linear \ --env.CK_LOADGEN_SCENARIO=MultiStream --env.CK_LOADGEN_MODE=AccuracyOnly \ --env.CK_LOADGEN_BUFFER_SIZE=500 --env.CK_LOADGEN_DATASET_SIZE=5000 \ --env.CK_LOADGEN_MULTISTREAMNESS=25 --env.CK_BATCH_SIZE=25 ``` SSD-ResNet34 ------------ (If generated on this machine:) Detect SSD-ResNet34 model for GPU: ```bash ck detect soft:model.tensorrt --full_path=/datasets/inference_results_v0.5/closed/NVIDIA/build/engines/Xavier/ssd-large/MultiStream/ssd-large-MultiStream-gpu-b2-int8.plan \ --extra_tags=maxbatch.2,int8,linear,gpu,ssd-resnet,object-detection,converted-by-nvidia,side.1200 \ --cus.version=ssd-resnet_nvidia_int8_linear \ --ienv.ML_MODEL_TENSORRT_PLUGIN=/datasets/inference_results_v0.5/closed/NVIDIA/build/plugins/NMSOptPlugin/libnmsoptplugin.so \ --ienv.ML_MODEL_CLASS_LABELS=/datasets/coco_flatlabels.txt \ --ienv.ML_MODEL_SKIPS_ORIGINAL_DATASET_CLASSES=12,26,29,30,45,66,68,69,71,83 \ --ienv.ML_MODEL_MAX_BATCH_SIZE=2 \ --ienv.ML_MODEL_COLOUR_CHANNELS_BGR=NO \ --ienv.ML_MODEL_IMAGE_HEIGHT=1200 \ --ienv.ML_MODEL_IMAGE_WIDTH=1200 \ --ienv.ML_MODEL_INPUT_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_TYPE=int8 \ --ienv.ML_MODEL_DATA_LAYOUT=NCHW \ --ienv.ML_MODEL_NORMALIZE_DATA=NO \ --ienv.ML_MODEL_SUBTRACT_MEAN=YES \ --ienv.ML_MODEL_GIVEN_CHANNEL_MEANS="123.68 116.78 103.94" \ --ienv.ML_MODEL_MAX_PREDICTIONS=200 ``` (Otherwise:) Install SSD-ResNet34 model: ```bash ck install package --tags=downloaded,tensorrt,model,ssd-resnet ``` Preprocess all 5k coco-2017 images using-opencv for SSD-ResNet34: ```bash ck install package --tags=dataset,preprocessed,using-opencv,coco.2017,full,side.1200 ``` Standalone SSD-ResNet34 without LoadGen: ```bash time ck run program:object-detection-tensorrt-py \ --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.weights=ssd-resnet --dep_add_tags.dataset=side.1200 \ --env.CK_BATCH_SIZE=2 --env.CK_BATCH_COUNT=2500 ``` Standalone SSD-ResNet34 with LoadGen: ```bash time ck run program:object-detection-tensorrt-loadgen-py --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.weights=ssd-resnet,gpu --dep_add_tags.dataset=side.1200 \ --env.CK_LOADGEN_SCENARIO=Offline --env.CK_LOADGEN_MODE=AccuracyOnly \ --env.CK_LOADGEN_BUFFER_SIZE=500 --env.CK_LOADGEN_DATASET_SIZE=5000 \ --env.CK_BATCH_SIZE=2 ``` ZeroMQ SSD-ResNet34 without LoadGen: ```bash Worker$ ck run program:zpp-worker-tensorrt-py --env.CK_WORKER_OUTPUT_FORMAT=direct_return \ --dep_add_tags.weights=ssd-resnet,int8,linear Hub$ time ck run program:object-detection-zpp-hub-py \ --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.dataset=side.1200 --dep_add_tags.weights=ssd-resnet,int8,linear \ --env.CK_BATCH_SIZE=2 --env.CK_BATCH_COUNT=2500 ``` ZeroMQ SSD-ResNet34 with LoadGen: ```bash Worker$ ck run program:zpp-worker-tensorrt-py --env.CK_WORKER_OUTPUT_FORMAT=direct_return \ --dep_add_tags.weights=ssd-resnet,int8,linear Hub$ time ck run program:object-detection-zpp-hub-loadgen-py \ --skip_print_timers --env.CK_SILENT_MODE \ --dep_add_tags.dataset=side.1200 --dep_add_tags.weights=ssd-resnet,int8,linear \ --env.CK_LOADGEN_SCENARIO=MultiStream --env.CK_LOADGEN_MODE=AccuracyOnly \ --env.CK_LOADGEN_BUFFER_SIZE=250 --env.CK_LOADGEN_DATASET_SIZE=5000 \ --env.CK_LOADGEN_MULTISTREAMNESS=2 --env.CK_BATCH_SIZE=2 ```