{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# MLPerf Inference v0.7 - Reproducing Xavier results with JetPack 4.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
"\n",
"1. [System](#system)\n",
"1. [Installation](#installation)\n",
" 1. [Clone the repo](#installation_repo)\n",
" 1. [Link the datasets](#installation_datasets)\n",
" 1. [Download the models](#installation_models)\n",
" 1. [Build the harness](#installation_harness)\n",
"1. [ResNet50](#resnet50)\n",
" 1. [Offline](#resnet50_offline)\n",
" 1. [Build](#resnet50_offline_build)\n",
" 1. [Performance](#resnet50_offline_performance)\n",
" 1. [Accuracy](#resnet50_offline_accuracy)\n",
" 1. [SingleStream](#resnet50_singlestream)\n",
" 1. [Build](#resnet50_singlestream_build)\n",
" 1. [Performance](#resnet50_singlestream_performance)\n",
" 1. [Accuracy](#resnet50_singlestream_accuracy)\n",
" 1. [MultiStream](#resnet50_multistream)\n",
" 1. [Build](#resnet50_multistream_build)\n",
" 1. [Performance](#resnet50_multistream_performance)\n",
" 1. [Accuracy](#resnet50_multistream_accuracy)\n",
"1. [SSD-ResNet34](#ssd-resnet34)\n",
" 1. [Offline](#ssd-resnet34_offline)\n",
" 1. [Build](#ssd-resnet34_offline_build)\n",
" 1. [Performance](#ssd-resnet34_offline_performance)\n",
" 1. [Accuracy](#ssd-resnet34_offline_accuracy)\n",
" 1. [SingleStream](#ssd-resnet34_singlestream)\n",
" 1. [Build](#ssd-resnet34_singlestream_build)\n",
" 1. [Performance](#ssd-resnet34_singlestream_performance)\n",
" 1. [Accuracy](#ssd-resnet34_singlestream_accuracy)\n",
" 1. [MultiStream](#ssd-resnet34_multistream)\n",
" 1. [Build](#ssd-resnet34_multistream_build)\n",
" 1. [Performance](#ssd-resnet34_multistream_performance)\n",
" 1. [Accuracy](#ssd-resnet34_multistream_accuracy)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# System: [NVIDIA AGX Xavier](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/systems/AGX_Xavier_TRT.json)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Stats: JetPack 4.5\n",
"\n",
"anton@xavier:~$ sudo -H python3 -m pip install jetson-stats -U\n",
"...\n",
"Successfully installed jetson-stats-3.0.3\n",
"\n",
"\n",
"\n",
"anton@xavier:~$ jetson_release \n",
" - NVIDIA Jetson AGX Xavier [16GB]\n",
" * Jetpack 4.5 [L4T 32.5.0]\n",
" * NV Power Mode: MAXN - Type: 0\n",
" * jetson_stats.service: active\n",
" - Libraries:\n",
" * CUDA: 10.2.89\n",
" * cuDNN: 8.0.0.180\n",
" * TensorRT: 7.1.3.0\n",
" * Visionworks: 1.6.0.501\n",
" * OpenCV: 4.1.1 compiled CUDA: NO\n",
" * VPI: ii libnvvpi1 1.0.12 arm64 NVIDIA Vision Programming Interface library\n",
" * Vulkan: 1.2.70\n",
"\n",
"\n",
"## OS: Ubuntu 18.04.5 LTS\n",
"\n",
"\n",
"anton@xavier:~$ cat /etc/lsb-release\n",
"DISTRIB_ID=Ubuntu\n",
"DISTRIB_RELEASE=18.04\n",
"DISTRIB_CODENAME=bionic\n",
"DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"\n",
"\n",
"\n",
"## CPU: 8-core ARMv8 @ 2265 MHz\n",
"\n",
"\n",
"anton@xavier:~$ lscpu\n",
"Architecture: aarch64\n",
"Byte Order: Little Endian\n",
"CPU(s): 8\n",
"On-line CPU(s) list: 0-7\n",
"Thread(s) per core: 1\n",
"Core(s) per socket: 2\n",
"Socket(s): 4\n",
"Vendor ID: Nvidia\n",
"Model: 0\n",
"Model name: ARMv8 Processor rev 0 (v8l)\n",
"Stepping: 0x0\n",
"CPU max MHz: 2265.6001\n",
"CPU min MHz: 115.2000\n",
"BogoMIPS: 62.50\n",
"L1d cache: 64K\n",
"L1i cache: 128K\n",
"L2 cache: 2048K\n",
"L3 cache: 4096K\n",
"Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp\n",
"\n",
"\n",
"## GPU\n",
"\n",
"\n",
"anton@xavier:~$ sudo jetson_clocks\n",
"anton@xavier:~$ sudo jetson_clocks --show\n",
"SOC family:tegra194 Machine:Jetson-AGX\n",
"Online CPUs: 0-7\n",
"cpu0: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu1: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu2: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu3: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu4: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu5: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu6: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu7: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000\n",
"EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1\n",
"Fan: PWM=0\n",
"NV Power Mode: MAXN\n",
"\n",
"\n",
"\n",
"\n",
"anton@xavier:~$ ck compile program:tool-print-cuda-devices\n",
"anton@xavier:~$ ck run program:tool-print-cuda-devices\n",
"...\n",
"GPU Device ID: 0\n",
"GPU Name: Xavier\n",
"GPU compute capability: 7.2\n",
"CUDA driver version: 10.2\n",
"CUDA runtime version: 10.2\n",
"Global memory: 33479647232\n",
"Max clock rate: 1377.000000 MHz\n",
"Total amount of shared memory per block: 49152\n",
"Total number of registers available per block: 65536\n",
"Warp size: 32\n",
"Maximum number of threads per multiprocessor: 2048\n",
"Maximum number of threads per block: 1024\n",
"Max dimension size of a thread block X: 1024\n",
"Max dimension size of a thread block Y: 1024\n",
"Max dimension size of a thread block Z: 64\n",
"Max dimension size of a grid size X: 2147483647\n",
"Max dimension size of a grid size Y: 65535\n",
"Max dimension size of a grid size Z: 65535\n",
"\n",
"\n",
"\n",
"## Disks\n",
"\n",
"anton@xavier:~$ df -h\n",
"Filesystem Size Used Avail Use% Mounted on\n",
"/dev/mmcblk0p1 28G 17G 11G 62% /\n",
"none 16G 0 16G 0% /dev\n",
"tmpfs 16G 52K 16G 1% /dev/shm\n",
"tmpfs 16G 30M 16G 1% /run\n",
"tmpfs 5.0M 4.0K 5.0M 1% /run/lock\n",
"tmpfs 16G 0 16G 0% /sys/fs/cgroup\n",
"/dev/mmcblk1p1 361G 300G 42G 88% /sd\n",
"tmpfs 3.2G 12K 3.2G 1% /run/user/120\n",
"tmpfs 3.2G 0 3.2G 0% /run/user/1000\n",
"tmpfs 3.2G 0 3.2G 0% /run/user/1004\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Clone the v0.7 results repo\n",
"\n",
"anton@xavier:~$ git clone https://github.com/mlcommons/inference_results_v0.7.git
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Link the datasets\n",
"\n",
"We reused the datasets we generated while reproducing the v0.5 results.\n",
"\n",
"\n",
"anton@xavier:~$ export MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path\n",
"anton@xavier:~$ mkdir $MLPERF_SCRATCH_PATH\n",
"anton@xavier:~$ cd $MLPERF_SCRATCH_PATH\n",
"anton@xavier:/datasets/mlperf_scratch_path$ ln -s /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/preprocessed_data preprocessed_data\n",
"anton@xavier:/datasets/mlperf_scratch_path$ ln -s /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/data data\n",
"anton@xavier:/datasets/mlperf_scratch_path$ ls -la /datasets/mlperf_scratch_path/\n",
"total 20\n",
"drwxrwsr-x 3 anton dvdt 4096 Jan 28 11:10 .\n",
"drwxrwsr-x 17 root dvdt 4096 Jan 28 09:53 ..\n",
"lrwxrwxrwx 1 anton dvdt 64 Jan 28 11:10 data -> /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/data\n",
"drwxrwsr-x 5 anton dvdt 4096 Jan 28 09:54 models\n",
"lrwxrwxrwx 1 anton dvdt 77 Jan 28 11:10 preprocessed_data -> /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/preprocessed_data\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Download the models\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ git diff Makefile\n",
"diff --git a/closed/NVIDIA/Makefile b/closed/NVIDIA/Makefile\n",
"index 4679ec29..5bdf6a59 100644\n",
"--- a/closed/NVIDIA/Makefile\n",
"+++ b/closed/NVIDIA/Makefile\n",
"@@ -218,7 +218,7 @@ endif\n",
"\n",
" ############################## DOWNLOAD_MODEL ##############################\n",
"\n",
"-BENCHMARKS = resnet50 ssd-resnet34 ssd-mobilenet bert dlrm rnnt 3d-unet\n",
"+BENCHMARKS = resnet50 ssd-resnet34 ssd-mobilenet bert rnnt # dlrm 3d-unet\n",
"\n",
" .PHONY: download_model\n",
" download_model: link_dirs\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path make download_model
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Build the harness\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ git diff code/harness/\n",
"diff --git a/closed/NVIDIA/code/harness/lwis/include/lwis_buffers.h b/closed/NVIDIA/code/harness/lwis/include/lwis_buffers.h\n",
"index 5a79260c..4a9e52de 100644\n",
"--- a/closed/NVIDIA/code/harness/lwis/include/lwis_buffers.h\n",
"+++ b/closed/NVIDIA/code/harness/lwis/include/lwis_buffers.h\n",
"@@ -80,10 +80,10 @@ inline int64_t volume(const nvinfer1::Dims& d, const nvinfer1::TensorFormat& for\n",
" case nvinfer1::TensorFormat::kCHW2: spv = 2; channelDim = d_new.nbDims - 3; break;\n",
" case nvinfer1::TensorFormat::kCHW4: spv = 4; channelDim = d_new.nbDims - 3; break;\n",
" case nvinfer1::TensorFormat::kHWC8: spv = 8; channelDim = d_new.nbDims - 3; break;\n",
"- case nvinfer1::TensorFormat::kDHWC8: spv = 8; channelDim = d_new.nbDims - 4; break;\n",
"+ //case nvinfer1::TensorFormat::kDHWC8: spv = 8; channelDim = d_new.nbDims - 4; break;\n",
" case nvinfer1::TensorFormat::kCHW16: spv = 16; channelDim = d_new.nbDims - 3; break;\n",
" case nvinfer1::TensorFormat::kCHW32: spv = 32; channelDim = d_new.nbDims - 3; break;\n",
"- case nvinfer1::TensorFormat::kCDHW32: spv = 32; channelDim = d_new.nbDims - 4; break;\n",
"+ //case nvinfer1::TensorFormat::kCDHW32: spv = 32; channelDim = d_new.nbDims - 4; break;\n",
" case nvinfer1::TensorFormat::kLINEAR:\n",
" default: spv = 1; channelDim = -1; break;\n",
" }\n",
"\n",
"\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ make build\n",
"...\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ ls -la build/\n",
"total 20\n",
"drwxrwsr-x 5 anton dvdt 4096 Jan 28 16:06 .\n",
"drwxrwsr-x 12 anton dvdt 4096 Jan 28 09:53 ..\n",
"lrwxrwxrwx 1 anton dvdt 34 Jan 28 09:54 data -> /datasets/mlperf_scratch_path/data\n",
"drwxrwsr-x 3 anton dvdt 4096 Jan 28 11:10 engines\n",
"drwxrwsr-x 16 anton dvdt 4096 Jan 28 16:05 inference\n",
"lrwxrwxrwx 1 anton dvdt 36 Jan 28 09:54 models -> /datasets/mlperf_scratch_path/models\n",
"drwxrwsr-x 3 anton dvdt 4096 Jan 28 16:06 plugins\n",
"lrwxrwxrwx 1 anton dvdt 47 Jan 28 09:54 preprocessed_data -> /datasets/mlperf_scratch_path/preprocessed_data\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# ResNet50"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlperf/inference_results_v0.7/blob/master/closed/NVIDIA/configs/resnet50/Offline/config.json)\n",
"\n",
"anton@xavier:~$ grep AGX_Xavier /datasets/inference_results_v0.7/closed/NVIDIA/configs/resnet50/Offline/config.json -A 17\n",
" "AGX_Xavier": {\n",
" "concurrent_offline_expected_qps": 2181,\n",
" "dla_batch_size": 32,\n",
" "dla_copy_streams": 1,\n",
" "dla_core": 0,\n",
" "dla_inference_streams": 1,\n",
" "dla_offline_expected_qps": 396,\n",
" "gpu_batch_size": 64,\n",
" "gpu_copy_streams": 1,\n",
" "gpu_inference_streams": 1,\n",
" "gpu_offline_expected_qps": 1478.33,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/imagenet/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=resnet50 --scenarios=offline"\n",
"...\n",
"[2021-01-29 08:04:08,685 main.py:153 INFO] Finished building engines for resnet50 benchmark in Offline scenario.\n",
"Time taken to generate engines: 65.47717833518982 seconds\n",
"\n",
"real 1m6.612s\n",
"user 0m24.812s\n",
"sys 0m7.088s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ find . \\\n",
"-name resnet50-Offline*.plan -exec du -hs {} \\;\n",
"48M ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-gpu-b64-int8.default.plan\n",
"33M ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-dla-b32-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/Offline/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per second: 2074.53\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment (~53 W)\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=offline --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server\n",
"Scenario : Offline\n",
"Mode : Performance\n",
"Samples per second: 2087.07\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"Min latency (ns) : 47041359\n",
"Max latency (ns) : 68970284749\n",
"Mean latency (ns) : 34509141573\n",
"50.00 percentile latency (ns) : 34507536580\n",
"90.00 percentile latency (ns) : 62052439865\n",
"95.00 percentile latency (ns) : 65528745973\n",
"97.00 percentile latency (ns) : 66902483725\n",
"99.00 percentile latency (ns) : 68276603731\n",
"99.90 percentile latency (ns) : 68878743387\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 143946\n",
"target_qps : 2181\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 2048\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 1 batches of size 10\n",
" 1551 batches of size 64\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 1408\n",
" BatchedCudaMemcpy Calls: 1530\n",
"Device Device:0.DLA-0 processed:\n",
" 696 batches of size 32\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 0\n",
"Device Device:0.DLA-1 processed:\n",
" 700 batches of size 32\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 0\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-02-10 22:03:53,421 main.py:341 INFO] Result: Samples per second: 2087.07 and Result is : VALID\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: Samples per second: 2087.07 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 1m24.386s\n",
"user 1m13.924s\n",
"sys 0m5.764s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/Offline/accuracy/accuracy.txt)\n",
"\n",
"\n",
"accuracy=76.004%, good=38002, total=50000\n",
"hash=745b138bf6552d30a42b23fde8f2519867d16a6ca319aad954e4dae897bf452c\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=offline --test_mode=AccuracyOnly"\n",
"[2021-01-28 23:26:07,164 main.py:692 INFO] Detected System ID: AGX_Xavier\n",
"[2021-01-28 23:26:07,164 main.py:546 INFO] Using config files: configs/resnet50/Offline/config.json\n",
"[2021-01-28 23:26:07,165 __init__.py:283 INFO] Parsing config file configs/resnet50/Offline/config.json ...\n",
"[2021-01-28 23:26:07,166 main.py:559 INFO] Processing config \"AGX_Xavier_resnet50_Offline\"\n",
"[2021-01-28 23:26:07,166 main.py:294 INFO] Running harness for resnet50 benchmark in Offline scenario...\n",
"concurrent_offline_expected_qps : 2181\n",
"dla_batch_size : 32\n",
"dla_copy_streams : 1 \n",
"dla_core : 0 \n",
"dla_inference_streams : 1\n",
"dla_offline_expected_qps : 396\n",
"gpu_batch_size : 64\n",
"gpu_copy_streams : 1 \n",
"gpu_inference_streams : 1\n",
"gpu_offline_expected_qps : 1478.33\n",
"input_dtype : int8 \n",
"input_format : linear\n",
"map_path : data_maps/imagenet/val_map.txt\n",
"precision : int8\n",
"tensor_path : ${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear\n",
"use_graphs : False\n",
"system_id : AGX_Xavier scenario : Offline\n",
"benchmark : resnet50 \n",
"config_name : AGX_Xavier_resnet50_Offline\n",
"accuracy_level : 99% \n",
"optimization_level : plugin-enabled\n",
"inference_server : lwis\n",
"config_ver : default \n",
"system_name : None\n",
"test_mode : AccuracyOnly\n",
"log_dir : /datasets/inference_results_v0.7/closed/NVIDIA/build/logs/2021.01.28-23.26.06\n",
"gpu_num_bundles : 2\n",
"[2021-01-28 23:26:07,182 __init__.py:207 INFO] Running command: ./build/bin/harness_default --logfile_outdir=\"/datasets/inference_results_v0.7/closed/NVIDIA/build/logs/2021.01.28-23.26.06/AGX_Xavier_TRT$\n",
"resnet50/Offline\" --logfile_prefix=\"mlperf_log_\" --performance_sample_count=2048 --test_mode=\"AccuracyOnly\" --gpu_copy_streams=1 --gpu_inference_streams=1 --dla_batch_size=32 --dla_copy_streams=1 --dla_$\n",
"nference_streams=1 --use_graphs=false --gpu_batch_size=64 --map_path=\"data_maps/imagenet/val_map.txt\" --tensor_path=\"${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear\" --gpu_engines=\"./build/engine$\n",
"/AGX_Xavier/resnet50/Offline/resnet50-Offline-gpu-b64-int8.default.plan\" --mlperf_conf_path=\"measurements/AGX_Xavier_TRT/resnet50/Offline/mlperf.conf\" --user_conf_path=\"measurements/AGX_Xavier_TRT/resne$\n",
"50/Offline/user.conf\" --dla_engines=\"./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-dla-b32-int8.default.plan\" --scenario Offline --model resnet50\n",
"[2021-01-28 23:26:07,183 __init__.py:213 INFO] Overriding Environment\n",
"&&&& RUNNING Default_Harness # ./build/bin/harness_default\n",
"[I] mlperf.conf path: measurements/AGX_Xavier_TRT/resnet50/Offline/mlperf.conf\n",
"[I] user.conf path: measurements/AGX_Xavier_TRT/resnet50/Offline/user.conf\n",
"[I] Device:0: ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-gpu-b64-int8.default.plan has been successfully loaded.\n",
"[I] Device:0.DLA-0: ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-dla-b32-int8.default.plan has been successfully loaded.\n",
"[I] Device:0.DLA-1: ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-dla-b32-int8.default.plan has been successfully loaded.\n",
"[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: false\n",
"Starting warmup. Running for a minimum of 5 seconds.\n",
"Finished warmup. Ran for 7.81217s.\n",
"Starting running actual test.\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 1 batches of size 16\n",
" 537 batches of size 64\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 538\n",
"Device Device:0.DLA-0 processed:\n",
" 244 batches of size 32\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 0\n",
"Device Device:0.DLA-1 processed:\n",
" 244 batches of size 32\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 0\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-28 23:27:03,219 main.py:341 INFO] Result: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"[2021-01-28 23:27:03,252 __init__.py:207 INFO] Running command: python3 build/inference/vision/classification_and_detection/tools/accuracy-imagenet.py --mlperf-accuracy-file /datasets/inference_results_v\n",
"0.7/closed/NVIDIA/build/logs/2021.01.28-23.26.06/AGX_Xavier_TRT/resnet50/Offline/mlperf_log_accuracy.json --imagenet-val-file data_maps/imagenet/val_map.txt --dtype int32\n",
"accuracy=76.040%, good=38020, total=50000\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: Accuracy = 76.040, Threshold = 75.695. Accuracy test PASSED.\n",
"\n",
"\n",
"real0m57.327s\n",
"user0m45.560s\n",
"sys 0m7.316s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## SingleStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlperf/inference_results_v0.7/blob/master/closed/NVIDIA/configs/resnet50/SingleStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v0.7/closed/NVIDIA/configs/resnet50/SingleStream/config.json -A 11\n",
" "AGX_Xavier": {\n",
" "gpu_batch_size": 1,\n",
" "gpu_copy_streams": 1,\n",
" "gpu_inference_streams": 1,\n",
" "gpu_single_stream_expected_latency_ns": 2273000,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/imagenet/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=resnet50 --scenarios=singlestream"\n",
"...\n",
"[2021-01-29 23:14:18,111 main.py:153 INFO] Finished building engines for resnet50 benchmark in SingleStream scenario.\n",
"Time taken to generate engines: 56.77854871749878 seconds\n",
"\n",
"real 0m58.002s\n",
"user 0m23.136s\n",
"sys 0m6.292s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ find . \\\n",
"-name resnet50-SingleStream*.plan -exec du -hs {} \\;\n",
"44M ./build/engines/AGX_Xavier/resnet50/SingleStream/resnet50-SingleStream-gpu-b1-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/SingleStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"90th percentile latency (ns) : 2144898\n",
"Result is : VALID\n",
"...\n",
"QPS w/ loadgen overhead : 470.07\n",
"QPS w/o loadgen overhead : 472.77\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=SingleStream --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary \n",
"================================================\n",
"SUT name : LWIS_Server \n",
"Scenario : Single Stream \n",
"Mode : Performance\n",
"90th percentile latency (ns) : 2078719\n",
"Result is : VALID \n",
" Min duration satisfied : Yes \n",
" Min queries satisfied : Yes \n",
"\n",
"================================================ \n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 485.07\n",
"QPS w/o loadgen overhead : 488.48\n",
"\n",
"Min latency (ns) : 1980890 \n",
"Max latency (ns) : 15165301\n",
"Mean latency (ns) : 2047146\n",
"50.00 percentile latency (ns) : 2038621\n",
"90.00 percentile latency (ns) : 2078719\n",
"95.00 percentile latency (ns) : 2098912\n",
"97.00 percentile latency (ns) : 2118465\n",
"99.00 percentile latency (ns) : 2187844\n",
"99.90 percentile latency (ns) : 2564821\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 439.947\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 2048\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 29106 batches of size 1\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 29106\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 23:25:15,883 main.py:341 INFO] Result: 90th percentile latency (ns) : 2078719 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: 90th percentile latency (ns) : 2078719 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 1m16.374s\n",
"user 1m6.484s\n",
"sys 0m5.004s\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/SingleStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
"accuracy=76.064%, good=38032, total=50000\n",
"hash=7458cd3f1154670a0d063c87b38d2eba7aa8c1921f2558a46333cfef8d9b4036\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=SingleStream --test_mode=AccuracyOnly"\n",
"...\n",
"accuracy=76.078%, good=38039, total=50000\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: Accuracy = 76.078, Threshold = 75.695. Accuracy test PASSED.\n",
"\n",
"\n",
"real 4m27.773s\n",
"user 2m9.392s\n",
"sys 0m11.332s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## MultiStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlperf/inference_results_v0.7/blob/master/closed/NVIDIA/configs/resnet50/MultiStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v0.7/closed/NVIDIA/configs/resnet50/MultiStream/config.json -A 17\n",
" "AGX_Xavier": {\n",
" "concurrent_multi_stream_samples_per_query": 96,\n",
" "dla_batch_size": 15,\n",
" "dla_copy_streams": 2,\n",
" "dla_core": 0,\n",
" "dla_inference_streams": 4,\n",
" "dla_multi_stream_samples_per_query": 15,\n",
" "gpu_batch_size": 66,\n",
" "gpu_copy_streams": 2,\n",
" "gpu_inference_streams": 4,\n",
" "gpu_multi_stream_samples_per_query": 66,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/imagenet/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=resnet50 --scenarios=multistream"\n",
"...\n",
"[2021-01-28 23:38:23,569 main.py:153 INFO] Finished building engines for resnet50 benchmark in MultiStream scenario.\n",
"Time taken to generate engines: 113.68001079559326 seconds\n",
"\n",
"real 1m54.783s\n",
"user 0m31.604s\n",
"sys 0m15.164s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ find . \\\n",
"-name resnet50-MultiStream*.plan -exec du -hs {} \\;\n",
"96M ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-gpu-b66-int8.default.plan\n",
"29M ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-dla-b15-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/MultiStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per query : 96\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=MultiStream --test_mode=PerformanceOnly"\n",
"[2021-01-28 23:40:13,526 main.py:692 INFO] Detected System ID: AGX_Xavier\n",
"[2021-01-28 23:40:13,527 main.py:546 INFO] Using config files: configs/resnet50/MultiStream/config.json\n",
"[2021-01-28 23:40:13,527 __init__.py:283 INFO] Parsing config file configs/resnet50/MultiStream/config.json ...\n",
"[2021-01-28 23:40:13,528 main.py:559 INFO] Processing config \"AGX_Xavier_resnet50_MultiStream\"\n",
"[2021-01-28 23:40:13,528 main.py:294 INFO] Running harness for resnet50 benchmark in MultiStream scenario...\n",
"concurrent_multi_stream_samples_per_query : 96\n",
"dla_batch_size : 15\n",
"dla_copy_streams : 2\n",
"dla_core : 0\n",
"dla_inference_streams : 4\n",
"dla_multi_stream_samples_per_query : 15\n",
"gpu_batch_size : 66 \n",
"gpu_copy_streams : 2\n",
"gpu_inference_streams : 4 \n",
"gpu_multi_stream_samples_per_query : 66\n",
"input_dtype : int8\n",
"input_format : linear\n",
"map_path : data_maps/imagenet/val_map.txt\n",
"precision : int8 \n",
"tensor_path : ${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear\n",
"use_graphs : False\n",
"system_id : AGX_Xavier\n",
"scenario : MultiStream \n",
"benchmark : resnet50\n",
"config_name : AGX_Xavier_resnet50_MultiStream \n",
"accuracy_level : 99%\n",
"optimization_level : plugin-enabled \n",
"inference_server : lwis \n",
"config_ver : default\n",
"system_name : None\n",
"test_mode : PerformanceOnly \n",
"log_dir : /datasets/inference_results_v0.7/closed/NVIDIA/build/logs/2021.01.28-23.40.13\n",
"gpu_num_bundles : 2\n",
"[2021-01-28 23:40:13,542 __init__.py:207 INFO] Running command: ./build/bin/harness_default --logfile_outdir=\"/datasets/inference_results_v0.7/closed/NVIDIA/build/logs/2021.01.28-23.40.13/AGX_Xavier_TRT/\n",
"resnet50/MultiStream\" --logfile_prefix=\"mlperf_log_\" --performance_sample_count=2048 --test_mode=\"PerformanceOnly\" --gpu_copy_streams=2 --gpu_inference_streams=4 --dla_batch_size=15 --dla_copy_streams=2\n",
"--dla_inference_streams=4 --use_graphs=false --gpu_batch_size=66 --map_path=\"data_maps/imagenet/val_map.txt\" --tensor_path=\"${PREPROCESSED_DATA_DIR}/imagenet/ResNet50/int8_linear\" --gpu_engines=\"./build/\n",
"engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-gpu-b66-int8.default.plan\" --mlperf_conf_path=\"measurements/AGX_Xavier_TRT/resnet50/MultiStream/mlperf.conf\" --user_conf_path=\"measurements/AG\n",
"X_Xavier_TRT/resnet50/MultiStream/user.conf\" --dla_engines=\"./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-dla-b15-int8.default.plan\" --scenario MultiStream --model resnet50\n",
"[2021-01-28 23:40:13,542 __init__.py:213 INFO] Overriding Environment\n",
"&&&& RUNNING Default_Harness # ./build/bin/harness_default \n",
"[I] mlperf.conf path: measurements/AGX_Xavier_TRT/resnet50/MultiStream/mlperf.conf\n",
"[I] user.conf path: measurements/AGX_Xavier_TRT/resnet50/MultiStream/user.conf\n",
"[I] Device:0: ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-gpu-b66-int8.default.plan has been successfully loaded.\n",
"[I] Device:0.DLA-0: ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-dla-b15-int8.default.plan has been successfully loaded.\n",
"[I] Device:0.DLA-1: ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-dla-b15-int8.default.plan has been successfully loaded.\n",
"[E] [TRT] Profile 0 has been chosen by another IExecutionContext. Use another profileIndex or destroy the IExecutionContext that use this profile.\n",
"[W] [TRT] Could not set default profile 0 for execution context. Profile index must be set explicitly.\n",
"[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: false\n",
"Starting warmup. Running for a minimum of 5 seconds.\n",
"Finished warmup. Ran for 6.81739s.\n",
"Starting running actual test.\n",
"\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server\n",
"Scenario : Multi Stream\n",
"Mode : Performance \n",
"Samples per query : 96\n",
"Result is : VALID \n",
"Performance constraints satisfied : Yes\n",
"Min duration satisfied : Yes\n",
"Min queries satisfied : Yes \n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"Intervals between each IssueQuery:\"qps\" : 20, \"ms\" : 50\n",
"50.00 percentile : 1\n",
"90.00 percentile : 1 \n",
"95.00 percentile : 1\n",
"97.00 percentile : 1\n",
"99.00 percentile : 1\n",
"99.90 percentile : 1 \n",
" \n",
"Per-query latency:\"target_ns\" : 50000000, \"target_ms\" : 50\n",
"50.00 percentile latency (ns) : 48075642\n",
"90.00 percentile latency (ns) : 48764781\n",
"95.00 percentile latency (ns) : 48964945\n",
"97.00 percentile latency (ns) : 49093855\n",
"99.00 percentile latency (ns) : 49334248\n",
"99.90 percentile latency (ns) : 49785859\n",
"\n",
"Per-sample latency:\n",
"Min latency (ns): 45713517\n",
"Max latency (ns): 62771126\n",
"Mean latency (ns) : 47592969\n",
"50.00 percentile latency (ns) : 47553785 \n",
"90.00 percentile latency (ns) : 48336265 \n",
"95.00 percentile latency (ns) : 48552497\n",
"97.00 percentile latency (ns) : 48713470 \n",
"99.00 percentile latency (ns) : 48975182\n",
"99.90 percentile latency (ns) : 49500776\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================ \n",
"samples_per_query : 96\n",
"target_qps : 20\n",
"target_latency (ns): 50000000 \n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 270336\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903 \n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 2048\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Equivalent QPS computed by samples_per_query*target_qps : 1920\n",
"Device Device:0 processed:\n",
"1 batches of size 30\n",
"270336 batches of size 66\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 270337\n",
"Device Device:0.DLA-0 processed:\n",
"270335 batches of size 15\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 0\n",
"Device Device:0.DLA-1 processed:\n",
"270335 batches of size 15\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 0\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 03:25:56,386 main.py:341 INFO] Result: Samples per query : 96 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
"resnet50: Samples per query : 96 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
"resnet50: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real225m43.401s\n",
"user8m35.936s\n",
"sys 3m57.940s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/MultiStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
"accuracy=76.044%, good=38022, total=50000\n",
"hash=3183baa0e60b647fc54e5469b23fc91353bf8bccc807dd249766cd3ed04d0593\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=MultiStream --test_mode=AccuracyOnly"\n",
"...\n",
"accuracy=76.046%, good=38023, total=50000\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" resnet50: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" resnet50: Accuracy = 76.046, Threshold = 75.695. Accuracy test PASSED.\n",
"\n",
"\n",
"real 0m59.871s\n",
"user 0m24.236s\n",
"sys 0m6.596s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# SSD-ResNet34"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlperf/inference_results_v0.7/blob/master/closed/NVIDIA/configs/ssd-resnet34/Offline/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v0.7/closed/NVIDIA/configs/ssd-resnet34/Offline/config.json -A 18\n",
" "AGX_Xavier": {\n",
" "concurrent_offline_expected_qps": 50,\n",
" "dla_batch_size": 1,\n",
" "dla_copy_streams": 1,\n",
" "dla_core": 0,\n",
" "dla_inference_streams": 1,\n",
" "dla_offline_expected_qps": 10,\n",
" "gpu_batch_size": 2,\n",
" "gpu_copy_streams": 4,\n",
" "gpu_inference_streams": 1,\n",
" "gpu_offline_expected_qps": 35.1243,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/coco/val_map.txt",\n",
" "min_query_count": 24576,\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/coco/val2017/SSDResNet34/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=offline"\n",
"...\n",
"[2021-01-29 17:52:19,860 main.py:153 INFO] Finished building engines for ssd-resnet34 benchmark in Offline scenario.\n",
"Time taken to generate engines: 464.4344849586487 seconds\n",
"\n",
"real 7m46.577s\n",
"user 2m12.112s\n",
"sys 0m15.832s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ find . \\\n",
"-name ssd-resnet34-Offline*.plan -exec du -hs {} \\;\n",
"148M ./build/engines/AGX_Xavier/ssd-resnet34/Offline/ssd-resnet34-Offline-gpu-b2-int8.default.plan\n",
"23M ./build/engines/AGX_Xavier/ssd-resnet34/Offline/ssd-resnet34-Offline-dla-b1-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/Offline/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per second: 50.9757\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=Offline --test_mode=PerformanceOnly"\n",
"...\n",
"================================================ \n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server\n",
"Scenario : Offline \n",
"Mode : Performance\n",
"Samples per second: 50.8099\n",
"Result is : VALID\n",
"Min duration satisfied : Yes\n",
"Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats \n",
"================================================\n",
"Min latency (ns): 58655035\n",
"Max latency (ns): 483684969003 \n",
"Mean latency (ns) : 241873131399\n",
"50.00 percentile latency (ns) : 241862755973\n",
"90.00 percentile latency (ns) : 435313276430 \n",
"95.00 percentile latency (ns) : 459447810260\n",
"97.00 percentile latency (ns) : 469144908922\n",
"99.00 percentile latency (ns) : 478833559170 \n",
"99.90 percentile latency (ns) : 483161611392\n",
"\n",
"================================================ \n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 24576 \n",
"target_qps : 50\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1 \n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 64\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
"8193 batches of size 2\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 8193\n",
"Device Device:0.DLA-0 processed:\n",
"4095 batches of size 1\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 0\n",
"Device Device:0.DLA-1 processed:\n",
"4095 batches of size 1\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 0\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 22:55:27,391 main.py:341 INFO] Result: Samples per second: 50.8099 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
"ssd-resnet34: Samples per second: 50.8099 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
"ssd-resnet34: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real8m15.281s\n",
"user8m8.460s\n",
"sys 0m6.888s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/Offline/accuracy/accuracy.txt)\n",
"\n",
"\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.200\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.187\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.257\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.331\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.352\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.428\n",
"mAP=20.048%\n",
"hash=95b218c56f20ce3e77d5af209df4b8219bf86c6339aebf93ac3d954126f60fc9\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=Offline --test_mode=AccuracyOnly"\n",
"...\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.187\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.257\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.331\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.352\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.428\n",
"mAP=20.057%\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" ssd-resnet34: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" ssd-resnet34: Accuracy = 20.057, Threshold = 19.800. Accuracy test PASSED.\n",
"\n",
"\n",
"real 8m52.357s\n",
"user 5m58.536s\n",
"sys 0m15.148s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## SingleStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlperf/inference_results_v0.7/blob/master/closed/NVIDIA/configs/ssd-resnet34/SingleStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v0.7/closed/NVIDIA/configs/ssd-resnet34/SingleStream/config.json -A 11\n",
" "AGX_Xavier": {\n",
" "gpu_batch_size": 1,\n",
" "gpu_copy_streams": 1,\n",
" "gpu_inference_streams": 1,\n",
" "gpu_single_stream_expected_latency_ns": 29478000,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/coco/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/coco/val2017/SSDResNet34/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=singlestream"\n",
"...\n",
"[2021-01-29 17:45:05,185 main.py:153 INFO] Finished building engines for ssd-resnet34 benchmark in SingleStream scenario.\n",
"Time taken to generate engines: 63.35827445983887 seconds\n",
"\n",
"real 1m5.118s\n",
"user 0m15.400s\n",
"sys 0m5.076s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ find . \\\n",
"-name ssd-resnet34-SingleStream*.plan -exec du -hs {} \\;\n",
"37M ./build/engines/AGX_Xavier/ssd-resnet34/SingleStream/ssd-resnet34-SingleStream-gpu-b1-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/SingleStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"90th percentile latency (ns) : 28531845\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=SingleStream --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server \n",
"Scenario : Single Stream\n",
"Mode : Performance\n",
"90th percentile latency (ns) : 28554901\n",
"Result is : VALID \n",
" Min duration satisfied : Yes \n",
" Min queries satisfied : Yes \n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 35.17 \n",
"QPS w/o loadgen overhead : 35.23\n",
"\n",
"Min latency (ns) : 28109905\n",
"Max latency (ns) : 32556991\n",
"Mean latency (ns) : 28385289\n",
"50.00 percentile latency (ns) : 28361924\n",
"90.00 percentile latency (ns) : 28554901\n",
"95.00 percentile latency (ns) : 28625392\n",
"97.00 percentile latency (ns) : 28688816\n",
"99.00 percentile latency (ns) : 28884610\n",
"99.90 percentile latency (ns) : 29821107\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 33.9236\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 64\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 2111 batches of size 1\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 2111\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 22:39:22,362 main.py:341 INFO] Result: 90th percentile latency (ns) : 28554901 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: 90th percentile latency (ns) : 28554901 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 1m9.437s\n",
"user 1m4.720s\n",
"sys 0m3.964s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/SingleStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"hash=526aac286ebb67218a3528397b4aecbff9269cbe01307069569345d9c3fbb445\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=SingleStream --test_mode=AccuracyOnly"\n",
"...\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: Accuracy = 20.111, Threshold = 19.800. Accuracy test PASSED.\n",
"\n",
"\n",
"real 12m55.238s\n",
"user 8m38.988s\n",
"sys 0m16.408s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## MultiStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlperf/inference_results_v0.7/blob/master/closed/NVIDIA/configs/ssd-resnet34/MultiStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v0.7/closed/NVIDIA/configs/ssd-resnet34/MultiStream/config.json -A 9\n",
" "AGX_Xavier": {\n",
" "gpu_batch_size": 2,\n",
" "gpu_multi_stream_samples_per_query": 2,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/coco/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/coco/val2017/SSDResNet34/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=multistream"\n",
"...\n",
"[2021-01-29 07:33:48,592 main.py:153 INFO] Finished building engines for ssd-resnet34 benchmark in MultiStream scenario.\n",
"Time taken to generate engines: 263.8264467716217 seconds\n",
"\n",
"real 4m25.496s\n",
"user 0m32.000s\n",
"sys 0m35.784s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ find . \\\n",
"-name ssd-resnet34-MultiStream*.plan -exec du -hs {} \\;\n",
"148M ./build/engines/AGX_Xavier/ssd-resnet34/MultiStream/ssd-resnet34-MultiStream-gpu-b2-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/MultiStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per query : 2\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=MultiStream --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server \n",
"Scenario : Multi Stream \n",
"Mode : Performance \n",
"Samples per query : 2 \n",
"Result is : VALID \n",
" Performance constraints satisfied : Yes\n",
" Min duration satisfied : Yes \n",
" Min queries satisfied : Yes \n",
" \n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"Intervals between each IssueQuery: \"qps\" : 15, \"ms\" : 66.6667\n",
"50.00 percentile : 1 \n",
"90.00 percentile : 1 \n",
"95.00 percentile : 1 \n",
"97.00 percentile : 1 \n",
"99.00 percentile : 1 \n",
"99.90 percentile : 1\n",
" \n",
"Per-query latency: \"target_ns\" : 66666666, \"target_ms\" : 66.6667\n",
"50.00 percentile latency (ns) : 55624812 \n",
"90.00 percentile latency (ns) : 55913099 \n",
"95.00 percentile latency (ns) : 56006924\n",
"97.00 percentile latency (ns) : 56067621 \n",
"99.00 percentile latency (ns) : 56189283\n",
"99.90 percentile latency (ns) : 56401684\n",
" \n",
"Per-sample latency:\n",
"Min latency (ns) : 55180175\n",
"Max latency (ns) : 61310689 \n",
"Mean latency (ns) : 55646311\n",
"50.00 percentile latency (ns) : 55624812\n",
"90.00 percentile latency (ns) : 55913099 \n",
"95.00 percentile latency (ns) : 56006924\n",
"97.00 percentile latency (ns) : 56067621\n",
"99.00 percentile latency (ns) : 56189283\n",
"99.90 percentile latency (ns) : 56401684\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 2\n",
"target_qps : 15\n",
"target_latency (ns): 66666666\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 270336\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 64\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Equivalent QPS computed by samples_per_query*target_qps : 30\n",
"Device Device:0 processed:\n",
" 270336 batches of size 2\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 270336\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 13:12:12,503 main.py:341 INFO] Result: Samples per query : 2 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: Samples per query : 2 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 300m37.676s\n",
"user 5m11.760s\n",
"sys 0m27.856s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v0.7/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/MultiStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"hash=859242388c9a94513b189eb58a55bd11ad1d2f7d094880dfb72157a7ac5e45fd\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v0.7/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=MultiStream --test_mode=AccuracyOnly"\n",
"...\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: Accuracy = 20.111, Threshold = 19.800. Accuracy test PASSED.\n",
"\n",
"\n",
"real 11m44.285s\n",
"user 4m22.432s\n",
"sys 0m17.144s\n",
""
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}