{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# MLPerf Inference v1.0 - Reproducing Xavier results with JetPack 4.5.1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
"\n",
"1. [System](#system)\n",
"1. [Installation](#installation)\n",
" 1. [Clone the repo](#installation_repo)\n",
" 1. [Install the dependencies](#installation_deps) \n",
" 1. [Link the datasets](#installation_datasets)\n",
" 1. [Download the models](#installation_models)\n",
" 1. [Build the harness](#installation_harness)\n",
"1. [ResNet50](#resnet50)\n",
" 1. [Offline](#resnet50_offline)\n",
" 1. [Build](#resnet50_offline_build)\n",
" 1. [Performance](#resnet50_offline_performance)\n",
" 1. [Accuracy](#resnet50_offline_accuracy)\n",
" 1. [SingleStream](#resnet50_singlestream)\n",
" 1. [Build](#resnet50_singlestream_build)\n",
" 1. [Performance](#resnet50_singlestream_performance)\n",
" 1. [Accuracy](#resnet50_singlestream_accuracy)\n",
" 1. [MultiStream](#resnet50_multistream)\n",
" 1. [Build](#resnet50_multistream_build)\n",
" 1. [Performance](#resnet50_multistream_performance)\n",
" 1. [Accuracy](#resnet50_multistream_accuracy)\n",
"1. [SSD-ResNet34](#ssd-resnet34)\n",
" 1. [Offline](#ssd-resnet34_offline)\n",
" 1. [Build](#ssd-resnet34_offline_build)\n",
" 1. [Performance](#ssd-resnet34_offline_performance)\n",
" 1. [Accuracy](#ssd-resnet34_offline_accuracy)\n",
" 1. [SingleStream](#ssd-resnet34_singlestream)\n",
" 1. [Build](#ssd-resnet34_singlestream_build)\n",
" 1. [Performance](#ssd-resnet34_singlestream_performance)\n",
" 1. [Accuracy](#ssd-resnet34_singlestream_accuracy)\n",
" 1. [MultiStream](#ssd-resnet34_multistream)\n",
" 1. [Build](#ssd-resnet34_multistream_build)\n",
" 1. [Performance](#ssd-resnet34_multistream_performance)\n",
" 1. [Accuracy](#ssd-resnet34_multistream_accuracy)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# System: [NVIDIA AGX Xavier](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/systems/AGX_Xavier_TRT.json)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Stats: JetPack 4.5.1\n",
"\n",
"anton@xavier:~$ sudo -H python3 -m pip install jetson-stats -U\n",
"...\n",
"Successfully installed jetson-stats-3.1.0\n",
"\n",
"\n",
"\n",
"anton@xavier:~$ jetson_release \n",
" Current serial number in output stream: 19\n",
" - NVIDIA Jetson AGX Xavier [16GB]\n",
" * Jetpack 4.5.1 [L4T 32.5.1]\n",
" * NV Power Mode: MAXN - Type: 0\n",
" * jetson_stats.service: active\n",
" - Libraries:\n",
" * CUDA: 10.2.89\n",
" * cuDNN: 8.0.0.180\n",
" * TensorRT: 7.1.3.0\n",
" * Visionworks: 1.6.0.501\n",
" * OpenCV: 4.1.1 compiled CUDA: NO\n",
" * VPI: ii libnvvpi1 1.0.15 arm64 NVIDIA Vision Programming Interface library\n",
" * Vulkan: 1.2.70\n",
"\n",
"\n",
"## OS: Ubuntu 18.04.5 LTS\n",
"\n",
"\n",
"anton@xavier:~$ cat /etc/lsb-release\n",
"DISTRIB_ID=Ubuntu\n",
"DISTRIB_RELEASE=18.04\n",
"DISTRIB_CODENAME=bionic\n",
"DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"\n",
"\n",
"\n",
"## CPU: 8-core ARMv8 @ 2265 MHz\n",
"\n",
"\n",
"anton@xavier:~$ lscpu\n",
"Architecture: aarch64\n",
"Byte Order: Little Endian\n",
"CPU(s): 8\n",
"On-line CPU(s) list: 0-7\n",
"Thread(s) per core: 1\n",
"Core(s) per socket: 2\n",
"Socket(s): 4\n",
"Vendor ID: Nvidia\n",
"Model: 0\n",
"Model name: ARMv8 Processor rev 0 (v8l)\n",
"Stepping: 0x0\n",
"CPU max MHz: 2265.6001\n",
"CPU min MHz: 115.2000\n",
"BogoMIPS: 62.50\n",
"L1d cache: 64K\n",
"L1i cache: 128K\n",
"L2 cache: 2048K\n",
"L3 cache: 4096K\n",
"Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp\n",
"\n",
"\n",
"## GPU\n",
"\n",
"\n",
"anton@xavier:~$ sudo jetson_clocks\n",
"anton@xavier:~$ sudo jetson_clocks --show\n",
"SOC family:tegra194 Machine:Jetson-AGX\n",
"Online CPUs: 0-7\n",
"cpu0: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu1: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu2: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu3: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu4: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu5: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu6: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu7: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000\n",
"EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1\n",
"Fan: PWM=0\n",
"NV Power Mode: MAXN\n",
"\n",
"\n",
"\n",
"\n",
"anton@xavier:~$ ck compile program:tool-print-cuda-devices\n",
"anton@xavier:~$ ck run program:tool-print-cuda-devices\n",
"...\n",
"GPU Device ID: 0\n",
"GPU Name: Xavier\n",
"GPU compute capability: 7.2\n",
"CUDA driver version: 10.2\n",
"CUDA runtime version: 10.2\n",
"Global memory: 33479647232\n",
"Max clock rate: 1377.000000 MHz\n",
"Total amount of shared memory per block: 49152\n",
"Total number of registers available per block: 65536\n",
"Warp size: 32\n",
"Maximum number of threads per multiprocessor: 2048\n",
"Maximum number of threads per block: 1024\n",
"Max dimension size of a thread block X: 1024\n",
"Max dimension size of a thread block Y: 1024\n",
"Max dimension size of a thread block Z: 64\n",
"Max dimension size of a grid size X: 2147483647\n",
"Max dimension size of a grid size Y: 65535\n",
"Max dimension size of a grid size Z: 65535\n",
"\n",
"\n",
"\n",
"## Disks\n",
"\n",
"anton@xavier:~$ df -h\n",
"Filesystem Size Used Avail Use% Mounted on\n",
"/dev/mmcblk0p1 28G 17G 9.4G 64% /\n",
"none 16G 0 16G 0% /dev\n",
"tmpfs 16G 56K 16G 1% /dev/shm\n",
"tmpfs 16G 42M 16G 1% /run\n",
"tmpfs 5.0M 4.0K 5.0M 1% /run/lock\n",
"tmpfs 16G 0 16G 0% /sys/fs/cgroup\n",
"/dev/mmcblk1p1 361G 314G 28G 92% /sd\n",
"tmpfs 3.2G 12K 3.2G 1% /run/user/120\n",
"tmpfs 3.2G 0 3.2G 0% /run/user/1000\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Clone the v1.0 results repo\n",
"\n",
"anton@xavier:~$ git clone https://github.com/mlcommons/inference_results_v1.0.git
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the dependencies\n",
"\n",
"To [quote](https://github.com/mlcommons/inference_results_v1.0/tree/master/closed/NVIDIA#prerequisites) NVIDIA:\n",
"> Note that this might take a while, on the order of several hours.\n",
"\n",
"anton@xavier:~$ chmod +x inference_results_v1.0/closed/NVIDIA/scripts/install_xavier_dependencies.sh\n",
"anton@xavier:~$ time sudo inference_results_v1.0/closed/NVIDIA/scripts/install_xavier_dependencies.sh\n",
"anton@xavier:~$ python3 -m pip install onnx\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Link the datasets\n",
"\n",
"We reused the datasets we generated while reproducing the v0.5 results.\n",
"\n",
"\n",
"anton@xavier:~$ export MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path\n",
"anton@xavier:~$ mkdir $MLPERF_SCRATCH_PATH\n",
"anton@xavier:~$ cd $MLPERF_SCRATCH_PATH\n",
"anton@xavier:/datasets/mlperf_scratch_path$ ln -s /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/preprocessed_data preprocessed_data\n",
"anton@xavier:/datasets/mlperf_scratch_path$ ln -s /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/data data\n",
"anton@xavier:/datasets/mlperf_scratch_path$ ls -la /datasets/mlperf_scratch_path/\n",
"total 20\n",
"drwxrwsr-x 3 anton dvdt 4096 Jan 28 11:10 .\n",
"drwxrwsr-x 17 root dvdt 4096 Jan 28 09:53 ..\n",
"lrwxrwxrwx 1 anton dvdt 64 Jan 28 11:10 data -> /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/data\n",
"drwxrwsr-x 5 anton dvdt 4096 Jan 28 09:54 models\n",
"lrwxrwxrwx 1 anton dvdt 77 Jan 28 11:10 preprocessed_data -> /datasets/inference_results_v0.5-nvidia/closed/NVIDIA/build/preprocessed_data\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Download the models\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ git diff Makefile\n",
"diff --git a/closed/NVIDIA/Makefile b/closed/NVIDIA/Makefile\n",
"index e7647663..2b9356f6 100644\n",
"--- a/closed/NVIDIA/Makefile\n",
"+++ b/closed/NVIDIA/Makefile\n",
"@@ -360,7 +360,7 @@ endif\n",
" \n",
" ############################## DOWNLOAD_MODEL ##############################\n",
" \n",
"-BENCHMARKS = resnet50 ssd-resnet34 ssd-mobilenet bert dlrm rnnt 3d-unet\n",
"+BENCHMARKS = resnet50 ssd-resnet34 ssd-mobilenet # bert dlrm rnnt 3d-unet\n",
" \n",
" .PHONY: download_model\n",
" download_model: link_dirs\n",
"\n",
"@@ -474,7 +474,6 @@ ifeq ($(ARCH), x86_64)\n",
" cd build/plugins/DLRMBottomMLPPlugin \\\n",
" && cmake -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) $(PROJECT_ROOT)/code/plugin/DLRMBottomMLPPlugin \\\n",
" && make -j\n",
"-endif\n",
" mkdir -p build/plugins/RNNTOptPlugin\n",
" cd build/plugins/RNNTOptPlugin \\\n",
" && cmake -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) $(PROJECT_ROOT)/code/plugin/RNNTOptPlugin \\\n",
"@@ -491,6 +490,7 @@ endif\n",
" cd build/plugins/conv3D1X1X1K4Plugin \\\n",
" && cmake -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) $(PROJECT_ROOT)/code/plugin/conv3D1X1X1K4Plugin \\\n",
" && make -j\n",
"+endif\n",
" \n",
" # Build LoadGen.\n",
" .PHONY: build_loadgen\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path make download_model\n",
"...\n",
"Finished downloading all the models!\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ make build\n",
"...\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ ls -la build/\n",
"total 28\n",
"drwxrwsr-x 7 anton dvdt 4096 Apr 27 11:48 .\n",
"drwxrwsr-x 14 anton dvdt 4096 Apr 27 11:29 ..\n",
"drwxrwsr-x 2 anton dvdt 4096 Apr 27 11:49 bin\n",
"lrwxrwxrwx 1 anton dvdt 34 Apr 27 11:48 data -> /datasets/mlperf_scratch_path/data\n",
"drwxrwsr-x 5 anton dvdt 4096 Apr 27 11:49 harness\n",
"drwxrwsr-x 16 anton dvdt 4096 Apr 27 11:24 inference\n",
"lrwxrwxrwx 1 anton dvdt 36 Apr 27 11:48 models -> /datasets/mlperf_scratch_path/models\n",
"drwxrwsr-x 5 anton dvdt 4096 Apr 27 11:27 plugins\n",
"drwxrwsr-x 7 anton dvdt 4096 Apr 27 11:24 power-dev\n",
"lrwxrwxrwx 1 anton dvdt 47 Apr 27 11:48 preprocessed_data -> /datasets/mlperf_scratch_path/preprocessed_data\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# ResNet50"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/configs/resnet50/Offline/config.json)\n",
"\n",
"anton@xavier:~$ grep AGX_Xavier /datasets/inference_results_v1.0/closed/NVIDIA/configs/resnet50/Offline/config.json -A 15\n",
" "AGX_Xavier": {\n",
" "concurrent_offline_expected_qps": 2181,\n",
" "config_ver": {\n",
" "maxq": {\n",
" "concurrent_offline_expected_qps": 1530\n",
" }\n",
" },\n",
" "dla_batch_size": 32,\n",
" "dla_copy_streams": 1,\n",
" "dla_core": 0,\n",
" "dla_inference_streams": 1,\n",
" "dla_offline_expected_qps": 396,\n",
" "gpu_batch_size": 64,\n",
" "gpu_copy_streams": 1,\n",
" "gpu_offline_expected_qps": 1478.33\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=resnet50 --scenarios=offline"\n",
"...\n",
"[2021-04-27 17:07:37,429 main.py:112 INFO] Finished building engines for resnet50 benchmark in Offline scenario.\n",
"Time taken to generate engines: 65.42682266235352 seconds\n",
"\n",
"real 1m6.766s\n",
"user 0m26.980s\n",
"sys 0m6.420s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ find . \\\n",
"-name resnet50-Offline*.plan -exec du -hs {} \\;\n",
"48M ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-gpu-b64-int8.default.plan\n",
"33M ./build/engines/AGX_Xavier/resnet50/Offline/resnet50-Offline-dla-b32-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/Offline/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per second: 2071.51\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment (~53 W)\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=offline --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server\n",
"Scenario : Offline \n",
"Mode : PerformanceOnly \n",
"Samples per second: 2086.92 \n",
"Result is : VALID \n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes \n",
" \n",
"================================================\n",
"Additional Stats \n",
"================================================\n",
"Min latency (ns) : 155755993\n",
"Max latency (ns) : 689754096859\n",
"Mean latency (ns) : 345039571231 \n",
"50.00 percentile latency (ns) : 345089223369 \n",
"90.00 percentile latency (ns) : 620810076791\n",
"95.00 percentile latency (ns) : 655270079960 \n",
"97.00 percentile latency (ns) : 669065261726\n",
"99.00 percentile latency (ns) : 682850845028\n",
"99.90 percentile latency (ns) : 689068993553 \n",
"\n",
"================================================\n",
"Test Parameters Used \n",
"================================================\n",
"samples_per_query : 1439460 \n",
"target_qps : 2181 \n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 600000\n",
"max_duration (ms): 0\n",
"min_query_count : 1\n",
"max_query_count : 0 \n",
"qsl_rng_seed : 7322528924094909334 \n",
"sample_index_rng_seed : 1570999273408051088 \n",
"schedule_rng_seed : 3507442325620259414 \n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : 0\n",
"performance_issue_unique : 0\n",
"performance_issue_same : 0\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 2048\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 1 batches of size 36\n",
" 15545 batches of size 64\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 10368\n",
" BatchedCudaMemcpy Calls: 15384\n",
"Device Device:0.DLA-0 processed:\n",
" 6938 batches of size 32\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 0\n",
"Device Device:0.DLA-1 processed:\n",
" 6954 batches of size 32\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 0\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-04-27 17:22:17,724 main.py:280 INFO] Result: result_samples_per_second: 2086.92, Result is VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: result_samples_per_second: 2086.92, Result is VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 11m52.559s\n",
"user 11m30.528s\n",
"sys 0m12.236s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/Offline/accuracy/accuracy.txt)\n",
"\n",
"\n",
"accuracy=76.074%, good=38037, total=50000\n",
"hash=3dc4add63a23f3f1f23e44abbf6b5e8f9c8a12e44412e4aa6a77f09cac22eee2\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=offline --test_mode=AccuracyOnly"\n",
"...\n",
"accuracy=76.040%, good=38020, total=50000\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" resnet50: Accuracy = 76.040, Threshold = 75.695. Accuracy test PASSED.\n",
"\n",
"\n",
"real 3m29.147s\n",
"user 0m58.836s\n",
"sys 0m17.444s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## SingleStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/configs/resnet50/SingleStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v1.0/closed/NVIDIA/configs/resnet50/SingleStream/config.json -A 6\n",
" "AGX_Xavier": {\n",
" "config_ver": {\n",
" "maxq": {}\n",
" },\n",
" "gpu_single_stream_expected_latency_ns": 2273000,\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=resnet50 --scenarios=singlestream"\n",
"...\n",
"[2021-04-27 17:35:34,949 main.py:112 INFO] Finished building engines for resnet50 benchmark in SingleStream scenario.\n",
"Time taken to generate engines: 42.95994567871094 seconds\n",
"\n",
"real 0m44.209s\n",
"user 0m23.192s\n",
"sys 0m4.620s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ find . \\\n",
"-name resnet50-SingleStream*.plan -exec du -hs {} \\;\n",
"44M ./build/engines/AGX_Xavier/resnet50/SingleStream/resnet50-SingleStream-gpu-b1-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/SingleStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"90th percentile latency (ns) : 1980582\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"...\n",
"QPS w/ loadgen overhead : 509.00\n",
"QPS w/o loadgen overhead : 512.43\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=SingleStream --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary \n",
"================================================\n",
"SUT name : LWIS_Server \n",
"Scenario : Single Stream \n",
"Mode : Performance\n",
"90th percentile latency (ns) : 2078719\n",
"Result is : VALID \n",
" Min duration satisfied : Yes \n",
" Min queries satisfied : Yes \n",
"\n",
"================================================ \n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 485.07\n",
"QPS w/o loadgen overhead : 488.48\n",
"\n",
"Min latency (ns) : 1980890 \n",
"Max latency (ns) : 15165301\n",
"Mean latency (ns) : 2047146\n",
"50.00 percentile latency (ns) : 2038621\n",
"90.00 percentile latency (ns) : 2078719\n",
"95.00 percentile latency (ns) : 2098912\n",
"97.00 percentile latency (ns) : 2118465\n",
"99.00 percentile latency (ns) : 2187844\n",
"99.90 percentile latency (ns) : 2564821\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 439.947\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 2048\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 29106 batches of size 1\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 29106\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 23:25:15,883 main.py:341 INFO] Result: 90th percentile latency (ns) : 2078719 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: 90th percentile latency (ns) : 2078719 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 1m16.374s\n",
"user 1m6.484s\n",
"sys 0m5.004s\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/SingleStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
"accuracy=76.064%, good=38032, total=50000\n",
"hash=7458cd3f1154670a0d063c87b38d2eba7aa8c1921f2558a46333cfef8d9b4036\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=SingleStream --test_mode=AccuracyOnly"\n",
"...\n",
"accuracy=76.078%, good=38039, total=50000\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" resnet50: Accuracy = 76.078, Threshold = 75.695. Accuracy test PASSED.\n",
"\n",
"\n",
"real 4m27.773s\n",
"user 2m9.392s\n",
"sys 0m11.332s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## MultiStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/configs/resnet50/MultiStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v1.0/closed/NVIDIA/configs/resnet50/MultiStream/config.json -A 14\n",
" "AGX_Xavier": {\n",
" "concurrent_multi_stream_samples_per_query": 96,\n",
" "config_ver": {\n",
" "maxq": {}\n",
" },\n",
" "dla_batch_size": 15,\n",
" "dla_copy_streams": 2,\n",
" "dla_core": 0,\n",
" "dla_inference_streams": 4,\n",
" "dla_multi_stream_samples_per_query": 15,\n",
" "gpu_batch_size": 66,\n",
" "gpu_copy_streams": 2,\n",
" "gpu_inference_streams": 4,\n",
" "gpu_multi_stream_samples_per_query": 66\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=resnet50 --scenarios=multistream"\n",
"...\n",
"[2021-04-27 20:34:39,925 main.py:112 INFO] Finished building engines for resnet50 benchmark in MultiStream scenario.\n",
"Time taken to generate engines: 118.26688885688782 seconds\n",
"\n",
"real 1m59.516s\n",
"user 0m35.180s\n",
"sys 0m15.528s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ find . \\\n",
"-name resnet50-MultiStream*.plan -exec du -hs {} \\;\n",
"96M ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-gpu-b66-int8.default.plan\n",
"29M ./build/engines/AGX_Xavier/resnet50/MultiStream/resnet50-MultiStream-dla-b15-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/MultiStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per query : 96\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=MultiStream --test_mode=PerformanceOnly"\n",
"[2021-04-27 20:38:49,061 main.py:701 INFO] Detected System ID: AGX_Xavier \n",
"[2021-04-27 20:38:49,062 main.py:529 INFO] Using config files: configs/resnet50/MultiStream/config.json\n",
"[2021-04-27 20:38:49,063 __init__.py:341 INFO] Parsing config file configs/resnet50/MultiStream/config.json ...\n",
"[2021-04-27 20:38:49,063 main.py:542 INFO] Processing config \"AGX_Xavier_resnet50_MultiStream\"\n",
"[2021-04-27 20:38:49,064 main.py:224 INFO] Running harness for resnet50 benchmark in MultiStream scenario...\n",
"...\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/resnet50/MultiStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
"accuracy=76.008%, good=38004, total=50000\n",
"hash=12988c6c56d7af58b5e816d9b9d64b20f5ae80497e355dc51653ed2433b583bb\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=resnet50 --scenarios=MultiStream --test_mode=AccuracyOnly"\n",
"...\n",
"accuracy=76.020%, good=38010, total=50000\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" resnet50: requested_multi_stream_samples_per_query: 96, Result validity unknown\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" resnet50: Accuracy = 76.020, Threshold = 75.695. Accuracy test PASSED.\n",
"\n",
"\n",
"real 1m2.343s\n",
"user 0m24.100s\n",
"sys 0m7.468s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# SSD-ResNet34"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/configs/ssd-resnet34/Offline/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v1.0/closed/NVIDIA/configs/ssd-resnet34/Offline/config.json -A 18\n",
" "AGX_Xavier": {\n",
" "concurrent_offline_expected_qps": 50,\n",
" "dla_batch_size": 1,\n",
" "dla_copy_streams": 1,\n",
" "dla_core": 0,\n",
" "dla_inference_streams": 1,\n",
" "dla_offline_expected_qps": 10,\n",
" "gpu_batch_size": 2,\n",
" "gpu_copy_streams": 4,\n",
" "gpu_inference_streams": 1,\n",
" "gpu_offline_expected_qps": 35.1243,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/coco/val_map.txt",\n",
" "min_query_count": 24576,\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/coco/val2017/SSDResNet34/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=offline"\n",
"...\n",
"[2021-01-29 17:52:19,860 main.py:153 INFO] Finished building engines for ssd-resnet34 benchmark in Offline scenario.\n",
"Time taken to generate engines: 464.4344849586487 seconds\n",
"\n",
"real 7m46.577s\n",
"user 2m12.112s\n",
"sys 0m15.832s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ find . \\\n",
"-name ssd-resnet34-Offline*.plan -exec du -hs {} \\;\n",
"148M ./build/engines/AGX_Xavier/ssd-resnet34/Offline/ssd-resnet34-Offline-gpu-b2-int8.default.plan\n",
"23M ./build/engines/AGX_Xavier/ssd-resnet34/Offline/ssd-resnet34-Offline-dla-b1-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/Offline/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per second: 50.9757\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=Offline --test_mode=PerformanceOnly"\n",
"...\n",
"================================================ \n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server\n",
"Scenario : Offline \n",
"Mode : Performance\n",
"Samples per second: 50.8099\n",
"Result is : VALID\n",
"Min duration satisfied : Yes\n",
"Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats \n",
"================================================\n",
"Min latency (ns): 58655035\n",
"Max latency (ns): 483684969003 \n",
"Mean latency (ns) : 241873131399\n",
"50.00 percentile latency (ns) : 241862755973\n",
"90.00 percentile latency (ns) : 435313276430 \n",
"95.00 percentile latency (ns) : 459447810260\n",
"97.00 percentile latency (ns) : 469144908922\n",
"99.00 percentile latency (ns) : 478833559170 \n",
"99.90 percentile latency (ns) : 483161611392\n",
"\n",
"================================================ \n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 24576 \n",
"target_qps : 50\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1 \n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 64\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
"8193 batches of size 2\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 8193\n",
"Device Device:0.DLA-0 processed:\n",
"4095 batches of size 1\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 0\n",
"Device Device:0.DLA-1 processed:\n",
"4095 batches of size 1\n",
"Memcpy Calls: 0\n",
"PerSampleCudaMemcpy Calls: 0\n",
"BatchedCudaMemcpy Calls: 0\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 22:55:27,391 main.py:341 INFO] Result: Samples per second: 50.8099 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
"ssd-resnet34: Samples per second: 50.8099 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
"ssd-resnet34: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real8m15.281s\n",
"user8m8.460s\n",
"sys 0m6.888s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/Offline/accuracy/accuracy.txt)\n",
"\n",
"\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.200\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.187\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.257\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.331\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.352\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.428\n",
"mAP=20.048%\n",
"hash=95b218c56f20ce3e77d5af209df4b8219bf86c6339aebf93ac3d954126f60fc9\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=Offline --test_mode=AccuracyOnly"\n",
"...\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.187\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.257\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.331\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.352\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.428\n",
"mAP=20.057%\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" ssd-resnet34: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-Offline:\n",
" ssd-resnet34: Accuracy = 20.057, Threshold = 19.800. Accuracy test PASSED.\n",
"\n",
"\n",
"real 8m52.357s\n",
"user 5m58.536s\n",
"sys 0m15.148s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## SingleStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/configs/ssd-resnet34/SingleStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v1.0/closed/NVIDIA/configs/ssd-resnet34/SingleStream/config.json -A 11\n",
" "AGX_Xavier": {\n",
" "gpu_batch_size": 1,\n",
" "gpu_copy_streams": 1,\n",
" "gpu_inference_streams": 1,\n",
" "gpu_single_stream_expected_latency_ns": 29478000,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/coco/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/coco/val2017/SSDResNet34/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=singlestream"\n",
"...\n",
"[2021-01-29 17:45:05,185 main.py:153 INFO] Finished building engines for ssd-resnet34 benchmark in SingleStream scenario.\n",
"Time taken to generate engines: 63.35827445983887 seconds\n",
"\n",
"real 1m5.118s\n",
"user 0m15.400s\n",
"sys 0m5.076s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ find . \\\n",
"-name ssd-resnet34-SingleStream*.plan -exec du -hs {} \\;\n",
"37M ./build/engines/AGX_Xavier/ssd-resnet34/SingleStream/ssd-resnet34-SingleStream-gpu-b1-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/SingleStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"90th percentile latency (ns) : 28531845\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=SingleStream --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server \n",
"Scenario : Single Stream\n",
"Mode : Performance\n",
"90th percentile latency (ns) : 28554901\n",
"Result is : VALID \n",
" Min duration satisfied : Yes \n",
" Min queries satisfied : Yes \n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 35.17 \n",
"QPS w/o loadgen overhead : 35.23\n",
"\n",
"Min latency (ns) : 28109905\n",
"Max latency (ns) : 32556991\n",
"Mean latency (ns) : 28385289\n",
"50.00 percentile latency (ns) : 28361924\n",
"90.00 percentile latency (ns) : 28554901\n",
"95.00 percentile latency (ns) : 28625392\n",
"97.00 percentile latency (ns) : 28688816\n",
"99.00 percentile latency (ns) : 28884610\n",
"99.90 percentile latency (ns) : 29821107\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 33.9236\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 64\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Device Device:0 processed:\n",
" 2111 batches of size 1\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 2111\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 22:39:22,362 main.py:341 INFO] Result: 90th percentile latency (ns) : 28554901 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: 90th percentile latency (ns) : 28554901 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 1m9.437s\n",
"user 1m4.720s\n",
"sys 0m3.964s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/SingleStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"hash=526aac286ebb67218a3528397b4aecbff9269cbe01307069569345d9c3fbb445\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=SingleStream --test_mode=AccuracyOnly"\n",
"...\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-SingleStream:\n",
" ssd-resnet34: Accuracy = 20.111, Threshold = 19.800. Accuracy test PASSED.\n",
"\n",
"\n",
"real 12m55.238s\n",
"user 8m38.988s\n",
"sys 0m16.408s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## MultiStream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Config](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/configs/ssd-resnet34/MultiStream/config.json)\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ grep AGX_Xavier /datasets/inference_results_v1.0/closed/NVIDIA/configs/ssd-resnet34/MultiStream/config.json -A 9\n",
" "AGX_Xavier": {\n",
" "gpu_batch_size": 2,\n",
" "gpu_multi_stream_samples_per_query": 2,\n",
" "input_dtype": "int8",\n",
" "input_format": "linear",\n",
" "map_path": "data_maps/coco/val_map.txt",\n",
" "precision": "int8",\n",
" "tensor_path": "${PREPROCESSED_DATA_DIR}/coco/val2017/SSDResNet34/int8_linear",\n",
" "use_graphs": false\n",
" },\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Build\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make generate_engines RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=multistream"\n",
"...\n",
"[2021-01-29 07:33:48,592 main.py:153 INFO] Finished building engines for ssd-resnet34 benchmark in MultiStream scenario.\n",
"Time taken to generate engines: 263.8264467716217 seconds\n",
"\n",
"real 4m25.496s\n",
"user 0m32.000s\n",
"sys 0m35.784s\n",
"\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ find . \\\n",
"-name ssd-resnet34-MultiStream*.plan -exec du -hs {} \\;\n",
"148M ./build/engines/AGX_Xavier/ssd-resnet34/MultiStream/ssd-resnet34-MultiStream-gpu-b2-int8.default.plan\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Performance\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/MultiStream/performance/run_1/mlperf_log_summary.txt#L7)\n",
"\n",
"\n",
"Samples per query : 2\n",
"Result is : VALID\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=MultiStream --test_mode=PerformanceOnly"\n",
"...\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : LWIS_Server \n",
"Scenario : Multi Stream \n",
"Mode : Performance \n",
"Samples per query : 2 \n",
"Result is : VALID \n",
" Performance constraints satisfied : Yes\n",
" Min duration satisfied : Yes \n",
" Min queries satisfied : Yes \n",
" \n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"Intervals between each IssueQuery: \"qps\" : 15, \"ms\" : 66.6667\n",
"50.00 percentile : 1 \n",
"90.00 percentile : 1 \n",
"95.00 percentile : 1 \n",
"97.00 percentile : 1 \n",
"99.00 percentile : 1 \n",
"99.90 percentile : 1\n",
" \n",
"Per-query latency: \"target_ns\" : 66666666, \"target_ms\" : 66.6667\n",
"50.00 percentile latency (ns) : 55624812 \n",
"90.00 percentile latency (ns) : 55913099 \n",
"95.00 percentile latency (ns) : 56006924\n",
"97.00 percentile latency (ns) : 56067621 \n",
"99.00 percentile latency (ns) : 56189283\n",
"99.90 percentile latency (ns) : 56401684\n",
" \n",
"Per-sample latency:\n",
"Min latency (ns) : 55180175\n",
"Max latency (ns) : 61310689 \n",
"Mean latency (ns) : 55646311\n",
"50.00 percentile latency (ns) : 55624812\n",
"90.00 percentile latency (ns) : 55913099 \n",
"95.00 percentile latency (ns) : 56006924\n",
"97.00 percentile latency (ns) : 56067621\n",
"99.00 percentile latency (ns) : 56189283\n",
"99.90 percentile latency (ns) : 56401684\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 2\n",
"target_qps : 15\n",
"target_latency (ns): 66666666\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 270336\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 64\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
"Finished running actual test.\n",
"Equivalent QPS computed by samples_per_query*target_qps : 30\n",
"Device Device:0 processed:\n",
" 270336 batches of size 2\n",
" Memcpy Calls: 0\n",
" PerSampleCudaMemcpy Calls: 0\n",
" BatchedCudaMemcpy Calls: 270336\n",
"&&&& PASSED Default_Harness # ./build/bin/harness_default\n",
"[2021-01-29 13:12:12,503 main.py:341 INFO] Result: Samples per query : 2 and Result is : VALID\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: Samples per query : 2 and Result is : VALID\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: No accuracy results in PerformanceOnly mode.\n",
"\n",
"\n",
"real 300m37.676s\n",
"user 5m11.760s\n",
"sys 0m27.856s\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Accuracy\n",
"\n",
"#### [Submitted experiment](https://github.com/mlcommons/inference_results_v1.0/blob/master/closed/NVIDIA/results/AGX_Xavier_TRT/ssd-resnet34/MultiStream/accuracy/accuracy.txt)\n",
"\n",
"\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"hash=859242388c9a94513b189eb58a55bd11ad1d2f7d094880dfb72157a7ac5e45fd\n",
"\n",
"\n",
"#### Reproduced experiment\n",
"\n",
"anton@xavier:/datasets/inference_results_v1.0/closed/NVIDIA$ time \\\n",
"MLPERF_SCRATCH_PATH=/datasets/mlperf_scratch_path \\\n",
"make run_harness RUN_ARGS="--benchmarks=ssd-resnet34 --scenarios=MultiStream --test_mode=AccuracyOnly"\n",
"...\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.201\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.188\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.238\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.179\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.411\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.430\n",
"mAP=20.111%\n",
"\n",
"======================= Perf harness results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: Cannot find performance result. Maybe you are running in AccuracyOnly mode.\n",
"\n",
"\n",
"======================= Accuracy results: =======================\n",
"\n",
"AGX_Xavier_TRT-default-MultiStream:\n",
" ssd-resnet34: Accuracy = 20.111, Threshold = 19.800. Accuracy test PASSED.\n",
"\n",
"\n",
"real 11m44.285s\n",
"user 4m22.432s\n",
"sys 0m17.144s\n",
""
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}