{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# [PUBLIC] MLPerf Inference - Object Detection - TFLite\n",
"\n",
"# Table of Contents\n",
"\n",
"1. [Platforms](#platforms)\n",
" 1. [`xavier`](#platforms_xavier)\n",
" 1. [`rpi4coral`](#platforms_rpi4coral)\n",
"\n",
"1. [Installation](#install)\n",
" 1. [Install system-wide prerequisites](#install_system)\n",
" 1. [Ubuntu 20.04](#install_system_ubuntu)\n",
" 1. [Install CK](#install_ck)\n",
" 1. [Detect Python](#detect_python)\n",
" 1. [Detect GCC](#detect_gcc)\n",
" 1. [Install Python dependencies](#install_python_deps)\n",
" 1. [Download the COCO 2017 validation dataset](#download_dataset)\n",
" 1. [Preprocess the COCO 2017 validation dataset](#preprocess_dataset)\n",
" 1. [For SSD-MobileNet](#preprocess_dataset_for_ssd_mobilenet)\n",
" - [Preprocess using OpenCV](#preprocess_dataset_for_ssd_mobilenet_using_opencv)\n",
" - [Preprocess using Pillow](#preprocess_dataset_for_ssd_mobilenet_using_pillow)\n",
" 1. [Install CMake](#install_cmake)\n",
" 1. [Install TFLite](#install_tflite)\n",
" 1. [Install the TFLite model](#install_model)\n",
" 1. [Install the TF Model Garden](#install_model_garden)\n",
" 1. [Install the Python COCO API](#install_tool_coco)\n",
" 1. [Test](#test) (without LoadGen)\n",
" 1. [Quick](#test_quick)\n",
" - [Preprocessed using OpenCV](#test_quick_preprocessed_using_opencv)\n",
" - [Preprocessed using Pillow](#test_quick_preprocessed_using_pillow)\n",
" 1. [Full](#test_full)\n",
" - [Preprocessed using OpenCV](#test_full_preprocessed_using_opencv)\n",
" - [Preprocessed using Pillow](#test_full_preprocessed_using_pillow)\n",
" 1. [Install LoadGen](#install_loadgen)\n",
"\n",
"1. [Benchmarking accuracy](#benchmarking_accuracy)\n",
" 1. [Single Stream](#benchmarking_accuracy_singlestream)\n",
" 1. [Offline](#benchmarking_accuracy_offline)\n",
"\n",
"1. [Benchmarking performance](#benchmarking_performance)\n",
" 1. [Single Stream](#benchmarking_performance_singlestream)\n",
" 1. [`xavier`](#benchmarking_performance_singlestream_xavier)\n",
" 1. [`rpi4coral`](#benchmarking_performance_singlestream_rpi4coral)\n",
" 1. [Offline](#benchmarking_performance_offline)\n",
" 1. [`xavier`](#benchmarking_performance_offline_xavier)\n",
" 1. [`rpi4coral`](#benchmarking_performance_offline_rpi4coral) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Platforms"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## `xavier` ([NVIDIA Jetson AGX Xavier](https://www.nvidia.com/en-gb/autonomous-machines/embedded-systems/jetson-agx-xavier/))\n",
"\n",
"arjun@xavier:~$ uname -a\n",
"Linux xavier 4.9.201-tegra #1 SMP PREEMPT Fri Jan 15 14:54:23 PST 2021 aarch64 aarch64 aarch64 GNU/Linux\n",
"\n",
"\n",
"arjun@xavier:~$ cat /etc/lsb-release\n",
"DISTRIB_ID=Ubuntu\n",
"DISTRIB_RELEASE=18.04\n",
"DISTRIB_CODENAME=bionic\n",
"DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"\n",
"\n",
"arjun@xavier:~$ lscpu\n",
"Architecture: aarch64\n",
"Byte Order: Little Endian\n",
"CPU(s): 8\n",
"On-line CPU(s) list: 0-7\n",
"Thread(s) per core: 1\n",
"Core(s) per socket: 2\n",
"Socket(s): 4\n",
"Vendor ID: Nvidia\n",
"Model: 0\n",
"Model name: ARMv8 Processor rev 0 (v8l)\n",
"Stepping: 0x0\n",
"CPU max MHz: 2265.6001\n",
"CPU min MHz: 115.2000\n",
"BogoMIPS: 62.50\n",
"L1d cache: 64K\n",
"L1i cache: 128K\n",
"L2 cache: 2048K\n",
"L3 cache: 4096K\n",
"Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp\n",
"\n",
"\n",
"\n",
"arjun@xavier:~$ sudo /usr/sbin/nvpmodel -m 0\n",
"arjun@xavier:~$ sudo /usr/sbin/nvpmodel -d cool\n",
"arjun@xavier:~$ sudo /usr/bin/jetson_clocks --store\n",
"arjun@xavier:~$ sudo /usr/bin/jetson_clocks\n",
"arjun@xavier:~$ sudo /usr/bin/jetson_clocks --show\n",
"SOC family:tegra194 Machine:Jetson-AGX\n",
"Online CPUs: 0-7\n",
"cpu0: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu1: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu2: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu3: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu4: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu5: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu6: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"cpu7: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0\n",
"GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000\n",
"EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1\n",
"Fan: PWM=77\n",
"NV Power Mode: MAXN\n",
"arjun@xavier:~$ sudo /usr/bin/jetson_clocks --restore\n",
"arjun@xavier:~$ sudo /usr/bin/jetson_clocks --show\n",
"SOC family:tegra194 Machine:Jetson-AGX\n",
"Online CPUs: 0-7\n",
"cpu0: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1\n",
"cpu1: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1958400 IdleStates: C1=1 c6=1\n",
"cpu2: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1958400 IdleStates: C1=1 c6=1\n",
"cpu3: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1420800 IdleStates: C1=1 c6=1\n",
"cpu4: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1804800 IdleStates: C1=1 c6=1\n",
"cpu5: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1267200 IdleStates: C1=1 c6=1\n",
"cpu6: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1497600 IdleStates: C1=1 c6=1\n",
"cpu7: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2188800 IdleStates: C1=1 c6=1\n",
"GPU MinFreq=318750000 MaxFreq=1377000000 CurrentFreq=318750000\n",
"EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=408000000 FreqOverride=0\n",
"Fan: PWM=0\n",
"NV Power Mode: MAXN\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## `rpi4coral` ([Raspberry Pi 4](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/specifications/))\n",
"\n",
"arjun@rpi4coral:~$ uname -a\n",
"Linux rpi4coral 5.4.0-1028-raspi #31-Ubuntu SMP PREEMPT Wed Jan 20 11:30:45 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux\n",
"\n",
"\n",
"arjun@rpi4coral:~$ cat /etc/lsb-release\n",
"DISTRIB_ID=Ubuntu\n",
"DISTRIB_RELEASE=20.04\n",
"DISTRIB_CODENAME=focal\n",
"DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"\n",
"\n",
"\n",
"arjun@rpi4coral:~$ lscpu\n",
"Architecture: aarch64\n",
"CPU op-mode(s): 32-bit, 64-bit\n",
"Byte Order: Little Endian\n",
"CPU(s): 4\n",
"On-line CPU(s) list: 0-3\n",
"Thread(s) per core: 1\n",
"Core(s) per socket: 4\n",
"Socket(s): 1\n",
"Vendor ID: ARM\n",
"Model: 3\n",
"Model name: Cortex-A72\n",
"Stepping: r0p3\n",
"CPU max MHz: 1500.0000\n",
"CPU min MHz: 600.0000\n",
"BogoMIPS: 108.00\n",
"Vulnerability Itlb multihit: Not affected\n",
"Vulnerability L1tf: Not affected\n",
"Vulnerability Mds: Not affected\n",
"Vulnerability Meltdown: Not affected\n",
"Vulnerability Spec store bypass: Vulnerable\n",
"Vulnerability Spectre v1: Mitigation; __user pointer sanitization\n",
"Vulnerability Spectre v2: Vulnerable\n",
"Vulnerability Srbds: Not affected\n",
"Vulnerability Tsx async abort: Not affected\n",
"Flags: fp asimd evtstrm crc32 cpuid\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install system-wide prerequisites\n",
"\n",
"**NB:** Run the below commands for your Linux system with `sudo` or as superuser.\n",
"\n",
"\n",
"#### Ubuntu 20.04 or similar\n",
"\n",
"\n",
"$ sudo apt update -y\n",
"$ sudo apt install -y apt-utils\n",
"$ sudo apt upgrade -y\n",
"$ sudo apt install -y\\\n",
" python3 python3-pip gcc g++\\\n",
" autoconf autogen libtool make cmake patch\\\n",
" git curl wget zip libz-dev libssl-dev vim\n",
"$ sudo apt clean\n",
"\n",
"\n",
"\n",
"#### CentOS 8 or similar\n",
"\n",
"**TODO**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install [Collective Knowledge](http://cknowledge.org/) (CK)\n",
"\n",
"anton@xavier:~$ export CK_PYTHON=/usr/bin/python3\n",
"anton@xavier:~$ $CK_PYTHON -m pip install --ignore-installed pip setuptools testresources --user\n",
"anton@xavier:~$ $CK_PYTHON -m pip install ck\n",
"anton@xavier:~$ echo 'export PATH=$HOME/.local/bin:$PATH' >> $HOME/.bashrc\n",
"anton@xavier:~$ source $HOME/.bashrc\n",
"anton@xavier:~$ ck version\n",
"V1.55.2\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install CK repositories\n",
"\n",
"### Install public repositories\n",
"\n",
"\n",
"anton@xavier:~$ ck pull repo --url=https://github.com/krai/ck-mlperf\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Use generic Linux settings with dummy frequency setting scripts\n",
"\n",
"anton@xavier:~$ ck detect platform.os --platform_init_uoa=generic-linux-dummy\n",
"\n",
"OS CK UOA: linux-64 (4258b5fe54828a50)\n",
"\n",
"OS name: Ubuntu 18.04.5 LTS\n",
"Short OS name: Linux 4.9.201\n",
"Long OS name: Linux-4.9.201-tegra-aarch64-with-Ubuntu-18.04-bionic\n",
"OS bits: 64\n",
"OS ABI: aarch64\n",
"\n",
"Platform init UOA: -\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) Python\n",
"\n",
"\n",
"anton@xavier:~$ export CK_PYTHON=/usr/bin/python3\n",
"anton@xavier:~$ ck detect soft:compiler.python --full_path=$CK_PYTHON\n",
"anton@xavier:~$ ck show env --tags=compiler,python\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"cae4f0c2690e6b80 linux-64 64 python 3.6.9 64bits,compiler,host-os-linux-64,lang-python,python,target-os-linux-64,v3,v3.6,v3.6.9\n",
"\n",
"\n",
"**NB:** CK can normally detect available Python interpreters automatically, but we are playing safe here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) GCC\n",
"\n",
"\n",
"anton@xavier:~$ export CK_CC=/usr/bin/gcc\n",
"anton@xavier:~$ ck detect soft:compiler.gcc --full_path=$CK_CC\n",
"anton@xavier:~$ ck show env --tags=compiler,gcc\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"d57004dc9b28d525 linux-64 64 GNU C compiler 7.5.0 64bits,compiler,gcc,host-os-linux-64,lang-c,lang-cpp,target-os-linux-64,v7,v7.5,v7.5.0\n",
"\n",
"**NB:** CK can normally detect compilers automatically, but we are playing safe here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install Python dependencies (in userspace)\n",
"\n",
"#### Install implicit dependencies via pip\n",
"\n",
"**NB:** These dependencies are _implicit_, i.e. CK will not try to satisfy them. If they are not installed, however, the workflow will fail.\n",
"\n",
"\n",
"$ export CK_PYTHON=/usr/bin/python3\n",
"$ $CK_PYTHON -m pip install --user --upgrade \\\n",
" wheel\n",
"Successfully installed...\n",
"\n",
"\n",
"#### Install explicit dependencies via CK (also via `pip`, but register with CK at the same time)\n",
"\n",
"**NB:** These dependencies are _explicit_, i.e. CK will try to satisfy them automatically.\n",
"\n",
"You can still install them explicitly as follows:\n",
"\n",
"anton@xavier:~$ ck install package --tags=python-package,numpy\n",
"anton@xavier:~$ ck install package --tags=python-package,pillow\n",
"anton@xavier:~$ ck install package --tags=python-package,matplotlib\n",
"anton@xavier:~$ ck install package --tags=python-package,opencv-python-headless\n",
"anton@xavier:~$ ck install package --tags=python-package,absl\n",
"anton@xavier:~$ ck install package --tags=python-package,cython\n",
"\n",
"\n",
"\n",
"anton@xavier:~$ ck show env --tags=python-package\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"61046178de6ea4f9 linux-64 64 Python Pillow library 8.1.0 64bits,PIL,host-os-linux-64,lib,needs-python,needs-python-3.6.9,pillow,python-package,target-os-linux-64,v8,v8.1,v8.1.0,vmaster\n",
"ea795b39db83ac6d linux-64 64 Python OpenCV library (OpenCV without contribs or GUI) 4.5.1.48 64bits,cv2,headless,host-os-linux-64,lib,needs-python,needs-python-3.6.9,opencv,opencv-python-headless,python-package,target-os-linux-64,v4,v4.5,v4.5.1,v4.5.1.48,without-contribs,without-gui\n",
"34fc9a86b613bd92 linux-64 64 Python NumPy library 1.19.5 64bits,host-os-linux-64,lib,needs-python,needs-python-3.6.9,numpy,python-package,target-os-linux-64,v1,v1.19,v1.19.5,vmaster\n",
"a39e2fe603c41d98 linux-64 64 Python Matplotlib library 3.3.4 64bits,host-os-linux-64,lib,matplotlib,needs-python,needs-python-3.6.9,python-package,target-os-linux-64,v3,v3.3,v3.3.4,vmaster\n",
"ea921a2ee978fe56 linux-64 64 Python Abseil library unversioned 64bits,absl,absl-py,host-os-linux-64,lib,needs-python,needs-python-3.6.9,python-package,target-os-linux-64,v0,vmaster\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Download the dataset\n",
"\n",
"**NB:** The COCO 2017 validation dataset (5,000 images) takes ~1.6G. Use `--ask` to confirm the destination directory.\n",
"\n",
"\n",
"anton@xavier:~$ ck install package --tags=dataset,coco,val,2017 --ask\n",
"anton@xavier:~$ du -hs $(ck locate env --tags=dataset,coco,val,2017)\n",
"1.6G /datasets/dataset-coco-2017-val\n",
"\n",
"\n",
"**NB:** To save disk space, you can clean the training annotations after the installation:\n",
"\n",
"\n",
"anton@xavier:~$ du -hsc $(ck locate env --tags=dataset,coco,val,2017)/annotations/*train2017.json\n",
"88M /datasets/dataset-coco-2017-val/annotations/captions_train2017.json\n",
"449M /datasets/dataset-coco-2017-val/annotations/instances_train2017.json\n",
"228M /datasets/dataset-coco-2017-val/annotations/person_keypoints_train2017.json\n",
"764M total\n",
"anton@xavier:~$ rm $(ck locate env --tags=dataset,coco,val,2017)/annotations/*train2017.json\n",
"anton@xavier:~$ du -hs $(ck locate env --tags=dataset,coco,val,2017)\n",
"839M /datasets/dataset-coco-2017-val\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Preprocess the dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### For SSD-MobileNet-v1\n",
"\n",
"SSD-MobileNet-v1 requires resizing images to the 300x300 resolution.\n",
"\n",
"**NB:** As the COCO 2017 validation dataset preprocessed to 300x300 takes ~1.3G, you may want to use the `--ask` flag to confirm the destination directory interactively."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Preprocess using OpenCV\n",
"\n",
"anton@xavier:~$ ck install package --tags=dataset,coco.2017,preprocessed,using-opencv,full,side.300 --ask\n",
"anton@xavier:~$ du -hs $(ck locate env --tags=dataset,coco.2017,preprocessed,using-opencv,full,side.300)\n",
"1.3G /home/anton/CK_TOOLS/dataset-object-detection-preprocessed-using-opencv-coco.2017-full-side.300\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Preprocess using Pillow\n",
"\n",
"anton@xavier:~$ ck install package --tags=dataset,coco.2017,preprocessed,using-pillow,full,side.300 --ask\n",
"anton@xavier:~$ du -hs $(ck locate env --tags=dataset,coco.2017,preprocessed,using-pillow,full,side.300)\n",
"1.3G /home/anton/CK_TOOLS/dataset-object-detection-preprocessed-using-pillow-coco.2017-full-side.300\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) CMake or install CMake from source\n",
"\n",
"### Detect\n",
"\n",
"Try to detect CMake on your system:\n",
"\n",
"anton@xavier:~$ ck detect soft --tags=tool,cmake\n",
"anton@xavier:~$ ck show env --tags=tool,cmake\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"0305242ae838cf05 linux-64 64 cmake 3.17.3 64bits,cmake,host-os-linux-64,target-os-linux-64,tool,v3,v3.17,v3.17.3\n",
"\n",
"\n",
"### Install\n",
"\n",
"If this fails or if the CMake version is pre-`3.16`, try to install it from source:\n",
"\n",
"anton@xavier:~$ ck install package --tags=tool,cmake,from.source\n",
"anton@xavier:~$ ck show env --tags=tool,cmake,from.source\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"49c8514914650815 linux-64 64 cmake 3.19.5 64bits,cmake,compiled,compiled-by-gcc,compiled-by-gcc-7.5.0,from.source,host-os-linux-64,source,target-os-linux-64,tool,v3,v3.19,v3.19.5\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the TFLite library (inference engine)\n",
"\n",
"### Generic\n",
"\n",
"anton@xavier:~$ ck install package --tags=lib,tflite,via-cmake\n",
"\n",
"\n",
"### `xavier`\n",
"\n",
"arjun@xavier:~$ ck install package --tags=lib,via-cmake,with.ruy\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the SSD-MobileNet-v1 TFLite model\n",
"\n",
"anton@xavier:~$ ck install package --tags=model,tflite,ssd-mobilenet --no_tags=edgetpu\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the [TensorFlow Model Garden (API)](https://github.com/tensorflow/models)\n",
"\n",
"anton@xavier:~$ ck install package --tags=tensorflowmodel,api\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the [Python COCO API](https://github.com/cocodataset/cocoapi)\n",
"\n",
"anton@xavier:~$ ck install package --tags=tool,coco,api\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Test (without LoadGen)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Quick\n",
"\n",
"\n",
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.lib-tflite=via-cmake \\\n",
"--env.CK_BATCH_COUNT=50 --env.CK_BATCH_SIZE=1\n",
"...\n",
"Evaluate metrics as coco ...\n",
"loading annotations into memory...\n",
"Done (t=0.91s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.01s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.41s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.59s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.409\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.272\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.026\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.226\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.610\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.223\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.268\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.270\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.030\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.228\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644\n",
"\n",
"Summary:\n",
"-------------------------------\n",
"All images loaded in 0.268237s\n",
"Average image load time: 0.005365s\n",
"All images detected in 1.657856s\n",
"Average detection time: 0.031017s\n",
"Total NMS time: 0.062935s\n",
"Average NMS time: 0.001259s\n",
"mAP: 0.25566726658312644\n",
"Recall: 0.2696417278289413\n",
"--------------------------------\n",
"\n",
"\n",
"\n",
"#### Preprocessed using Pillow\n",
"\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-pillow,side.300 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.lib-tflite=via-cmake \\\n",
"--env.CK_BATCH_COUNT=50 --env.CK_BATCH_SIZE=1\n",
"...\n",
"Evaluate metrics as coco ...\n",
"loading annotations into memory...\n",
"Done (t=0.89s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.00s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.31s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.46s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.228\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.359\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.236\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.560\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.205\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.243\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.243\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.026\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.196\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.596\n",
"\n",
"Summary:\n",
"-------------------------------\n",
"All images loaded in 0.062045s\n",
"Average image load time: 0.001241s\n",
"All images detected in 1.728882s\n",
"Average detection time: 0.032712s\n",
"Total NMS time: 0.060354s\n",
"Average NMS time: 0.001207s\n",
"mAP: 0.2275409744679146\n",
"Recall: 0.24346685802358603\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Full\n",
"\n",
"\n",
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.lib-tflite=via-cmake \\\n",
"--env.CK_BATCH_COUNT=5000 --env.CK_BATCH_SIZE=1\n",
"...\n",
"Evaluate metrics as coco ...\n",
"loading annotations into memory...\n",
"Done (t=0.88s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.35s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=28.90s).\n",
"Accumulating evaluation results...\n",
"DONE (t=4.60s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.231\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.350\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.254\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.018\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.167\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.529\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.209\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.263\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.264\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.604\n",
"\n",
"Summary:\n",
"-------------------------------\n",
"All images loaded in 46.736076s\n",
"Average image load time: 0.009347s\n",
"All images detected in 138.532166s\n",
"Average detection time: 0.026915s\n",
"Total NMS time: 3.927624s\n",
"Average NMS time: 0.000786s\n",
"mAP: 0.23129300490236354\n",
"Recall: 0.263527133917118\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Preprocessed using Pillow\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-pillow,side.300 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.lib-tflite=via-cmake \\\n",
"--env.CK_BATCH_COUNT=5000 --env.CK_BATCH_SIZE=1\n",
"...\n",
"Evaluate metrics as coco ...\n",
"loading annotations into memory...\n",
"Done (t=0.89s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.14s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=27.72s).\n",
"Accumulating evaluation results...\n",
"DONE (t=4.25s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.223\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.341\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.247\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.015\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.160\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.515\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.255\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.255\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.182\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.593\n",
"\n",
"Summary:\n",
"-------------------------------\n",
"All images loaded in 5.572460s\n",
"Average image load time: 0.001114s\n",
"All images detected in 138.749115s\n",
"Average detection time: 0.026947s\n",
"Total NMS time: 3.966329s\n",
"Average NMS time: 0.000793s\n",
"mAP: 0.22349681099305302\n",
"Recall: 0.2550505369422975\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the MLPerf Inference repo and build LoadGen\n",
"\n",
"anton@xavier:~$ ck install package --tags=mlperf,inference,source\n",
"anton@xavier:~$ ck install package --tags=python-package,mlperf,loadgen\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Benchmarking accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Single Stream"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 50 samples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=50 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-opencv.50 \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-opencv,50\n",
"...\n",
"Graph file: /home/anton/CK_TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"Image dir: /home/anton/CK_TOOLS/dataset-object-detection-preprocessed-using-opencv-coco.2017-full-side.300\n",
"Image list: original_dimensions.txt\n",
"Image size: 300*300\n",
"Image channels: 3\n",
"Result dir: detections\n",
"Batch count: 1\n",
"Batch size: 1\n",
"Normalize: 1\n",
"Subtract mean: 0\n",
"Image count in file: 50\n",
"Graph file: /home/anton/CK_TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"Image dir: /home/anton/CK_TOOLS/dataset-object-detection-preprocessed-using-opencv-coco.2017-full-side.300\n",
"Image list: original_dimensions.txt\n",
"Image size: 300*300\n",
"Image channels: 3\n",
"Result dir: detections\n",
"Batch count: 1\n",
"Batch size: 1\n",
"Normalize: 1\n",
"Subtract mean: 0\n",
"Image count in file: 50\n",
"\n",
"Loading graph...\n",
"Loaded model /home/anton/CK_TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"resolved reporter\n",
"\n",
"Number of threads: 8\n",
"tensors size: 184\n",
"nodes size: 64\n",
"number of inputs: 1\n",
"number of outputs: 4\n",
"input(0) name: normalized_input_image_tensor\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.88s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.02s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.29s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.50s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.409\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.272\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.026\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.226\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.610\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.223\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.268\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.270\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.030\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.228\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644\n",
"mAP=25.554%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using Pillow\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-pillow,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=50 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-pillow,50 \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-pillow,50\n",
"...\n",
"Graph file: /home/anton/CK_TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"Image dir: /home/anton/CK_TOOLS/dataset-object-detection-preprocessed-using-pillow-coco.2017-full-side.300\n",
"Image list: original_dimensions.txt\n",
"Image size: 300*300\n",
"Image channels: 3\n",
"Result dir: detections\n",
"Batch count: 1\n",
"Batch size: 1\n",
"Normalize: 1\n",
"Subtract mean: 0\n",
"Image count in file: 50\n",
"Graph file: /home/anton/CK_TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"Image dir: /home/anton/CK_TOOLS/dataset-object-detection-preprocessed-using-pillow-coco.2017-full-side.300\n",
"Image list: original_dimensions.txt\n",
"Image size: 300*300\n",
"Image channels: 3\n",
"Result dir: detections\n",
"Batch count: 1\n",
"Batch size: 1\n",
"Normalize: 1\n",
"Subtract mean: 0\n",
"Image count in file: 50\n",
"\n",
"Loading graph...\n",
"Loaded model /home/anton/CK_TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"resolved reporter\n",
"\n",
"Number of threads: 8\n",
"tensors size: 184\n",
"nodes size: 64\n",
"number of inputs: 1\n",
"number of outputs: 4\n",
"input(0) name: normalized_input_image_tensor\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.89s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.02s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.25s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.44s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.228\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.359\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.236\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.560\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.205\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.243\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.243\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.026\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.196\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.596\n",
"mAP=22.754%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 500 samples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-opencv.500 \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-opencv,500\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQppQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQppQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.87s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.03s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=2.46s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.18s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.254\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.377\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.280\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.020\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.181\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.543\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.228\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.281\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.281\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.190\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.585\n",
"mAP=25.382%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using Pillow\n",
"\n",
"arjun@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-pillow,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-pillow.500 \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-pillow,500\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQppQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQppQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"...\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.87s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.02s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=2.33s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.15s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.245\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.362\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.268\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.014\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.177\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.517\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.223\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.271\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.272\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.016\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.186\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.561\n",
"mAP=24.475%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5000 samples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=5000 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-opencv.5000 \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-opencv,5000\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBl...\n",
"...\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"...\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.89s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.23s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=24.71s).\n",
"Accumulating evaluation results...\n",
"DONE (t=4.14s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.231\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.350\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.254\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.018\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.167\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.529\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.209\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.263\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.264\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.604\n",
"mAP=23.137%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using Pillow\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-pillow,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=5000 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-pillow.5000 \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-pillow,5000\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBl\n",
"...\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.89s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.21s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=23.94s).\n",
"Accumulating evaluation results...\n",
"DONE (t=3.82s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.224\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.341\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.247\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.015\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.160\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.515\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.255\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.255\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.182\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.593\n",
"mAP=22.350%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 50 samples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=50 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-opencv.50.offline \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-opencv,50,offline\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"Qpppppppppppppppppppppppppppppppppppppppppppppppppp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.97s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.02s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.25s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.44s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.409\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.272\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.026\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.226\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.610\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.223\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.268\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.270\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.030\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.228\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644\n",
"mAP=25.554%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 500 samples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-opencv.500.offline \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-opencv,500,offline\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"Qpppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.89s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.03s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=2.42s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.12s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.254\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.377\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.280\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.020\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.181\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.543\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.228\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.281\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.281\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.190\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.585\n",
"mAP=25.382%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5000 samples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Preprocessed using OpenCV\n",
"\n",
"anton@xavier:~$ ck benchmark program:object-detection-tflite-loadgen \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis \\\n",
"--dep_add_tags.dataset=preprocessed,using-opencv,side.300 \\\n",
"--dep_add_tags.library=tflite,via-cmake,v2.4 \\\n",
"--dep_add_tags.weights=ssd-mobilenet \\\n",
"--dep_add_tags.python=v3.7 --dep_add_tags.tool-coco=needs-python-3.7.5 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=5000 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--record --record_repo=local --process_multi_keys \\\n",
"--record_uoa=mlperf.object-detection.ssd-mobilenet.tflite.accuracy.using-opencv.5000.offline \\\n",
"--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy,using-opencv,5000,offline\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: AccuracyOnly\n",
"CBl...\n",
"...\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"loading annotations into memory...\n",
"Done (t=0.90s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"DONE (t=0.22s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=24.21s).\n",
"Accumulating evaluation results...\n",
"DONE (t=3.96s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.231\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.350\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.254\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.018\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.167\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.529\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.209\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.263\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.264\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.604\n",
"mAP=23.137%\n",
"\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Benchmarking performance"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Single Stream\n",
"\n",
"A valid SingleStream performance run [must reach](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios) a) the minimum duration of 600 seconds (**NB: increased from 60 seconds for v1.0**), _and_ b) the minimum of 1,024 queries. Increasing the _expected_ SingleStream target latency in `user.conf` from 10 milliseconds to above ~60 milliseconds decreases the number of queries that LoadGen issues from 6,000 (actually, 12,000 to account for variability) to 1,024. Note that it does not matter whether the expected latency is, say, 100 ms or 1000 ms, as long as it is above ~60 ms."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`xavier`](#platforms_xavier)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"arjun@xavier:~$ ck benchmark program:object-detection-tflite-loadgen --repetitions=1 --env.CK_METRIC_TYPE=COCO --env.CK_LOADGEN_SCENARIO=SingleStream --record --tags=mlperf,object-detection,ssd-mobilenet,tflite,performance --process_multi_keys --skip_stat_analysis --env.CK_LOADGEN_MODE=PerformanceOnly --env.CK_LOADGEN_DATASET_SIZE=500 --record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tflite-performance --skip_print_timers
\n",
"...\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/arjun/CK-TOOLS/mlperf-inference-r0.7/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: PerformanceOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"\n",
"------------------------------------------------------------\n",
"| LATENCIES (in nanoseconds and fps) |\n",
"------------------------------------------------------------\n",
"Number of queries run: 1843\n",
"Min latency: 26793327ns (37.3227 fps)\n",
"Median latency: 31173978ns (32.078 fps)\n",
"Average latency: 32579448ns (30.6942 fps)\n",
"90 percentile latency: 36719640ns (27.2334 fps)\n",
"Max latency: 57667165ns (17.3409 fps)\n",
"------------------------------------------------------------\n",
"U\n",
" (post processing via CK (/home/arjun/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
"\n",
"\n",
" (reading fine grain timers from tmp-ck-timer.json ...)\n",
"\n",
"\n",
"Execution time: 0.000 sec.\n",
"***************************************************************************************\n",
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"arjun@xavier:~$ cat $( find program:object-detection-tflite-loadgen)/tmp/mlperf_log_summary.txt\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : TFLite_SUT\n",
"Scenario : Single Stream\n",
"Mode : Performance\n",
"90th percentile latency (ns) : 36719640\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 30.68\n",
"QPS w/o loadgen overhead : 30.69\n",
"\n",
"Min latency (ns) : 26793327\n",
"Max latency (ns) : 57667165\n",
"Mean latency (ns) : 32579448\n",
"50.00 percentile latency (ns) : 31173978\n",
"90.00 percentile latency (ns) : 36719640\n",
"95.00 percentile latency (ns) : 39314159\n",
"97.00 percentile latency (ns) : 41228743\n",
"99.00 percentile latency (ns) : 44026890\n",
"99.90 percentile latency (ns) : 54121338\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 100\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 256\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`rpi4coral`](#platforms_rpi4coral)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline\n",
"\n",
"A valid Offline performance run [must reach](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios) a) the minimum duration of 60 seconds (**NB: increased to 600 seconds for v1.0**), _and_ b) the minimum of 24,576 samples."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`rpi4coral`](#platforms_rpi4coral)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"arjun@rpi4coral:~$ ck benchmark program:object-detection-tflite-loadgen --repetitions=1 --env.CK_METRIC_TYPE=COCO --env.CK_LOADGEN_SCENARIO=Offline --record --tags=mlperf,object-detection,ssd-mobilenet,tflite,performance --process_multi_keys --env.CK_LOADGEN_MODE=PerformanceOnly --env.CK_LOADGEN_DATASET_SIZE=5 --record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tflite-performance --skip_stat_analysis --skip_print_timers --env.CK_LOADGEN_TARGET_QPS=1 --env.CK_LOADGEN_BUFFER_SIZE=1024
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" (run ...)\n",
"executing code ...\n",
"Graph file: /home/arjun/CK-TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"Image dir: /home/arjun/CK-TOOLS/dataset-object-detection-preprocessed-using-pillow-coco.2017-first.20-side.300\n",
"Image list: original_dimensions.txt\n",
"Image size: 300*300\n",
"Image channels: 3\n",
"Result dir: detections\n",
"Batch count: 1\n",
"Batch size: 1\n",
"Normalize: 1\n",
"Subtract mean: 0\n",
"Image count in file: 5\n",
"Graph file: /home/arjun/CK-TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"Image dir: /home/arjun/CK-TOOLS/dataset-object-detection-preprocessed-using-pillow-coco.2017-first.20-side.300\n",
"Image list: original_dimensions.txt\n",
"Image size: 300*300\n",
"Image channels: 3\n",
"Result dir: detections\n",
"Batch count: 1\n",
"Batch size: 1\n",
"Normalize: 1\n",
"Subtract mean: 0\n",
"Image count in file: 5\n",
"\n",
"Loading graph...\n",
"Loaded model /home/arjun/CK-TOOLS/model-tflite-mlperf-ssd-mobilenet-downloaded-from-zenodo/detect_regular_nms.tflite\n",
"resolved reporter\n",
"\n",
"Number of threads: 4\n",
"tensors size: 213\n",
"nodes size: 64\n",
"number of inputs: 1\n",
"number of outputs: 4\n",
"input(0) name: normalized_input_image_tensor\n",
"\n",
"\n",
"Input tensor dimensions (NHWC): 1*300*300*3\n",
"Detection boxes tensor dimensions: 1*100*4\n",
"Detection classes tensor dimensions: 1*100\n",
"Detection scores tensor dimensions: 1*100\n",
"Number of detections tensor dimensions: 1*1\n",
"Path to mlperf.conf : /home/arjun/CK-TOOLS/mlperf-inference-r0.7/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: ssd-mobilenet\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: PerformanceOnly\n",
"CBlllll\n",
"Qpppppppppppppppppppppppppppppppp......\n",
"\n",
"------------------------------------------------------------\n",
"| LATENCIES (in nanoseconds and fps) |\n",
"------------------------------------------------------------\n",
"Number of queries run: 24576\n",
"Min latency: 5537079183743ns (0.000180601 fps)\n",
"Median latency: 5537079183743ns (0.000180601 fps)\n",
"Average latency: 5537079183743ns (0.000180601 fps)\n",
"90 percentile latency: 5537079183743ns (0.000180601 fps)\n",
"Max latency: 5537079183743ns (0.000180601 fps)\n",
"------------------------------------------------------------\n",
"U\n",
" (post processing via CK (/home/arjun/CK/ck-mlperf/script/object-detection, loadgen_postprocess)\n",
"\n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
""
]
}
],
"metadata": {
"celltoolbar": "Raw Cell Format",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}