{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# [PUBLIC] MLPerf Inference - Image Classification - [ArmNN-TFLite](https://github.com/ARM-software/armnn-mlperf/)\n",
"\n",
"# Table of Contents\n",
"\n",
"1. [Platforms](#platforms)\n",
" 1. [`xavier`](#platforms_xavier)\n",
" 1. [`rpi4coral`](#platforms_rpi4coral)\n",
"\n",
"1. [Installation](#install)\n",
" 1. [Install system-wide prerequisites](#install_system)\n",
" 1. [Ubuntu 20.04](#install_system_ubuntu)\n",
" 1. [Install CK](#install_ck)\n",
" 1. [Set platform scripts](#set_platform_scripts)\n",
" 1. [Detect Python](#detect_python)\n",
" 1. [Detect GCC](#detect_gcc)\n",
" 1. [Install Python dependencies](#install_python_deps)\n",
" 1. [Install the MLPerf Inference repo](#install_inference_repo)\n",
" 1. [Prepare an ImageNet dataset](#prepare_imagenet)\n",
" 1. [Full](#prepare_imagenet_full) (50,000 images)\n",
" 1. [Reduced](#prepare_imagenet_min) (500 images)\n",
" 1. [Install CMake](#install_cmake)\n",
" 1. [Install SCons](#install_scons)\n",
" 1. [Install ArmNN-TFLite](#install_armnn_tflite)\n",
" 1. [Install ResNet50](#install_resnet50)\n",
" 1. [Quick test](#install_test)\n",
"\n",
"1. [Benchmarking accuracy](#benchmarking_accuracy)\n",
" 1. [Single Stream](#benchmarking_accuracy_singlestream)\n",
" 1. [Offline](#benchmarking_accuracy_offline)\n",
"\n",
"1. [Benchmarking performance](#benchmarking_performance)\n",
" 1. [Single Stream](#benchmarking_performance_singlestream)\n",
" 1. [`xavier`](#benchmarking_performance_singlestream_xavier)\n",
" 1. [`rpi4coral`](#benchmarking_performance_singlestream_rpi4coral)\n",
" 1. [Offline](#benchmarking_performance_offline)\n",
" 1. [`xavier`](#benchmarking_performance_offline_xavier)\n",
" 1. [`rpi4coral`](#benchmarking_performance_offline_rpi4coral)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Platforms"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## `xavier` ([NVIDIA Jetson AGX Xavier](https://www.nvidia.com/en-gb/autonomous-machines/embedded-systems/jetson-agx-xavier/))\n",
"anton@xavier:~$ uname -a\n",
"Linux xavier 4.9.201-tegra #1 SMP PREEMPT Fri Jan 15 14:54:23 PST 2021 aarch64 aarch64 aarch64 GNU/Linux\n",
"\n",
"\n",
"anton@xavier:~$ cat /etc/lsb-release\n",
"DISTRIB_ID=Ubuntu\n",
"DISTRIB_RELEASE=18.04\n",
"DISTRIB_CODENAME=bionic\n",
"DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"\n",
"anton@xavier:~$ lscpu\n",
"Architecture: aarch64\n",
"Byte Order: Little Endian\n",
"CPU(s): 8\n",
"On-line CPU(s) list: 0-7\n",
"Thread(s) per core: 1\n",
"Core(s) per socket: 2\n",
"Socket(s): 4\n",
"Vendor ID: Nvidia\n",
"Model: 0\n",
"Model name: ARMv8 Processor rev 0 (v8l)\n",
"Stepping: 0x0\n",
"CPU max MHz: 2265.6001\n",
"CPU min MHz: 115.2000\n",
"BogoMIPS: 62.50\n",
"L1d cache: 64K\n",
"L1i cache: 128K\n",
"L2 cache: 2048K\n",
"L3 cache: 4096K\n",
"Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp\n",
"anton@xavier:~$ sudo jetson_clocks\n",
"anton@xavier:~$ sudo jetson_clocks --show\n",
"SOC family:tegra194 Machine:Jetson-AGX\n",
"Online CPUs: 0-7\n",
"CPU Cluster Switching: Disabled\n",
"cpu0: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu1: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu2: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu3: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu4: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu5: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu6: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"cpu7: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 \n",
"GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000\n",
"EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1\n",
"Fan: speed=255\n",
"NV Power Mode: MAXN\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## `rpi4coral` ([Raspberry Pi 4](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/specifications/))\n",
"anton@rpi4coral:~$ uname -a\n",
"Linux rpi4coral 5.4.0-1028-raspi #31-Ubuntu SMP PREEMPT Wed Jan 20 11:30:45 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux\n",
"\n",
"anton@rpi4coral:~$ cat /etc/lsb-release\n",
"DISTRIB_ID=Ubuntu\n",
"DISTRIB_RELEASE=20.04\n",
"DISTRIB_CODENAME=focal\n",
"DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"\n",
"\n",
"\n",
"anton@rpi4coral:~$ lscpu\n",
"Architecture: aarch64\n",
"CPU op-mode(s): 32-bit, 64-bit\n",
"Byte Order: Little Endian\n",
"CPU(s): 4\n",
"On-line CPU(s) list: 0-3\n",
"Thread(s) per core: 1\n",
"Core(s) per socket: 4\n",
"Socket(s): 1\n",
"Vendor ID: ARM\n",
"Model: 3\n",
"Model name: Cortex-A72\n",
"Stepping: r0p3\n",
"CPU max MHz: 1500.0000\n",
"CPU min MHz: 600.0000\n",
"BogoMIPS: 108.00\n",
"Vulnerability Itlb multihit: Not affected\n",
"Vulnerability L1tf: Not affected\n",
"Vulnerability Mds: Not affected\n",
"Vulnerability Meltdown: Not affected\n",
"Vulnerability Spec store bypass: Vulnerable\n",
"Vulnerability Spectre v1: Mitigation; __user pointer sanitization\n",
"Vulnerability Spectre v2: Vulnerable\n",
"Vulnerability Srbds: Not affected\n",
"Vulnerability Tsx async abort: Not affected\n",
"Flags: fp asimd evtstrm crc32 cpuid\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install system-wide prerequisites\n",
"\n",
"**NB:** Run the below commands for your Linux system with `sudo` or as superuser.\n",
"\n",
"\n",
"#### Ubuntu 20.04 or similar\n",
"\n",
"\n",
"$ sudo apt update -y\n",
"$ sudo apt install -y apt-utils\n",
"$ sudo apt upgrade -y\n",
"$ sudo apt install -y\\\n",
" python3 python3-pip gcc g++\\\n",
" autoconf autogen libtool make cmake scons patch\\\n",
" git curl wget zip libz-dev libssl-dev vim\n",
"$ sudo apt clean\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install [Collective Knowledge](http://cknowledge.org/) (CK)\n",
"\n",
"anton@rpi4coral:~$ export CK_PYTHON=/usr/bin/python3\n",
"anton@rpi4coral:~$ $CK_PYTHON -m pip install --ignore-installed pip setuptools testresources --user\n",
"anton@rpi4coral:~$ $CK_PYTHON -m pip install ck\n",
"anton@rpi4coral:~$ echo 'export PATH=$HOME/.local/bin:$PATH' >> $HOME/.bashrc\n",
"anton@rpi4coral:~$ source $HOME/.bashrc\n",
"anton@rpi4coral:~$ ck version\n",
"V1.55.2\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install CK repositories\n",
"\n",
"### Install public repositories\n",
"\n",
"\n",
"anton@rpi4coral:~$ ck pull repo --url=https://github.com/krai/ck-mlperf\n",
"anton@rpi4coral:~$ ck pull repo --url=https://github.com/krai/ck-armnn\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Set platform scripts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use Raspberry Pi 4 settings with CPU frequency settings scripts\n",
"\n",
"anton@rpi4coral:~$ ck detect platform.os --platform_init_uoa=rpi4\n",
"\n",
"OS CK UOA: linux-64 (4258b5fe54828a50)\n",
"\n",
"OS name: Ubuntu 20.04.2 LTS\n",
"Short OS name: Linux 5.4.0\n",
"Long OS name: Linux-5.4.0-1028-raspi-aarch64-with-glibc2.29\n",
"OS bits: 64\n",
"OS ABI: aarch64\n",
"\n",
"Platform init UOA: 7be8e08bd8b93d15\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use generic Linux settings with dummy frequency setting scripts\n",
"\n",
"anton@diviniti:~$ ck detect platform.os --platform_init_uoa=generic-linux-dummy\n",
"\n",
"OS CK UOA: linux-64 (4258b5fe54828a50)\n",
"\n",
"OS name: Ubuntu 20.04.1 LTS\n",
"Short OS name: Linux 5.4.0\n",
"Long OS name: Linux-5.4.0-58-generic-x86_64-with-glibc2.29\n",
"OS bits: 64\n",
"OS ABI: x86_64\n",
"\n",
"Platform init UOA: -\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) Python\n",
"\n",
"\n",
"anton@rpi4coral:~$ export CK_PYTHON=/usr/bin/python3\n",
"anton@rpi4coral:~$ ck detect soft:compiler.python --full_path=$CK_PYTHON\n",
"anton@rpi4coral:~$ ck show env --tags=compiler,python\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"6dd55d431ea19241 linux-64 64 python 3.8.5 64bits,compiler,host-os-linux-64,lang-python,python,target-os-linux-64,v3,v3.8,v3.8.5\n",
"\n",
"\n",
"**NB:** CK can normally detect available Python interpreters automatically, but we are playing safe here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) GCC\n",
"\n",
"\n",
"anton@rpi4coral:~$ export CK_CC=/usr/bin/gcc\n",
"anton@rpi4coral:~$ ck detect soft:compiler.gcc --full_path=$CK_CC\n",
"anton@rpi4coral:~$ ck show env --tags=compiler,gcc\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"ebd4a04ea0ae7003 linux-64 64 GNU C compiler 9.3.0 64bits,compiler,gcc,host-os-linux-64,lang-c,lang-cpp,target-os-linux-64,v9,v9.3,v9.3.0\n",
"\n",
"\n",
"**NB:** CK can normally detect compilers automatically, but we are playing safe here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install Python dependencies (in userspace)\n",
"\n",
"#### Install implicit dependencies via pip\n",
"\n",
"**NB:** These dependencies are _implicit_, i.e. CK will not try to satisfy them. If they are not installed, however, the workflow will fail.\n",
"\n",
"\n",
"$ export CK_PYTHON=/usr/bin/python3\n",
"$ $CK_PYTHON -m pip install --user --upgrade \\\n",
" ...\n",
"...\n",
"Successfully installed...\n",
"\n",
"\n",
"#### Install explicit dependencies via CK (also via `pip`, but register with CK at the same time)\n",
"\n",
"**NB:** These dependencies are _explicit_, i.e. CK will try to satisfy them automatically. On a machine with multiple versions of Python, things can get messy, so we are playing safe here.\n",
"\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=python-package,numpy\n",
"\n",
"\n",
"##### To preproces ImageNet using OpenCV (default)\n",
"\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=python-package,opencv-python-headless\n",
"\n",
"\n",
"##### To preprocess ImageNet using Pillow (universal)\n",
"\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=python-package,pillow\n",
"\n",
"\n",
"\n",
"##### To preprocess ImageNet using TensorFlow (x86 only, ResNet50 only)\n",
"\n",
"\n",
"anton@krai:~$ ck install package --tags=python-package,tensorflow\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the MLPerf Inference repo and build LoadGen\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=mlperf,inference,source\n",
"anton@rpi4coral:~$ ck install package --tags=python-package,mlperf,loadgen\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Prepare an ImageNet dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Full"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Detect\n",
"\n",
"Unfortunately, the ImageNet 2012 validation dataset (50,000 images) [cannot be freely downloaded](https://github.com/mlcommons/inference/issues/542).\n",
"If you have a copy of it e.g. under `/datasets/dataset-imagenet-ilsvrc2012-val/`, you can register it with CK (\"detect\") as follows:\n",
"\n",
"\n",
"anton@rpi4coral:~$ ck detect soft:dataset.imagenet.val --extra_tags=full \\\n",
"--full_path=/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00000001.JPEG\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Preprocess\n",
"\n",
"**NB:** Since the preprocessed ImageNet dataset takes up from 7.1G to 29G, you may wish to change its destination directory by appending `--ask` to the below commands.\n",
"\n",
"\n",
"##### Using OpenCV (default)\n",
"\n",
"anton@rpi4coral:~$ ck install package --dep_add_tags.dataset-source=full \\\n",
"--tags=dataset,imagenet,val,full,preprocessed,using-opencv,side.224\n",
"\n",
"\n",
"\n",
"##### Using Pillow (universal)\n",
"\n",
"anton@rpi4coral:~$ ck install package --dep_add_tags.dataset-source=full \\\n",
"--tags=dataset,imagenet,val,full,preprocessed,using-pillow,side.224\n",
"\n",
"\n",
"\n",
"##### Using TensorFlow (x86 only, ResNet50 only)\n",
"\n",
"anton@krai:~$ ck install package --dep_add_tags.dataset-source=full \\\n",
"--tags=dataset,imagenet,val,full,preprocessed,using-tensorflow,side.224\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Reduced"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Download\n",
"\n",
"For testing purposes, you can download a reduced ImageNet dataset (the first 500 images):\n",
"\n",
"anton@rpi4coral:~$ ck install package:imagenet-2012-val-min
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"#### Preprocess\n",
"\n",
"\n",
"##### Using OpenCV (default)\n",
"\n",
"anton@rpi4coral:~$ ck install package --dep_add_tags.dataset-source=min \\\n",
"--tags=dataset,imagenet,val,preprocessed,using-opencv,side.224,first.500\n",
"\n",
"\n",
"\n",
"##### Using Pillow (universal)\n",
"\n",
"anton@rpi4coral:~$ ck install package --dep_add_tags.dataset-source=min \\\n",
"--tags=dataset,imagenet,val,preprocessed,using-pillow,side.224\n",
"\n",
"\n",
"\n",
"##### Using TensorFlow (x86 only, ResNet50 only)\n",
"\n",
"anton@rpi4coral:~$ ck install package --dep_add_tags.dataset-source=min \\\n",
"--tags=dataset,imagenet,val,preprocessed,using-tensorflow,side.224\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) CMake or install CMake from source\n",
"\n",
"### Detect\n",
"\n",
"Try to detect CMake on your system:\n",
"\n",
"anton@rpi4coral:~$ ck detect soft --tags=tool,cmake\n",
"anton@rpi4coral:~$ ck show env --tags=cmake\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"3354bea4c9d04b8c linux-64 64 cmake 3.16.3 64bits,cmake,host-os-linux-64,target-os-linux-64,tool,v3,v3.16,v3.16.3\n",
"\n",
"\n",
"### Install\n",
"\n",
"If this fails, try to install CMake from source:\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=tool,cmake,from.source\n",
"anton@rpi4coral:~$ ck show env --tags=tool,cmake,from.source\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"415293550c8e9de3 linux-64 64 cmake 3.18.2 64bits,cmake,compiled-by-gcc,compiled-by-gcc-9.3.0,host-os-linux-64,source,target-os-linux-64,tool,v3,v3.18,v3.18.2\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Detect (system) SCons or install SCons from source\n",
"\n",
"### Detect\n",
"\n",
"Try to detect SCons on your system:\n",
"\n",
"anton@rpi4coral:~$ ck detect soft --tags=tool,scons\n",
"anton@rpi4coral:~$ ck show env --tags=scons\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"a973cf94ddc40319 linux-64 64 scons 3.1.2.bee7caf9defd6e108fc2998a2520ddb36a967691, 2019-12-17 02:07:09, by bdeegan on octodog 64bits,host-os-linux-64,scons,target-os-linux-64,tool,v3,v3.1,v3.1.2,v3.1.2.0,v3.1.2.0.12,v3.1.2.0.12.0\n",
"\n",
"\n",
"### Install\n",
"\n",
"If this fails, try to install SCons from source:\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=tool,scons,v3.0\n",
"anton@rpi4coral:~$ ck show env --tags=tool,scons,v3.0\n",
"Env UID: Target OS: Bits: Name: Version: Tags:\n",
"\n",
"e1b425e435ca07ca linux-64 64 scons 3.0.0 64bits,host-os-linux-64,needs-python,needs-python-3.8.5,scons,target-os-linux-64,tool,v3,v3.0,v3.0.0\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the ArmNN library (inference engine) with the TFLite frontend\n",
"\n",
"### Use Neon backend\n",
"\n",
"#### `rpi4coral`\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=lib,armnn,tflite,neon,rel.20.11 \\\n",
"--env.CK_HOST_CPU_NUMBER_OF_PROCESSORS=3\n",
"\n",
"\n",
"#### `xavier`\n",
"\n",
"anton@xavier:~$ ck install package --tags=lib,armnn,tflite,neon,rel.20.11\n",
"\n",
"\n",
"### Use both Neon and OpenCL backends\n",
"\n",
"#### `firefly`\n",
"\n",
"\n",
"anton@firefly:~$ ck install package --tags=lib,armnn,tflite,neon,opencl,rel.20.11\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Install the ResNet50 model\n",
"\n",
"anton@rpi4coral:~$ ck install package --tags=model,tflite,resnet50,no-argmax\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Quick test\n",
"\n",
"### [`xavier`](#platforms_xavier)\n",
"\n",
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite --env.USE_NEON \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis --skip_stat_analysis \\\n",
"--dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--dep_add_tags.library=tflite,neon \\\n",
"--dep_add_tags.weights=resnet50 \\\n",
"--env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10\n",
"...\n",
"Summary: \n",
"--------------------------------------- \n",
"Graph loaded in 0.896963s\n",
"All images loaded in 0.012759s\n",
"All images classified in 0.720751s\n",
"Average classification time: 0.062536s\n",
"Accuracy top 1: 0.8 (8 of 10)\n",
"Accuracy top 5: 1.0 (10 of 10)\n",
"---------------------------------------\n",
"\n",
"\n",
"### [`rpi4coral`](#platforms_rpi4coral)\n",
"\n",
"anton@rpi4coral:~$ ck benchmark program:image-classification-armnn-tflite --env.USE_NEON \\\n",
"--speed --repetitions=1 --skip_print_timers --skip_stat_analysis --skip_stat_analysis \\\n",
"--dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--dep_add_tags.library=tflite,neon \\\n",
"--dep_add_tags.weights=resnet50 \\\n",
"--env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10\n",
"...\n",
"Summary: \n",
"--------------------------------------- \n",
"Graph loaded in 5.306322s \n",
"All images loaded in 0.195584s \n",
"All images classified in 3.373179s \n",
"Average classification time: 0.316994s\n",
"Accuracy top 1: 0.8 (8 of 10)\n",
"Accuracy top 5: 1.0 (10 of 10)\n",
"---------------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Benchmarking accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Single Stream\n",
"\n",
"### 50 samples\n",
"\n",
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=50 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 50\n",
"\n",
"Loading graph...\n",
"************* Type 2\n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess) \n",
"\n",
"\n",
"--------------------------------\n",
"accuracy=80.000%, good=40, total=50\n",
"\n",
"--------------------------------\n",
"\n",
"\n",
" (reading fine grain timers from tmp-ck-timer.json ...)\n",
"\n",
"\n",
"Execution time: 0.000 sec.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 500 samples\n",
"\n",
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 500\n",
"\n",
"Loading graph...\n",
"************* Type 2\n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: AccuracyOnly \n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess) \n",
"\n",
"\n",
"--------------------------------\n",
"accuracy=75.200%, good=376, total=500\n",
"\n",
"--------------------------------\n",
"\n",
"\n",
" (reading fine grain timers from tmp-ck-timer.json ...)\n",
"\n",
"\n",
"Execution time: 0.000 sec.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline\n",
"\n",
"### 50 samples\n",
"\n",
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=50 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 50\n",
"\n",
"Loading graph...\n",
"************* Type 2\n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQpQp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess) \n",
"\n",
"\n",
"--------------------------------\n",
"accuracy=80.000%, good=40, total=50\n",
"\n",
"--------------------------------\n",
"\n",
"\n",
" (reading fine grain timers from tmp-ck-timer.json ...)\n",
"\n",
"\n",
"Execution time: 0.000 sec.\n",
"\n",
"\n",
"### 500 samples\n",
"\n",
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=AccuracyOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /home/anton/CK-TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224-universal-unmutilated\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 500\n",
"\n",
"Loading graph...\n",
"************* Type 2\n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: AccuracyOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"Qpppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess) \n",
"\n",
"\n",
"--------------------------------\n",
"accuracy=75.200%, good=376, total=500\n",
"\n",
"--------------------------------\n",
"\n",
"\n",
" (reading fine grain timers from tmp-ck-timer.json ...)\n",
"\n",
"\n",
"Execution time: 0.000 sec.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Benchmarking performance"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Single Stream\n",
"\n",
"A valid SingleStream performance run [must reach](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios) a) the minimum duration of 600 seconds (**increased from 60 seconds for v1.0**), _and_ b) the minimum of 1,024 queries. Increasing the _expected_ SingleStream target latency in `user.conf` from 10 milliseconds to above ~60 milliseconds decreases the number of queries that LoadGen issues from 6,000 (actually, 12,000 to account for variability) to 1,024. Note that it does not matter whether the expected latency is, say, 100 ms or 1000 ms, as long as it is above ~60 ms."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`xavier`](#platforms_xavier)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=PerformanceOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_TARGET_LATENCY=50 \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 500\n",
"\n",
"Loading graph... \n",
"************* Type 2 \n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: PerformanceOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQp...\n",
"\n",
"\n",
"------------------------------------------------------------\n",
"| LATENCIES (in nanoseconds and fps) |\n",
"------------------------------------------------------------\n",
"Number of queries run: 12392\n",
"Min latency: 46634166ns (21.4435 fps) \n",
"Median latency: 47996271ns (20.835 fps) \n",
"Average latency: 48412717ns (20.6557 fps) \n",
"90 percentile latency: 49190529ns (20.3291 fps)\n",
"Max latency: 104701649ns (9.55095 fps)\n",
"------------------------------------------------------------\n",
"\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"anton@xavier:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_summary.txt\n",
"\n",
"\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : ArmNN_TFLite_SUT\n",
"Scenario : Single Stream\n",
"Mode : Performance\n",
"90th percentile latency (ns) : 49190529\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 20.65\n",
"QPS w/o loadgen overhead : 20.66\n",
"\n",
"Min latency (ns) : 46634166\n",
"Max latency (ns) : 104701649\n",
"Mean latency (ns) : 48412717\n",
"50.00 percentile latency (ns) : 47996271\n",
"90.00 percentile latency (ns) : 49190529\n",
"95.00 percentile latency (ns) : 50181708\n",
"97.00 percentile latency (ns) : 52748684\n",
"99.00 percentile latency (ns) : 58561230\n",
"99.90 percentile latency (ns) : 75919821\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 20.00000\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 600000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : 0\n",
"performance_issue_unique : 0\n",
"performance_issue_same : 0\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 1024\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`rpi4coral`](#platforms_rpi4coral)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"anton@rpi4coral:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=PerformanceOnly --env.CK_LOADGEN_SCENARIO=SingleStream \\\n",
"--env.CK_LOADGEN_TARGET_LATENCY=350 \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /home/anton/CK-TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224-universal-unmutilated\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 500\n",
"\n",
"Loading graph... \n",
"************* Type 2 \n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: SingleStream\n",
"LoadGen Mode: PerformanceOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"QpQpQp...\n",
"\n",
"...\n",
"\n",
"------------------------------------------------------------\n",
"| LATENCIES (in nanoseconds and fps) |\n",
"------------------------------------------------------------\n",
"Number of queries run: 1868\n",
"Min latency: 305771236ns (3.27042 fps) \n",
"Median latency: 322356429ns (3.10216 fps)\n",
"Average latency: 321413539ns (3.11126 fps)\n",
"90 percentile latency: 336773519ns (2.96935 fps)\n",
"Max latency: 364361210ns (2.74453 fps)\n",
"------------------------------------------------------------\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"anton@rpi4coral:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_summary.txt\n",
"\n",
"\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : ArmNN_TFLite_SUT\n",
"Scenario : Single Stream\n",
"Mode : Performance\n",
"90th percentile latency (ns) : 336773519\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 3.11\n",
"QPS w/o loadgen overhead : 3.11\n",
"\n",
"Min latency (ns) : 305771236\n",
"Max latency (ns) : 364361210\n",
"Mean latency (ns) : 321413539\n",
"50.00 percentile latency (ns) : 322356429\n",
"90.00 percentile latency (ns) : 336773519\n",
"95.00 percentile latency (ns) : 340211139\n",
"97.00 percentile latency (ns) : 342023852\n",
"99.00 percentile latency (ns) : 348308987\n",
"99.90 percentile latency (ns) : 358433387\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 2.85714\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 600000\n",
"max_duration (ms): 0\n",
"min_query_count : 1024\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : false\n",
"performance_issue_unique : false\n",
"performance_issue_same : false\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 1024\n",
"\n",
"No warnings encountered during test.\n",
"\n",
"No errors encountered during test.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Offline\n",
"\n",
"A valid Offline performance run [must reach](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios) a) the minimum duration of 600 seconds (**NB: increased from 60 seconds for v1.0**), _and_ b) the minimum of 24,576 samples."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`xavier`](#platforms_xavier)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"anton@xavier:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=PerformanceOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_TARGET_QPS=20 \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /datasets/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 500\n",
"\n",
"Loading graph... \n",
"************* Type 2 \n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: PerformanceOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"Qppp...\n",
"\n",
"...\n",
"\n",
"------------------------------------------------------------\n",
"| LATENCIES (in nanoseconds and fps) |\n",
"------------------------------------------------------------\n",
"Number of queries run: 24576\n",
"Min latency: 1213126438432ns (0.000824316 fps)\n",
"Median latency: 1213126438432ns (0.000824316 fps)\n",
"Average latency: 1213126438432ns (0.000824316 fps)\n",
"90 percentile latency: 1213126438432ns (0.000824316 fps)\n",
"Max latency: 1213126438432ns (0.000824316 fps)\n",
"------------------------------------------------------------\n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess)\n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"anton@xavier:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_summary.txt\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : ArmNN_TFLite_SUT\n",
"Scenario : Offline\n",
"Mode : Performance\n",
"Samples per second : 20.2584\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"Min latency (ns) : 1213126438432\n",
"Max latency (ns) : 1213126438432\n",
"Mean latency (ns) : 1213126438432\n",
"50.00 percentile latency (ns) : 1213126438432\n",
"90.00 percentile latency (ns) : 1213126438432\n",
"95.00 percentile latency (ns) : 1213126438432\n",
"97.00 percentile latency (ns) : 1213126438432\n",
"99.00 percentile latency (ns) : 1213126438432\n",
"99.90 percentile latency (ns) : 1213126438432\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 24576\n",
"target_qps : 20\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 600000\n",
"max_duration (ms): 0\n",
"min_query_count : 1\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : 0\n",
"performance_issue_unique : 0\n",
"performance_issue_same : 0\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 1024\n",
"\n",
"1 warning encountered. See detailed log.\n",
"\n",
"No errors encountered during test.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"anton@xavier:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_detail.txt | grep -i warning_generic_message\n",
":::MLLOG {"key": "warning_generic_message", "value": "Logging allocation detected: tid: 10690 reserved_entries: 1024 max_entries: 24580", "time_ms": 1213169.172754, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": true, "file": "/home/anton/CK_TOOLS/mlperf-inference-r1.0/inference/loadgen/logging.cc", "line_no": 605, "pid": 10690, "tid": 10690}}\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### [`rpi4coral`](#platforms_rpi4coral)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"anton@rpi4coral:~$ ck benchmark program:image-classification-armnn-tflite-loadgen --env.USE_NEON \\\n",
"--dep_add_tags.weights=resnet50 --dep_add_tags.images=preprocessed,using-opencv,side.224 \\\n",
"--env.CK_LOADGEN_MODE=PerformanceOnly --env.CK_LOADGEN_SCENARIO=Offline \\\n",
"--env.CK_LOADGEN_TARGET_QPS=3 \\\n",
"--env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_LOADGEN_BUFFER_SIZE=1024 \\\n",
"--repetitions=1 --skip_print_timers --skip_stat_analysis\n",
"...\n",
"Graph file: /home/anton/CK-TOOLS/model-tflite-mlperf-resnet-no-argmax-downloaded/resnet50_v1.no-argmax.tflite\n",
"Image dir: /home/anton/CK-TOOLS/dataset-imagenet-preprocessed-using-opencv-crop.875-full-inter.linear-side.224-universal-unmutilated\n",
"Image list: image_list.txt\n",
"Image size: 224\n",
"Image channels: 3\n",
"Prediction classes: 1000\n",
"Result dir: predictions\n",
"How many images fit in memory buffer: 1024\n",
"Normalize: 0\n",
"Subtract mean: 1\n",
"Per-channel means to subtract: 123.68, 116.78, 103.94\n",
"Number of available imagefiles: 500\n",
"\n",
"Loading graph... \n",
"************* Type 2 \n",
"Output tensor dimensions: 1*1001\n",
"Path to mlperf.conf : /home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/mlperf.conf\n",
"Path to user.conf : user.conf\n",
"Model Name: resnet50\n",
"LoadGen Scenario: Offline\n",
"LoadGen Mode: PerformanceOnly\n",
"CBllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\n",
"Qppp...\n",
"\n",
"...\n",
"\n",
"------------------------------------------------------------\n",
"| LATENCIES (in nanoseconds and fps) |\n",
"------------------------------------------------------------\n",
"Number of queries run: 24576\n",
"Min latency: 7780623339576ns (0.000128524 fps)\n",
"Median latency: 7780623339576ns (0.000128524 fps)\n",
"Average latency: 7780623339576ns (0.000128524 fps)\n",
"90 percentile latency: 7780623339576ns (0.000128524 fps)\n",
"Max latency: 7780623339576ns (0.000128524 fps)\n",
"------------------------------------------------------------ \n",
"U\n",
" (post processing via CK (/home/anton/CK/ck-mlperf/script/image-classification, loadgen_postprocess) \n",
"\n",
"\n",
"--------------------------------\n",
"--------------------------------\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"anton@rpi4coral:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_summary.txt\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : ArmNN_TFLite_SUT\n",
"Scenario : Offline\n",
"Mode : Performance\n",
"Samples per second : 3.15862\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"Min latency (ns) : 7780623339576\n",
"Max latency (ns) : 7780623339576\n",
"Mean latency (ns) : 7780623339576\n",
"50.00 percentile latency (ns) : 7780623339576\n",
"90.00 percentile latency (ns) : 7780623339576\n",
"95.00 percentile latency (ns) : 7780623339576\n",
"97.00 percentile latency (ns) : 7780623339576\n",
"99.00 percentile latency (ns) : 7780623339576\n",
"99.90 percentile latency (ns) : 7780623339576\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 24576\n",
"target_qps : 3\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 600000\n",
"max_duration (ms): 0\n",
"min_query_count : 1\n",
"max_query_count : 0\n",
"qsl_rng_seed : 12786827339337101903\n",
"sample_index_rng_seed : 12640797754436136668\n",
"schedule_rng_seed : 3135815929913719677\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"accuracy_log_sampling_target : 0\n",
"print_timestamps : 0\n",
"performance_issue_unique : 0\n",
"performance_issue_same : 0\n",
"performance_issue_same_index : 0\n",
"performance_sample_count : 1024\n",
"\n",
"1 warning encountered. See detailed log.\n",
"\n",
"No errors encountered during test.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"anton@rpi4coral:~$ cat $(ck find program:image-classification-armnn-tflite-loadgen)/tmp/mlperf_log_detail.txt | grep -i warning_generic_message\n",
":::MLLOG {"key": "warning_generic_message", "value": "Logging allocation detected: tid: 1890651 reserved_entries: 1024 max_entries: 24580", "time_ms": 7780759.263010, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": true, "file": "/home/anton/CK-TOOLS/mlperf-inference-r1.0/inference/loadgen/logging.cc", "line_no": 605, "pid": 1890651, "tid": 1890651}}\n",
""
]
}
],
"metadata": {
"celltoolbar": "Raw Cell Format",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}