{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A study of Object Detection models by [dividiti](http://dividiti.com)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. [Overview](#overview)\n",
"1. [Platform](#platform)\n",
"1. [Settings](#settings)\n",
"1. [Get experimental data](#get_data)\n",
"1. [Access experimental data](#access_data)\n",
"1. [Plot experimental data](#plot_data)\n",
" 1. [Plot accuracy](#plot_accuracy)\n",
" 1. [Plot performance](#plot_performance)\n",
" 1. [Plot exploration](#plot_exploration)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This Jupyter Notebook studies performance (execution time per image, images per seconds) vs accuracy (mAP, Recall) of several Object Detection models on different size objects (large, medium and small). The experiments are performed via TensorFlow with several execution options on the CPU and the GPU."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Model | Unique CK Tags (``) | Is Custom? | mAP in % |\n",
"| --- | --- | --- | --- |\n",
"| [`faster_rcnn_nas_lowproposals_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `rcnn,nas,lowproposals,vcoco` | 0 | 44.340195 |\n",
"| [`faster_rcnn_resnet50_lowproposals_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `rcnn,resnet50,lowproposals` | 0 | 24.241037 |\n",
"| [`faster_rcnn_resnet101_lowproposals_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `rcnn,resnet101,lowproposals` | 0 | 32.594327 |\n",
"| [`faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `rcnn,inception-resnet-v2,lowproposals` | 0 | 36.520117 |\n",
"| [`faster_rcnn_inception_v2_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `rcnn,inception-v2` | 0 | 28.309626 |\n",
"| [`ssd_inception_v2_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `ssd,inception-v2` | 0 | 27.765988 |\n",
"| [`ssd_mobilenet_v1_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `ssd,mobilenet-v1,non-quantized,mlperf,tf` | 0 | 23.111170 |\n",
"| [`ssd_mobilenet_v1_quantized_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md)| `ssd,mobilenet-v1,quantized,mlperf,tf` | 0 | 23.591693 |\n",
"| [`ssd_mobilenet_v1_fpn_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `ssd,mobilenet-v1,fpn` | 0 | 35.353170 |\n",
"| [`ssd_resnet_50_fpn_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `ssd,resnet50,fpn` | 0 | 38.341120 |\n",
"| [`ssdlite_mobilenet_v2_coco`](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) | `ssdlite,mobilenet-v2,vcoco` | 0 | 24.281540 |\n",
"| [`yolo_v3_coco`](https://github.com/YunYang1994/tensorflow-yolov3) | `yolo-v3` | 1 | 28.532508 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Platform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"See our [Docker container](https://github.com/ctuning/ck-object-detection/blob/master/docker/object-detection-tf-py.tensorrt.ubuntu-18.04/README.md) for more information on the software configuration."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### CPU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" - Model:\n",
" - Intel Xeon E5-2650 v3\n",
" - Frequency:\n",
" - 2.3 GHz\n",
" - Number of physical cores:\n",
" - 10\n",
" - Number of virtual cores (hyperthreading):\n",
" - 20\n",
" - RAM:\n",
" - 32 GB\n",
" - OS:\n",
" - Ubuntu 16.04 LTS Linux"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### GPU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" - Model:\n",
" - NVIDIA GeForce GTX 1080\n",
" - Frequency:\n",
" - 1.6 GHz\n",
" - RAM:\n",
" - 8 GB\n",
" - CUDA version:\n",
" - 10.2\n",
" - Driver version:\n",
" - 430.14"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Settings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**NB:** Please ignore this section if you are not interested in re-running or modifying this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Includes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Standard"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"import json\n",
"import re"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Scientific"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If some of the scientific packages are missing, please install them using:\n",
"```bash\n",
"$ python -m pip install jupyter pandas numpy matplotlib --user\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import IPython as ip\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib as mp\n",
"import seaborn as sb"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print ('IPython version: %s' % ip.__version__)\n",
"print ('Pandas version: %s' % pd.__version__)\n",
"print ('NumPy version: %s' % np.__version__)\n",
"print ('Matplotlib version: %s' % mp.__version__)\n",
"print ('Seaborn version: %s' % sb.__version__)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Image, display\n",
"def display_in_full(df):\n",
" pd.options.display.max_columns = len(df.columns)\n",
" pd.options.display.max_rows = len(df.index)\n",
" display(df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"from matplotlib import cm\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"default_colormap = cm.autumn\n",
"default_fontsize = 16\n",
"default_barwidth = 0.8\n",
"default_figwidth = 24\n",
"default_figheight = 3\n",
"default_figdpi = 200\n",
"default_figsize = [default_figwidth, default_figheight]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if mp.__version__[0]=='2': mp.style.use('classic')\n",
"mp.rcParams['figure.max_open_warning'] = 200\n",
"mp.rcParams['figure.dpi'] = default_figdpi\n",
"mp.rcParams['font.size'] = default_fontsize\n",
"mp.rcParams['legend.fontsize'] = 'medium'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"save_fig_ext = 'png'\n",
"save_fig_dir = os.path.join(os.path.expanduser(\"~\"), 'omnibench')\n",
"if not os.path.exists(save_fig_dir):\n",
" os.mkdir(save_fig_dir)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pprint import pprint"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Collective Knowledge"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If CK is not installed, please install it using:\n",
"```bash\n",
"$ python -m pip install ck --user\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import ck.kernel as ck\n",
"print ('CK version: %s' % ck.__version__)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Get the experimental data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download experimental data and add CK repositories as follows:\n",
"```bash\n",
"$ wget https://www.dropbox.com/s/0mxkvlstico349n/ckr-medium-object-detection-accuracy.zip\n",
"$ ck add repo --zip=ckr-medium-object-detection-accuracy.zip\n",
"\n",
"$ wget https://www.dropbox.com/s/zy68dsmp1yzv703/ckr-medium-object-detection-performance-docker.zip\n",
"$ ck add repo --zip=ckr-medium-object-detection-performance-docker.zip\n",
"\n",
"$ wget https://www.dropbox.com/s/7829e4zmmbgkqyu/ckr-medium-object-detection-performance-native.zip\n",
"$ ck add repo --zip=ckr-medium-object-detection-performance-native.zip\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"repo_uoa = 'medium-object-detection-accuracy'\n",
"print (\"*\"*80)\n",
"print (repo_uoa)\n",
"print (\"*\"*80)\n",
"!ck list $repo_uoa:experiment:* | sort\n",
"print (\"\")\n",
"\n",
"perf_docker_repo_uoa = 'medium-object-detection-performance-docker'\n",
"print (\"*\"*80)\n",
"print (perf_docker_repo_uoa)\n",
"print (\"*\"*80)\n",
"!ck list $perf_docker_repo_uoa:experiment:* | sort\n",
"print (\"\")\n",
"\n",
"perf_native_repo_uoa = 'medium-object-detection-performance-native'\n",
"print (\"*\"*80)\n",
"print (perf_native_repo_uoa)\n",
"print (\"*\"*80)\n",
"!ck list $perf_native_repo_uoa:experiment:* | sort"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Access the experimental data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"def get_experimental_results(repo_uoa, module_uoa='experiment', tags='', accuracy=True):\n",
" r = ck.access({'action':'search', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'tags':tags})\n",
" #pprint (r)\n",
" if r['return']>0:\n",
" print('Error: %s' % r['error'])\n",
" exit(1)\n",
" experiments = r['lst']\n",
"\n",
" dfs = []\n",
" for experiment in experiments:\n",
" data_uoa = experiment['data_uoa']\n",
" r = ck.access({'action':'list_points', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'data_uoa':data_uoa})\n",
" \n",
" pipeline_file_path = os.path.join(r['path'], 'pipeline.json')\n",
" with open(pipeline_file_path) as pipeline_file:\n",
" pipeline_data_raw = json.load(pipeline_file)\n",
" weights_env = pipeline_data_raw['dependencies']['weights']['dict']['env']\n",
" image_width = np.int64(weights_env.get('CK_ENV_TENSORFLOW_MODEL_DEFAULT_WIDTH',-1))\n",
" image_height = np.int64(weights_env.get('CK_ENV_TENSORFLOW_MODEL_DEFAULT_HEIGHT',-1))\n",
" \n",
" tags = r['dict']['tags']\n",
" #print (tags)\n",
" for point in r['points']:\n",
" point_file_path = os.path.join(r['path'], 'ckp-%s.0001.json' % point)\n",
" with open(point_file_path) as point_file:\n",
" point_data_raw = json.load(point_file)\n",
" #pprint (point_data_raw['choices']['env'])\n",
" characteristics_list = point_data_raw['characteristics_list']\n",
" num_repetitions = len(characteristics_list)\n",
" #platform = point_data_raw['features']['platform']['platform']['model']\n",
" if np.int64(point_data_raw['choices']['env'].get('CK_ENABLE_BATCH',-1))==1:\n",
" batch_enabled = True \n",
" batch_size = np.int64(point_data_raw['choices']['env'].get('CK_BATCH_SIZE',-1))\n",
" batch_count = np.int64(point_data_raw['choices']['env'].get('CK_BATCH_COUNT',-1))\n",
" else:\n",
" batch_enabled = False\n",
" batch_size = 1\n",
" batch_count = np.int64(point_data_raw['choices']['env'].get('CK_BATCH_COUNT',-1)) * \\\n",
" np.int64(point_data_raw['choices']['env'].get('CK_BATCH_SIZE',-1))\n",
"\n",
" characteristics = characteristics_list[0]\n",
" if accuracy:\n",
" data = [\n",
" {\n",
" 'model': tags[0],\n",
" 'backend':'cuda',\n",
" 'batch_size': batch_size,\n",
" 'batch_count': batch_count,\n",
" 'batch_enabled': batch_enabled,\n",
" 'image_height': image_height,\n",
" 'image_width': image_width,\n",
" 'num_reps':1,\n",
" # runtime characteristics\n",
" 'Recall': characteristics['run'].get('recall',0)*100,\n",
" 'mAP': characteristics['run'].get('mAP',0)*100,\n",
" 'mAP_large': characteristics['run']['metrics'].get('DetectionBoxes_Recall/AR@100 (large)', 0)*100,\n",
" 'mAP_medium': characteristics['run']['metrics'].get('DetectionBoxes_Recall/AR@100 (medium)', 0)*100,\n",
" 'mAP_small': characteristics['run']['metrics'].get('DetectionBoxes_Recall/AR@100 (small)', 0)*100,\n",
" }\n",
" ]\n",
"# print(data[0]['model'])\n",
" else: # performance\n",
" ####### this conversion is still needed because some of the result have the old naming convention\n",
" backend = 'default'\n",
" trt = point_data_raw['choices']['env'].get('CK_ENABLE_TENSORRT',0)\n",
" trt_dyn = point_data_raw['choices']['env'].get('CK_TENSORRT_DYNAMIC',0)\n",
" if trt_dyn == '1':\n",
" backend = 'tensorrt-dynamic'\n",
" elif trt == '1':\n",
" backend = 'tensorrt'\n",
" elif tags[0] == 'tensorrt' or tags[0] =='source-cuda':\n",
" backend = 'cuda'\n",
" elif tags[0] == 'tf-src-cpu' or tags[0] =='source-cpu':\n",
" backend = 'cpu'\n",
" elif tags[0] == 'tf-prebuild-cpu' or tags[0] == 'prebuilt-cpu':\n",
" backend = 'cpu-prebuilt'\n",
" else:\n",
" backend = tags[0]\n",
" model = tags[1]\n",
" data = [\n",
" {\n",
" 'model': model,\n",
" 'backend': backend,\n",
" 'batch_size': batch_size,\n",
" 'batch_count': batch_count,\n",
" 'batch_enabled': batch_enabled,\n",
" 'image_height': image_height,\n",
" 'image_width': image_width,\n",
" 'num_reps' : num_repetitions,\n",
" # statistical repetition\n",
" 'repetition_id': repetition_id,\n",
" # runtime characteristics\n",
" 'avg_fps': characteristics['run'].get('avg_fps', 'n/a')*batch_size,\n",
" 'avg_time_ms': characteristics['run']['avg_time_ms']/batch_size,\n",
" 'graph_load_time_ms': characteristics['run']['graph_load_time_s']*1e+3,\n",
" 'images_load_time_avg_ms': characteristics['run']['images_load_time_avg_s']*1e+3,\n",
" }\n",
" for (repetition_id, characteristics) in zip(range(num_repetitions), characteristics_list)\n",
" ]\n",
" index = [\n",
" 'model', 'backend', 'batch_size', 'batch_count', 'batch_enabled', 'image_height', 'image_width', 'num_reps'\n",
" ]\n",
" # Construct a DataFrame.\n",
" df = pd.DataFrame(data)\n",
" df = df.set_index(index)\n",
" # Append to the list of similarly constructed DataFrames.\n",
" dfs.append(df)\n",
" if dfs:\n",
" # Concatenate all thus constructed DataFrames (i.e. stack on top of each other).\n",
" result = pd.concat(dfs)\n",
" result.sort_index(ascending=True, inplace=True)\n",
" else:\n",
" # Construct a dummy DataFrame the success status of which can be safely checked.\n",
" result = pd.DataFrame(columns=['success?'])\n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#!ck recache repo\n",
"dfs = get_experimental_results(repo_uoa, accuracy=True)\n",
"dfs_perf = get_experimental_results(perf_docker_repo_uoa, accuracy=False)\n",
"dfs_perf_native = get_experimental_results(perf_native_repo_uoa, accuracy=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"display_in_full(dfs)\n",
"display_in_full(dfs_perf)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Plot the experimental data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Plot accuracy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"def plot_accuracy(df, groupby_level='batch_enabled',\n",
" accuracy_metric=['mAP','mAP_large','mAP_medium','mAP_small'],\n",
" save_fig=False, save_fig_name='accuracy.resizing.', resize_type=['internal','external'],\n",
" title='', figsize=[default_figwidth, 8], rot=90, colormap=cm.autumn):\n",
" # Bars.\n",
" df_bar = pd.DataFrame(\n",
" data=df[accuracy_metric].values, columns=accuracy_metric,\n",
" index=pd.MultiIndex.from_tuples(\n",
" tuples=[ (m,be,nr) for (m,b,bs,bc,be,ih,iw,nr) in df.index.values ],\n",
" names=[ 'model','batch_enabled','num_reps' ]\n",
" )\n",
" )\n",
"\n",
" # Plot.\n",
" mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean()\n",
" std = pd.Series()\n",
" axes = mean \\\n",
" .groupby(level=groupby_level) \\\n",
" .plot(yerr=std, kind='bar', grid=True, rot=rot,\n",
" figsize=figsize, width=default_barwidth, fontsize=default_fontsize, colormap=colormap)\n",
"\n",
" xlabel = 'Model'\n",
" xtics = df_bar.index.get_level_values('model').drop_duplicates()\n",
" ylabel = 'mAP %'\n",
" for count, ax in enumerate(axes):\n",
" # Title.\n",
" ax.set_title('Accuracy with %s resizing' % resize_type[count])\n",
" # X label.\n",
" ax.set_xlabel(xlabel)\n",
" # X ticks.\n",
" ax.set_xticklabels(xtics)\n",
" # Y axis.\n",
" ax.set_ylabel(ylabel)\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name+resize_type[count], save_fig_ext))\n",
" ax.figure.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight') \n",
"\n",
"plot_accuracy(dfs, save_fig=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Resizing layer in the network"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are two types of resizing: __fixed__ and __keep_aspect_ratio__.\n",
"\n",
"The first one takes as input the image and returns an image of fixed dimensions, changing from network to network.\n",
"Using this layer are the following networks:\n",
"\n",
" - 'ssd_mobilenet_v1_fpn_coco', 640*640\n",
" - 'faster_rcnn_nas_lowproposals_coco', 1200*1200\n",
" - 'faster_rcnn_nas', 1200*1200\n",
" - 'ssd_inception_v2_coco', 300*300\n",
" - 'ssd_mobilenet_v1_coco', 300*300\n",
" - 'ssd_mobilenet_v1_quantized_coco', 300*300\n",
" - 'ssd_resnet_50_fpn_coco', 640*640\n",
" - 'ssdlite_mobilenet_v2_coco', 300*300\n",
" - 'yolo v3', 416*416\n",
" \n",
"The second one behaviour is actually not completely clear, and have a minimum and maximum dimension of the output images. network using this layer are:\n",
"\n",
" - 'faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco', min: 600 max: 1024\n",
" - 'faster_rcnn_inception_resnet_v2_atrous_coco', min: 600 max: 1024\n",
" - 'faster_rcnn_resnet101_lowproposal_coco', min: 600 max: 1024\n",
" - 'faster_rcnn_resnet50_lowproposals_coco', min: 600 max: 1024\n",
" \n",
"From the analysis on the performance-accuracy benchmarks, it seems that the models using fixed resizing performs better than the one with the keep aspect ratio.\n",
"\n",
"YOLO-v3 is actually in the middle ground: its preprocessing is doing a fixed resize to 416 * 416, however the resize is done keeping the aspect ratio and padding, outside the graph. The graph doesn't perform any resizing, but takes 416 * 416 images as input."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_accuracy_compare_resizing(df_raw, resize_type=['internal','external'],\n",
" unstack_level='batch_enabled', groupby_level='batch_size',\n",
" accuracy_metric=['mAP','mAP_large','mAP_medium','mAP_small'], \n",
" save_fig=False, save_fig_name='accuracy.resizing.internal_vs_external',\n",
" title='', figsize=[default_figwidth, 8], rot=90):\n",
" # Bars.\n",
" df_bar = pd.DataFrame(\n",
" data=df_raw[accuracy_metric].values, columns=accuracy_metric,\n",
" index=pd.MultiIndex.from_tuples(\n",
" tuples=[ (m,be,bs,nr) for (m,b,bs,bc,be,ih,iw,nr) in df_raw.index.values ],\n",
" names=[ 'model','batch_enabled','batch_size','num_reps' ]\n",
" )\n",
" )\n",
" for index, row in df_bar.iterrows():\n",
" (model, batch_enabled, batch_size, num_reps) = index\n",
" if model == 'yolo-v3':\n",
" # batch_size and num_reps are both 1 for accuracy experiments.\n",
" df_bar.loc[(model, True, 1, 1)] = df_bar.loc[(model, False, 1, 1)][accuracy_metric]\n",
"\n",
" # Colormap. \n",
" colornames = [ 'lightcoral','cyan','indianred','darkturquoise','brown','cadetblue','darkred','steelblue' ]\n",
" colormap = mp.colors.ListedColormap(colornames, name='from_list', N=len(colornames))\n",
"\n",
" # Plot.\n",
" mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean().unstack(unstack_level)\n",
" std = df_bar.groupby(level=df_bar.index.names[:-1]).std().unstack(unstack_level)\n",
" axes = mean \\\n",
" .groupby(level=groupby_level) \\\n",
" .plot(yerr=std, kind='bar', grid=True, rot=rot,\n",
" figsize=figsize, width=default_barwidth, fontsize=default_fontsize, colormap=colormap)\n",
" \n",
" xlabel = 'Model'\n",
" ylabel = 'mAP %'\n",
" xtics = df_bar.index.get_level_values('model').drop_duplicates()\n",
" for count, ax in enumerate(axes):\n",
" # Title.\n",
" ax.set_title('Compare resizing: %s vs %s' % (resize_type[0], resize_type[1]))\n",
" # X label.\n",
" ax.set_xlabel(xlabel)\n",
" # X ticks.\n",
" ax.set_xticklabels(xtics)\n",
" # Y axis.\n",
" ax.set_ylabel(ylabel)\n",
" patches, labels = ax.get_legend_handles_labels()\n",
" # Legend.\n",
" labels = [x.strip('(') for x in labels]\n",
" labels = [x.strip(')') for x in labels]\n",
" ax.legend(patches, labels, loc='best', title='Metric, External resizing?')\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name, save_fig_ext))\n",
" ax.figure.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight') \n",
"\n",
"plot_accuracy_compare_resizing(dfs, save_fig=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Plot performance"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_performance_per_backend(df_raw, groupby_level='backend', unstack_level=['batch_size'],\n",
" performance_metric=['avg_fps','avg_time_ms','graph_load_time_ms','images_load_time_avg_ms'],\n",
" save_fig=False, save_fig_name='performance.backend.',\n",
" title=None, figsize=[default_figwidth, 8], rot=90, colormap=cm.autumn):\n",
" # Bars.\n",
" df_bar = pd.DataFrame(\n",
" data=df_raw[performance_metric].values, columns=performance_metric,\n",
" index=pd.MultiIndex.from_tuples(\n",
" tuples=[ (m,b,bs,be,nr) for (m,b,bs,bc,be,ih,iw,nr) in df_raw.index.values ],\n",
" names=[ 'model','backend','batch_size','batch_enabled','num_reps' ]\n",
" )\n",
" )\n",
"\n",
" # Plot.\n",
" mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean().unstack(unstack_level)\n",
" std = df_bar.groupby(level=df_bar.index.names[:-1]).std().unstack(unstack_level)\n",
" axes = mean \\\n",
" .groupby(level=groupby_level) \\\n",
" .plot(yerr=std, kind='bar', grid=True, rot=rot, legend=False,\n",
" figsize=figsize, width=default_barwidth, fontsize=default_fontsize, colormap=colormap)\n",
"\n",
" xlabel = 'Model'\n",
" xtics = df_bar.index.get_level_values('model').drop_duplicates()\n",
" ylabel = 'Images Per Second'\n",
" for num, ax in enumerate(axes):\n",
" # Title.\n",
" ax.set_title('TensorFlow with the \\'{}\\' Backend'.format(axes.keys().to_numpy().item(num)))\n",
" # X label.\n",
" ax.set_xlabel(xlabel)\n",
" # X ticks.\n",
" ax.set_xticklabels(xtics)\n",
" # Y axis.\n",
" ax.set_ylabel(ylabel)\n",
" # Legend.\n",
" patches, labels = ax.get_legend_handles_labels()\n",
" labels = [label[1] for label in [x.split(',') for x in labels]]\n",
" labels = [x.strip(')') for x in labels]\n",
" ax.legend(patches, labels, loc='best', title='Batch Size')\n",
" # Save figure.\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name+axes.keys().to_numpy().item(num), save_fig_ext))\n",
" ax.figure.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight')\n",
"\n",
"plot_performance_per_backend(dfs_perf, performance_metric=['avg_fps'], save_fig=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"def plot_performance_per_model(df_raw, groupby_level='model', unstack_level=['batch_size'],\n",
" performance_metric=['avg_fps','avg_time_ms','graph_load_time_ms','images_load_time_avg_ms'],\n",
" save_fig=False, save_fig_name='performance.model.',\n",
" title=None, figsize=[default_figwidth, 8], rot=0, colormap=cm.autumn):\n",
" # Bars.\n",
" df_bar = pd.DataFrame(\n",
" data=df_raw[performance_metric].values, columns=performance_metric,\n",
" index=pd.MultiIndex.from_tuples(\n",
" tuples=[ (m,b,bs,be,nr) for (m,b,bs,bc,be,ih,iw,nr) in df_raw.index.values ],\n",
" names=[ 'model','backend','batch_size','batch_enabled','num_reps' ] \n",
" )\n",
" )\n",
"\n",
" # Plot.\n",
" mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean().unstack(unstack_level)\n",
" std = df_bar.groupby(level=df_bar.index.names[:-1]).std().unstack(unstack_level)\n",
" axes = mean \\\n",
" .groupby(level=groupby_level) \\\n",
" .plot(yerr=std, kind='bar', grid=True, rot=rot,\n",
" figsize=figsize, width=default_barwidth, fontsize=default_fontsize, legend=False, colormap=colormap)\n",
"\n",
" xlabel = 'Backend'\n",
" xtics = df_bar.index.get_level_values('backend').drop_duplicates()\n",
" ylabel = 'Images Per Second'\n",
" for num, ax in enumerate(axes):\n",
" # Title.\n",
" ax.set_title(axes.keys().to_numpy().item(num))\n",
" # X label.\n",
" ax.set_xlabel(xlabel)\n",
" # X ticks.\n",
" ax.set_xticklabels(xtics)\n",
" # Y axis.\n",
" ax.set_ylabel(ylabel)\n",
" # Legend.\n",
" patches, labels = ax.get_legend_handles_labels()\n",
" labels = [label[1] for label in [x.split(',') for x in labels]]\n",
" labels = [x.strip(')') for x in labels]\n",
" ax.legend(patches, labels, loc='best', title='Batch Size')\n",
" # Save figure.\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name+axes.keys().to_numpy().item(num), save_fig_ext))\n",
" ax.figure.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight')\n",
"\n",
"plot_performance_per_model(dfs_perf, performance_metric=['avg_fps'], save_fig=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"def plot_performance_compare_backends(df_raw, backends=['cuda','tensorrt-dynamic'],\n",
" groupby_level='batch_enabled', unstack_level = ['backend','batch_size'],\n",
" performance_metric=['avg_fps','avg_time_ms','graph_load_time_ms','images_load_time_avg_ms'], \n",
" save_fig=False, save_fig_name='performance.backend.cuda_vs_tensorrt-dynamic',\n",
" title='', figsize=[default_figwidth, 8], rot=90):\n",
" # Bars.\n",
" df_bar = pd.DataFrame(\n",
" data=df_raw[performance_metric].values, columns=performance_metric,\n",
" index=pd.MultiIndex.from_tuples(\n",
" tuples=[ (m,b,bs,be,nr) for (m,b,bs,bc,be,ih,iw,nr) in df_raw.index.values ],\n",
" names=[ 'model','backend','batch_size','batch_enabled','num_reps' ]\n",
" )\n",
" )\n",
" df_bar = df_bar.query('backend in @backends')\n",
"\n",
" # Colormap.\n",
" colormap = mp.colors.ListedColormap(['darkred','steelblue'], name='from_list', N=12)\n",
" \n",
" # Plot.\n",
" mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean().unstack(unstack_level[1]).unstack(unstack_level[0])\n",
" std = df_bar.groupby(level=df_bar.index.names[:-1]).std().unstack(unstack_level[1]).unstack(unstack_level[0])\n",
" axes = mean \\\n",
" .groupby(level=groupby_level) \\\n",
" .plot(yerr=std, kind='bar', grid=True, rot=rot, legend=False,\n",
" figsize=figsize, width=default_barwidth, fontsize=default_fontsize, colormap=colormap)\n",
"\n",
" xlabel = 'Model'\n",
" xtics = df_bar.index.get_level_values('model').drop_duplicates()\n",
" ylabel = 'Images Per Second'\n",
" for num, ax in enumerate(axes):\n",
" # Title.\n",
" ax.set_title('Compare Backends: \\'%s\\' vs \\'%s\\'' % (backends[0], backends[1]))\n",
" # X label.\n",
" ax.set_xlabel(xlabel)\n",
" # X ticks.\n",
" ax.set_xticklabels(xtics)\n",
" # Y axis.\n",
" ax.set_ylabel(ylabel)\n",
" # Legend.\n",
" patches, labels = ax.get_legend_handles_labels()\n",
" labels = [label[2] for label in [x.split(',') for x in labels]]\n",
" labels = [x.strip(')') for x in labels]\n",
" ax.legend(patches[:2], labels[:2], loc='left', title='Backend')\n",
"\n",
" # Save figure.\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name+str(num), save_fig_ext))\n",
" ax.figure.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight')\n",
"\n",
"plot_performance_compare_backends(dfs_perf, performance_metric=['avg_fps'], save_fig=True)\n",
"plot_performance_compare_backends(dfs_perf, performance_metric=['avg_fps'], save_fig=True,\n",
" backends=['cpu-prebuilt','cpu'], save_fig_name='performance.backend.cpu-prebuilt_vs_cpu')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"def plot_performance_compare_docker_vs_native(df_docker, df_native,\n",
" groupby_level='batch_enabled', unstack_level=['batch_size','backend'],\n",
" save_fig=False, save_fig_name='performance.docker_vs_native',\n",
" title='', figsize=[default_figwidth, 8], rot=90):\n",
" # Bars.\n",
" df_docker = df_docker[df_docker.index.get_level_values('batch_enabled').isin([True])]\n",
" df_bar = pd.merge(\n",
" df_docker, df_native,\n",
" how='inner', suffixes=('_docker', '_native'),\n",
" on=[ 'model', 'backend', 'batch_size', 'batch_enabled', 'num_reps' ])\n",
" df_bar = df_bar[['avg_fps_docker', 'avg_fps_native']]\n",
" df_bar['avg_fps_norm'] = df_bar['avg_fps_docker'] / df_bar['avg_fps_native']\n",
" df_bar = df_bar[['avg_fps_norm']]\n",
"\n",
" # Colormap.\n",
" reds = ['red', 'indianred', 'brown', 'firebrick', 'maroon', 'darkred']\n",
" yellows = ['cornsilk', 'lemonchiffon', 'palegoldenrod', 'khaki', 'yellow', 'gold']\n",
" greens = ['palegreen', 'lightgreen', 'limegreen', 'green', 'forestgreen', 'darkgreen']\n",
" blues = ['cornflowerblue', 'royalblue', 'mediumblue', 'blue', 'navy', 'midnightblue']\n",
" purples = ['orchid', 'fuchsia', 'mediumorchid', 'darkviolet', 'purple', 'indigo']\n",
" colornames = reds + yellows + greens + blues + purples\n",
" colormap = mp.colors.ListedColormap(colornames, name='from_list', N=len(colornames))\n",
" \n",
" # Plot.\n",
" mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean().unstack(unstack_level[1]).unstack(unstack_level[0])\n",
" std = df_bar.groupby(level=df_bar.index.names[:-1]).std().unstack(unstack_level[1]).unstack(unstack_level[0])\n",
" axes = mean \\\n",
" .groupby(level=groupby_level) \\\n",
" .plot(yerr=std, kind='bar', grid=True, rot=rot, ylim=[0.0,1.201],\n",
" figsize=figsize, width=default_barwidth, fontsize=default_fontsize, colormap=colormap)\n",
" \n",
" title = 'Compare Docker vs Native Performance (Normalized to Native) on 5 Models from the Pareto Frontier'\n",
" xtics = df_bar.groupby(level=df_bar.index.names[:-1]).median().index.get_level_values('model').drop_duplicates()\n",
" xlabel = 'Model'\n",
" ylabel = ''\n",
" for num, ax in enumerate(axes):\n",
" # Title.\n",
" ax.set_title(title)\n",
" # X label.\n",
" ax.set_xlabel(xlabel)\n",
" # X ticks.\n",
" ax.set_xticklabels(xtics)\n",
" # Y axis.\n",
" ax.set_ylabel(ylabel)\n",
" # Shrink current axis by 20%\n",
" box = ax.get_position()\n",
" ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])\n",
" patches, labels = ax.get_legend_handles_labels()\n",
" # Legend.\n",
" labels = [x.strip('(avg_fps_norm,') for x in labels]\n",
" labels = [x.strip(')') for x in labels]\n",
" ax.legend(patches, labels, title='Backend, Batch Size', loc='center left', bbox_to_anchor=(1, 0.5),fontsize=default_fontsize)\n",
" # Put a legend to the right of the current axis.\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name, save_fig_ext))\n",
" ax.figure.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight') \n",
" \n",
"plot_performance_compare_docker_vs_native(dfs_perf, dfs_perf_native, rot=45, save_fig=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Plot exploration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Plotting settings.\n",
"model_to_color = { \n",
" 'rcnn-inception-resnet-v2-lowproposals' : 'gold',\n",
" 'rcnn-inception-v2' : 'goldenrod',\n",
" 'rcnn-nas-lowproposals' : 'red',\n",
" 'rcnn-resnet101-lowproposals' : 'brown',\n",
" 'rcnn-resnet50-lowproposals' : 'orangered',\n",
" \n",
" 'ssd-mobilenet-v1-fpn' : 'cyan',\n",
" 'ssd-inception-v2' : 'cadetblue',\n",
" 'ssd-mobilenet-v1-non-quantized-mlperf' : 'deepskyblue',\n",
" 'ssd-mobilenet-v1-quantized-mlperf' : 'royalblue',\n",
" 'ssd-resnet50-fpn' : 'navy',\n",
" \n",
" 'ssdlite-mobilenet-v2' : 'violet',\n",
" 'yolo-v3' : 'gray'\n",
"}\n",
"\n",
"backend_to_marker = {\n",
" 'tensorrt' : '1',\n",
" 'tensorrt-dynamic' : '2',\n",
" 'cuda' : '3',\n",
" 'cpu' : '4',\n",
" 'cpu-prebuilt' : '+'\n",
"}\n",
"\n",
"resize_to_marker = {\n",
" 'no-resize' : 'x',\n",
" 'model-resize' : '*'\n",
"}\n",
"\n",
"bs_to_size = {\n",
" 1 : 1.0,\n",
" 2 : 1.5,\n",
" 4 : 2.0,\n",
" 8 : 2.5,\n",
" 16 : 3.0,\n",
" 32 : 3.5\n",
"}\n",
"\n",
"### not used anymore, left in case should change the policy\n",
"# model_to_real_name = { \n",
"# 'ssd-mobilenet-v1-fpn' : 'ssd_mobilenet_v1_fpn_coco',\n",
"# 'rcnn-inception-resnet-v2-lowproposals' : 'faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco',\n",
"# 'rcnn-nas-lowproposals' : 'faster_rcnn_nas_lowproposals_coco',\n",
"# 'rcnn-resnet101-lowproposals' : 'faster_rcnn_resnet101_lowproposals_coco',\n",
"# 'rcnn-inception-v2' : 'faster_rcnn_inception_resnet_v2_atrous_coco',\n",
"# 'rcnn-nas-non-lowproposal' : 'faster_rcnn_nas',\n",
"# 'ssd-inception-v2' : 'ssd_inception_v2_coco',\n",
"# 'ssd-mobilenet-v1-non-quantized-mlperf' : 'ssd_mobilenet_v1_coco',\n",
"# 'ssd-mobilenet-v1-quantized-mlperf' : 'ssd_mobilenet_v1_quantized_coco',\n",
"# 'ssd-resnet50-fpn' : 'ssd_resnet_50_fpn_coco',\n",
"# 'ssdlite-mobilenet-v2' : 'ssdlite_mobilenet_v2_coco',\n",
"# 'rcnn-resnet50-lowproposals' : 'faster_rcnn_resnet50_lowproposals_coco',\n",
"# 'yolo-v3' : 'yolo v3'\n",
"# }\n",
"\n",
"import matplotlib.lines as mlines\n",
"mark1 = mlines.Line2D([], [], color='black', marker='1', linestyle='None',\n",
" markersize=5, label='tensorrt')\n",
"mark2 = mlines.Line2D([], [], color='black', marker='2', linestyle='None',\n",
" markersize=5, label='tensorrt-dynamic')\n",
"mark3 = mlines.Line2D([], [], color='black', marker='3', linestyle='None',\n",
" markersize=5, label='cuda')\n",
"mark4 = mlines.Line2D([], [], color='black', marker='4', linestyle='None',\n",
" markersize=5, label=' cpu')\n",
"mark5 = mlines.Line2D([], [], color='black', marker='+', linestyle='None',\n",
" markersize=5, label=' cpu-prebuilt')\n",
"handles2 = [ mark1, mark2, mark3, mark4, mark5 ]\n",
"\n",
"#mark1 = mlines.Line2D([], [], color='black', marker='x', linestyle='None',\n",
"# markersize=5, label='no resize')\n",
"#mark2 = mlines.Line2D([], [], color='black', marker='*', linestyle='None',\n",
"# markersize=5, label='model resize')\n",
"#handles2 = [ mark1,mark2 ]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def merge_performance_accuracy(df_performance, df_accuracy, \n",
" performance_metric='avg_fps', accuracy_metric='mAP', ideal=False):\n",
" df = df_performance[[performance_metric]]\n",
" \n",
" for metric in accuracy_metric:\n",
" accuracy_list = []\n",
" for index, row in df.iterrows():\n",
" (model, backend, batch_size, batch_count, batch_enabled, image_height, image_width, num_reps) = index\n",
" if ideal:\n",
" accuracy = df_accuracy.loc[(model, 'cuda', 1, 5000, False, image_height, image_width, 1)][metric]\n",
" else:\n",
" resize = 'no-resize' if batch_size == 1 else 'model-resize'\n",
" if resize == 'no-resize' or model == 'yolo-v3':\n",
" accuracy = df_accuracy.loc[(model, 'cuda', 1, 5000, False, image_height, image_width, 1)][metric]\n",
" else:\n",
" accuracy = df_accuracy.loc[(model, 'cuda', 1, 5000, True, image_height, image_width, 1)][metric]\n",
" accuracy_list.append(accuracy)\n",
"\n",
" # Assign to the value of accuracy_metric.\n",
" kwargs = { metric : accuracy_list }\n",
" df = df.assign(**kwargs)\n",
" return df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def finalize_plot(ax, xmin, xmax, xstep, ymin, ymax, ystep, save_fig, save_fig_name, accuracy_metric):\n",
" # X axis.\n",
" xlabel = 'Images Per Second'\n",
" ax.set_xlabel(xlabel)\n",
" ax.set_xlim(xmin, xmax)\n",
" ax.set_xticks(np.arange(xmin, xmax, xstep))\n",
" for xtick in ax.xaxis.get_major_ticks(): xtick.label.set_fontsize(5)\n",
" # Y axis.\n",
" ylabel = accuracy_metric + ' %'\n",
" ax.set_ylabel(ylabel)\n",
" ax.set_ylim(ymin, ymax)\n",
" ax.set_yticks(np.arange(ymin, ymax, ystep))\n",
" for ytick in ax.yaxis.get_major_ticks(): ytick.label.set_fontsize(5)\n",
" # Legend.\n",
" handles = [ \n",
" mp.patches.Patch(color=color, label=model)\n",
" for (model, color) in sorted(model_to_color.items())\n",
" ]\n",
" handles += handles2\n",
" plt.legend(title='', handles=handles[::-1], loc='best', prop={'size': 5})\n",
" \n",
" # Show with grid on.\n",
" plt.grid(True)\n",
" fig1 = plt.gcf()\n",
" plt.show()\n",
" plt.draw()\n",
" \n",
" # Save figure.\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name, save_fig_ext))\n",
" fig1.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight') "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_dse(ideal=False,\n",
" performance_metric='avg_fps', accuracy_metric=['mAP'],\n",
" xmin=0.00, xmax=85.01, xstep=5,\n",
" ymin=18.00, ymax=46.01, ystep=4,\n",
" save_fig=False, save_fig_name='dse.full'):\n",
" fig = plt.figure(figsize=(8,4), dpi=default_figdpi)\n",
" ax = fig.gca()\n",
" if ideal:\n",
" save_fig_name += '.ideal'\n",
" ax.set_title('Full Space Exploration with Ideal Accuracy')\n",
" else:\n",
" ax.set_title('Full Space Exploration with Measured Accuracy')\n",
" df_performance_accuracy = merge_performance_accuracy(\n",
" dfs_perf, dfs, performance_metric=performance_metric, accuracy_metric=accuracy_metric, ideal=ideal)\n",
"\n",
" df = df_performance_accuracy\n",
" df = df.groupby(level=df.index.names[:-1]).mean()\n",
" #display_in_full(df)\n",
" accuracy_metric=accuracy_metric[0]\n",
" for index, row in df.iterrows():\n",
" (model, backend, batch_size, batch_count, batch_enabled, image_height, image_width) = index\n",
" performance = row[performance_metric]\n",
" accuracy = row[accuracy_metric]\n",
"\n",
" # Mark Pareto-optimal points.\n",
" is_on_pareto = True\n",
" for index1, row1 in df.iterrows():\n",
" is_no_slower = row1[performance_metric] >= row[performance_metric]\n",
" is_no_less_accurate = row1[accuracy_metric] >= row[accuracy_metric]\n",
" is_faster = row1[performance_metric] > row[performance_metric]\n",
" is_more_accurate = row1[accuracy_metric] > row[accuracy_metric]\n",
" if ((is_faster and is_no_less_accurate) or (is_more_accurate and is_no_slower)):\n",
" is_on_pareto = False\n",
" break\n",
"\n",
" # Select marker, color and size.\n",
" marker = backend_to_marker[backend]\n",
" color = model_to_color[model]\n",
" size = 2 + 4*bs_to_size[batch_size]\n",
"\n",
" # Plot.\n",
" ax.plot(performance, accuracy, marker, markerfacecolor=color, markeredgecolor=color, markersize=size)\n",
"\n",
" # Mark Pareto-optimal points with scaled black pluses.\n",
" if is_on_pareto:\n",
" ax.plot(performance, accuracy, 'o', markersize=4,markerfacecolor='black', markeredgecolor='black')\n",
"\n",
" finalize_plot(ax, xmin, xmax, xstep, ymin, ymax, ystep, save_fig, save_fig_name, accuracy_metric)\n",
" \n",
"# NB: The accuracy metric has to be an array of 1 element for this function.\n",
"plot_dse (ideal=False, ystep=2, save_fig=True, save_fig_name='dse.full')\n",
"plot_dse (ideal=True, ystep=2, save_fig=True, save_fig_name='dse.full')\n",
"plot_dse (ideal=True, accuracy_metric=['mAP_small'], ymin=0, ymax=22.01, ystep=2, save_fig=True, save_fig_name='dse.full.small')\n",
"plot_dse (ideal=True, accuracy_metric=['mAP_large'], ymin=30, ymax=78.01, ystep=4, save_fig=True, save_fig_name='dse.full.large')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_fastest(ideal=False,\n",
" performance_metric='avg_fps', accuracy_metric=['mAP','mAP_large','mAP_medium','mAP_small'],\n",
" xmin=0.0, xmax=85.01, xstep=5,\n",
" ymin=[22,42,16,0], ymax=[46.01,78.01,56.01,22.01], ystep=[2,4,4,2],\n",
" labelsize=5, ticksize=4,\n",
" save_fig=False, save_fig_name='dse.fastest'):\n",
" \n",
" fig,ax = plt.subplots(2,2,sharex='col')\n",
" if ideal:\n",
" save_fig_name += '.ideal'\n",
" fig.suptitle('Fastest Configuration with Ideal Accuracy')\n",
" else:\n",
" fig.suptitle('Fastest Configuration with Measured Accuracy')\n",
" df_performance_accuracy = merge_performance_accuracy(\n",
" dfs_perf, dfs, performance_metric=performance_metric, accuracy_metric=accuracy_metric, ideal=ideal)\n",
" df_full = df_performance_accuracy\n",
" for num,metric in enumerate (accuracy_metric):\n",
" pos = (int(num/2),num%2)\n",
" df = df_full[[metric,performance_metric]]\n",
" df = df.groupby(level=df.index.names[:-1]).mean()\n",
" df = df.groupby(level=df.index.names[:-4]).max()\n",
" #display_in_full(df)\n",
" points_to_plot=[]\n",
" for index, row in df.iterrows():\n",
" (model,backend, batch_size) = index\n",
" performance = row[performance_metric]\n",
" accuracy = row[metric]\n",
" plot = True\n",
" # Analyze points of the same model.\n",
" for index1, row1 in df.iterrows():\n",
" if index == index1:\n",
" continue\n",
" if index1[0] != model:\n",
" continue\n",
" is_faster = row1[performance_metric] > row[performance_metric]\n",
" if is_faster:\n",
" plot = False\n",
" continue\n",
"\n",
" if plot: # no faster points have been found with the same model.\n",
" # Select marker, color and size.\n",
" marker = backend_to_marker[backend]\n",
" color = model_to_color[model]\n",
" size = 2 + 4*bs_to_size[batch_size]\n",
" # Plot.\n",
" ax[pos[0],pos[1]].plot(performance, accuracy, marker, markerfacecolor=color, markeredgecolor=color, markersize=size)\n",
" ax[pos[0],pos[1]].grid(True)\n",
" \n",
" # X axis.\n",
" xlabel = 'Images Per Second'\n",
" ax[1,0].set_xlabel(xlabel, fontsize=labelsize)\n",
" ax[1,0].set_xlim(xmin, xmax)\n",
" ax[1,0].set_xticks(np.arange(xmin, xmax, xstep))\n",
" for xtick in ax[1,0].xaxis.get_major_ticks(): xtick.label.set_fontsize(ticksize)\n",
" ax[1,1].set_xlabel(xlabel, fontsize=labelsize)\n",
" ax[1,1].set_xlim(xmin, xmax)\n",
" ax[1,1].set_xticks(np.arange(xmin, xmax, xstep))\n",
" for xtick in ax[1,1].xaxis.get_major_ticks(): xtick.label.set_fontsize(ticksize)\n",
" \n",
" # Y axis.\n",
" ylabel = accuracy_metric[0]+' %'\n",
" ax[0,0].set_ylabel(ylabel, fontsize=labelsize)\n",
" ax[0,0].set_ylim(ymin[0], ymax[0])\n",
" ax[0,0].set_yticks(np.arange(ymin[0], ymax[0], ystep[0]))\n",
" for ytick in ax[0,0].yaxis.get_major_ticks(): ytick.label.set_fontsize(ticksize)\n",
" \n",
" ylabel = accuracy_metric[1]+' %'\n",
" ax[0,1].set_ylabel(ylabel, fontsize=labelsize)\n",
" ax[0,1].set_ylim(ymin[1], ymax[1])\n",
" ax[0,1].set_yticks(np.arange(ymin[1], ymax[1], ystep[1]))\n",
" for ytick in ax[0,1].yaxis.get_major_ticks(): ytick.label.set_fontsize(ticksize)\n",
" \n",
" ylabel = accuracy_metric[2]+' %'\n",
" ax[1,0].set_ylabel(ylabel, fontsize=labelsize)\n",
" ax[1,0].set_ylim(ymin[2], ymax[2])\n",
" ax[1,0].set_yticks(np.arange(ymin[2], ymax[2], ystep[2]))\n",
" for ytick in ax[1,0].yaxis.get_major_ticks(): ytick.label.set_fontsize(ticksize)\n",
" \n",
" ylabel = accuracy_metric[3]+' %'\n",
" ax[1,1].set_ylabel(ylabel, fontsize=labelsize)\n",
" ax[1,1].set_ylim(ymin[3], ymax[3])\n",
" ax[1,1].set_yticks(np.arange(ymin[3], ymax[3], ystep[3]))\n",
" for ytick in ax[1,1].yaxis.get_major_ticks(): ytick.label.set_fontsize(ticksize)\n",
" \n",
" # Legend.\n",
" handles = [ \n",
" mp.patches.Patch(color=color, label=model)\n",
" for (model, color) in sorted(model_to_color.items())\n",
" ]\n",
" handles += handles2\n",
" fig.legend(title='', handles=handles[::-1], \n",
" loc='center right',\n",
" borderaxespad=0.1,\n",
" fontsize=4)\n",
" plt.subplots_adjust(right=0.78)\n",
" # Show with grid on.\n",
" plt.grid(True)\n",
" fig1 = plt.gcf()\n",
" plt.show()\n",
" plt.draw()\n",
" \n",
" # Save figure.\n",
" if save_fig:\n",
" save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name, save_fig_ext))\n",
" fig1.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight') \n",
" \n",
"plot_fastest(ideal=True, save_fig=True)"
]
}
],
"metadata": {
"celltoolbar": "Raw Cell Format",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 1
}