{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# [cknowledge.org/ai](http://cknowledge.org/ai): Crowdsourcing benchmarking and optimization of AI" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* [Reproducible Quality-Efficient Systems Tournaments](http://cknowledge.org/request) ([ReQuEST initiative](http://cknowledge.org/request.html#organizers))\n", "* [AI artifacts](http://cknowledge.org/ai-artifacts) (cTuning foundation)\n", "* [Android app](https://play.google.com/store/apps/details?id=openscience.crowdsource.video.experiments) (dividiti)\n", "* [Desktop app](https://github.com/dividiti/ck-crowdsource-dnn-optimization) (dividiti)\n", "* [CK-Caffe](https://github.com/dividiti/ck-caffe) (Berkeley)\n", "* [CK-Caffe2](https://github.com/ctuning/ck-caffe2) (Facebook)\n", "* [CK-CNTK](https://github.com/ctuning/ck-cntk) (Microsoft)\n", "* [CK-KaNN](https://github.com/ctuning/ck-kann) (Kalray)\n", "* [CK-MVNC](https://github.com/ctuning/ck-mvnc) (Movidius / Intel)\n", "* [CK-MXNet](https://github.com/ctuning/ck-mxnet) (Apache)\n", "* [CK-NNTest](https://github.com/ctuning/ck-nntest) (cTuning foundation)\n", "* [CK-TensorFlow](https://github.com/ctuning/ck-tensorflow) (Google)\n", "* [CK-TensorRT](https://github.com/ctuning/ck-tensorrt) (NVIDIA)\n", "* etc." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# [dividiti](http://dividiti.com)'s submission to [ReQuEST @ ASPLOS'18](http://cknowledge.org/request-cfp-asplos2018.html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Table of Contents" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. [Overview](#overview)\n", "1. [Platforms](#platforms)\n", " 1. [Linaro HiKey960](#platforms_hikey) (**\"HiKey\"**)\n", " 1. [Firefly RK3399](#platforms_firefly) (**\"Firefly\"**)\n", "1. [Experimental data](#data) [for developers]\n", "1. [Data wrangling code](#code) [for developers]\n", "1. [Experiments on Hikey](#experiments_hikey)\n", " 1. [TensorFlow](#experiments_tensorflow_hikey)\n", " 1. [ArmCL](#experiments_armcl_hikey)\n", " 1. [ArmCL vs. TensorFlow](#experiments_armcl_tensorflow_hikey)\n", "1. [Experiments on Firefly](#experiments_firefly)\n", " 1. [TensorFlow](#experiments_tensorflow_firefly)\n", " 1. [ArmCL](#experiments_armcl_firefly)\n", " 1. [ArmCL vs. TensorFlow](#experiments_armcl_tensorflow_firefly)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This Jupyter Notebook studies performance (execution time) vs accuracy (top1 / top5) using the [Arm Compute Library](https://github.com/ARM-software/ComputeLibrary) on two development platforms:\n", "- [Linaro HiKey960](https://www.96boards.org/product/hikey960/);\n", "- [Firefly RK3399](http://en.t-firefly.com/index.php/product/rk3399.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Platforms" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Linaro HiKey960" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " - Chip:\n", " - [HiSilicon Kirin 960](http://www.hisilicon.com/en/Solutions/Kirin)\n", " - CPU (\"performance\" / \"big\"):\n", " - ARM® Cortex®-A73;\n", " - Max clock 2362 MHz;\n", " - 4 cores;\n", " - CPU (\"efficiency\" / \"LITTLE\"):\n", " - ARM® Cortex®-A53;\n", " - Max clock 1844 MHz;\n", " - 4 cores;\n", " - GPU:\n", " - ARM® Mali™ G71 architecture;\n", " - Max clock 1037 MHz;\n", " - 8 cores;\n", " - OpenCL driver (`hikey962`: `instr=1,clexperimental=1,softjobpatch`):\n", "```\n", "$ ck run program:tool-print-opencl-devices | grep \"version:\"\n", "OpenCL 2.0 v1.r6p0-01rel0.24c5f5e966f2b7f1f19b91d6f32ff53e\n", "```\n", "\n", " - RAM:\n", " - LPDDR4 SDRAM;\n", " - 3 GB;\n", "\n", " - BSP:\n", " - Debian Stretch (9) Linux\n", "```\n", "$ uname -a\n", "Linux hikey962 4.4.74-00216-g10816f6 #3 SMP PREEMPT Thu Jul 6 14:38:42 BST 2017 aarch64 GNU/Linux\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hikey_model = 'HiKey960\\x00'\n", "hikey_name = 'Linaro HiKey960'\n", "hikey_id = 'hikey-960'\n", "hikey_gpu = 'Mali-G71 MP8'\n", "hikey_gpu_mhz = 807" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Firefly RK3399" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " - Chip:\n", " - [Rockchip RK3399](http://rockchip.wikidot.com/rk3399)\n", " - CPU (\"big\"):\n", " - ARM® Cortex®-A72 architecture\n", " - Max clock 1800 MHz;\n", " - 2 cores;\n", " - CPU (\"LITTLE\"):\n", " - ARM® Cortex®-A53 architecture;\n", " - Max clock 1416 MHz;\n", " - 4 cores;\n", " - GPU:\n", " - ARM® Mali™-T860 architecture;\n", " - Max clock 800 MHz;\n", " - 4 cores;\n", " - OpenCL driver:\n", "```\n", "$ ck run program:tool-print-opencl-devices | grep \"version:\"\n", "v1.r13p0-00rel0-git(a4271c9).31ba04af2d3c01618138bef3aed66c2c\n", "```\n", "\n", " - RAM:\n", " - Samsung dual-channel DDR3;\n", " - 4 GB (8 GB swap);\n", " - BSP:\n", " - [Firefly-rk3399_xubuntu1604_201711301130.7z](https://drive.google.com/drive/u/0/folders/1lbaR7XVyHT4SnXkJ2ybj5YXAzAjDBWfT)\n", "```\n", "$ cat /etc/lsb-release\n", "DISTRIB_ID=Ubuntu\n", "DISTRIB_RELEASE=16.04\n", "DISTRIB_CODENAME=xenial\n", "DISTRIB_DESCRIPTION=\"Ubuntu 16.04.4 LTS\"\n", "$ uname -a\n", "Linux firefly 4.4.77 #554 SMP Thu Nov 30 11:30:11 HKT 2017 aarch64 aarch64 aarch64 GNU/Linux\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "firefly_model = 'Rockchip RK3399 Firefly Board (Linux Opensource)\\x00'\n", "firefly_name = 'Firefly RK3399'\n", "firefly_id = 'firefly'\n", "firefly_gpu = 'Mali-T860 MP4'\n", "firefly_gpu_mhz = 800" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Platform mappings" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_to_id = {\n", " firefly_model : firefly_id,\n", " hikey_model : hikey_id\n", "}\n", "id_to_name = {\n", " firefly_id : firefly_name,\n", " hikey_id : hikey_name\n", "}\n", "id_to_gpu = {\n", " firefly_id : firefly_gpu,\n", " hikey_id : hikey_gpu\n", "}\n", "id_to_gpu_mhz = {\n", " firefly_id : firefly_gpu_mhz,\n", " hikey_id : hikey_gpu_mhz\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Convolution method mapping" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "convolution_method_to_name = [\n", " 'gemm',\n", " 'direct',\n", " 'winograd'\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Get the experimental data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The experimental data can be downloaded and registered with CK as described below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL experiments on HiKey" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ArmCL accuracy experiments on 50,000 images" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ wget https://www.dropbox.com/s/tm1qlom7ehfbe0w/ck-request-asplos18-mobilenets-armcl-opencl-accuracy-50000.zip\n", "$ ck add repo --zip=ck-request-asplos18-mobilenets-armcl-opencl-accuracy-50000.zip\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "armcl_accuracy_50000_repo_uoa = 'ck-request-asplos18-mobilenets-armcl-opencl-accuracy-50000'\n", "!ck list $armcl_accuracy_50000_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ArmCL accuracy experiments on 500 images" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ wget https://www.dropbox.com/s/wqqchrhr36skm9y/ck-request-asplos18-mobilenets-armcl-opencl-accuracy-500.zip\n", "$ ck add repo --zip=ck-request-asplos18-mobilenets-armcl-opencl-accuracy-500.zip\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "armcl_accuracy_500_repo_uoa = 'ck-request-asplos18-mobilenets-armcl-opencl-accuracy-500'\n", "!ck list $armcl_accuracy_500_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ArmCL performance (latency) experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ wget https://www.dropbox.com/s/wm3ahhm20y7g04k/ck-request-asplos18-mobilenets-armcl-opencl-performance.zip\n", "$ ck add repo --zip=ck-request-asplos18-mobilenets-armcl-opencl-performance.zip\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "armcl_performance_repo_uoa = 'ck-request-asplos18-mobilenets-armcl-opencl-performance'\n", "!ck list $armcl_performance_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow experiments on HiKey" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TensorFlow accuracy experiments on 50000 images" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ wget https://www.dropbox.com/s/ro5txjz9n396s0t/ck-request-asplos18-mobilenets-tensorflow-accuracy-50000.zip\n", "$ ck add repo --zip=ck-request-asplos18-mobilenets-tensorflow-accuracy-50000.zip\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tensorflow_accuracy_50000_repo_uoa = 'ck-request-asplos18-mobilenets-tensorflow-accuracy-50000'\n", "!ck list $tensorflow_accuracy_50000_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TensorFlow accuracy experiments on 500 images" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ wget https://www.dropbox.com/s/k0xhhb7owwvyfgu/ck-request-asplos18-mobilenets-tensorflow-accuracy-500.zip\n", "$ ck add repo --zip=ck-request-asplos18-mobilenets-tensorflow-accuracy-500.zip\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tensorflow_accuracy_500_repo_uoa = 'ck-request-asplos18-mobilenets-tensorflow-accuracy-500'\n", "!ck list $tensorflow_accuracy_500_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TensorFlow performance (latency) experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ wget https://www.dropbox.com/s/1fagdonfaqsdfou/ck-request-asplos18-mobilenets-tensorflow-performance.zip\n", "$ ck add repo --zip=ck-request-asplos18-mobilenets-tensorflow-performance.zip\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tensorflow_performance_repo_uoa = 'ck-request-asplos18-mobilenets-tensorflow-performance'\n", "!ck list $tensorflow_performance_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow experiments on Firefly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TensorFlow accuracy experiments on 500 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "firefly_tensorflow_accuracy_500_repo_uoa = 'ck-request-asplos18-mobilenets-tensorflow-accuracy-500-firefly'\n", "!ck list $firefly_tensorflow_accuracy_500_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### TensorFlow performance (latency) experiments" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "firefly_tensorflow_performance_repo_uoa = 'ck-request-asplos18-mobilenets-tensorflow-performance-firefly'\n", "!ck list $firefly_tensorflow_performance_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL experiments on Firefly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ArmCL performance (latency) experiments" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "firefly_armcl_performance_repo_uoa = 'ck-request-asplos18-mobilenets-armcl-opencl-performance-firefly'\n", "!ck list $firefly_armcl_performance_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ArmCL accuracy experiments on 500 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "firefly_armcl_accuracy_500_repo_uoa = 'ck-request-asplos18-mobilenets-armcl-opencl-accuracy-500-firefly'\n", "!ck list $armcl_accuracy_500_repo_uoa:experiment:* | sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Data wrangling code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**NB:** Please ignore this section if you are not interested in re-running or modifying this notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Includes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Standard" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import sys\n", "import json\n", "import re" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Scientific" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If some of the scientific packages are missing, please install them using:\n", "```\n", "# pip install jupyter pandas numpy matplotlib\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import IPython as ip\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib as mp\n", "import seaborn as sb" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print ('IPython version: %s' % ip.__version__)\n", "print ('Pandas version: %s' % pd.__version__)\n", "print ('NumPy version: %s' % np.__version__)\n", "print ('Matplotlib version: %s' % mp.__version__)\n", "print ('Seaborn version: %s' % sb.__version__)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image, display\n", "def display_in_full(df):\n", " pd.options.display.max_columns = len(df.columns)\n", " pd.options.display.max_rows = len(df.index)\n", " display(df)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "from matplotlib import cm\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "default_colormap = cm.autumn\n", "default_fontsize = 16\n", "default_barwidth = 0.8\n", "default_figwidth = 24\n", "default_figheight = 3\n", "default_figdpi = 200\n", "default_figsize = [default_figwidth, default_figheight]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if mp.__version__[0]=='2': mp.style.use('classic')\n", "mp.rcParams['figure.max_open_warning'] = 200\n", "mp.rcParams['figure.dpi'] = default_figdpi\n", "mp.rcParams['font.size'] = default_fontsize\n", "mp.rcParams['legend.fontsize'] = 'medium'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Collective Knowledge" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If CK is not installed, please install it using:\n", "```\n", "# pip install ck\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import ck.kernel as ck\n", "print ('CK version: %s' % ck.__version__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Access experimental data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_experimental_results(repo_uoa, tags='explore-mobilenets-performance', accuracy=False,\n", " module_uoa='experiment', _library=None, _platform=None):\n", " r = ck.access({'action':'search', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'tags':tags})\n", " if r['return']>0:\n", " print('Error: %s' % r['error'])\n", " exit(1)\n", " experiments = r['lst']\n", "\n", " dfs = []\n", " for experiment in experiments:\n", " data_uoa = experiment['data_uoa']\n", " r = ck.access({'action':'list_points', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'data_uoa':data_uoa})\n", " if r['return']>0:\n", " print('Error: %s' % r['error'])\n", " exit(1)\n", " # Mapping of expected library tags to reader-friendly names.\n", " tag_to_name = {\n", " # ArmCL tags on HiKey.\n", " '17.12-48bc34ea' : 'armcl-17.12',\n", " '18.01-f45d5a9b' : 'armcl-18.01',\n", " '18.03-e40997bb' : 'armcl-18.03',\n", " 'request-d8f69c13' : 'armcl-dv/dt', # armcl-18.03+\n", " '18.05-b3a371bc' : 'armcl-18.05',\n", " '18.05-0acd60ed-request' : 'armcl-dv/dt', # armcl-18.05+\n", " # ArmCL tags on Firefly.\n", " '17.12-48bc34e' : 'armcl-17.12',\n", " '18.01-f45d5a9' : 'armcl-18.01',\n", " '18.03-e40997b' : 'armcl-18.03',\n", " '18.05-b3a371b' : 'armcl-18.05',\n", " # TensorFlow tags.\n", " 'tensorflow-1.7' : 'tensorflow-1.7',\n", " 'tensorflow-1.8' : 'tensorflow-1.8',\n", " }\n", " \n", " # Library.\n", " library_tags = [ tag for tag in r['dict']['tags'] if tag in tag_to_name.keys() ]\n", " if len(library_tags)==1:\n", " library = tag_to_name[library_tags[0]]\n", " else:\n", " print('[Warning] Bad library tags. Skipping experiment with tags:')\n", " print(r['dict']['tags'])\n", " continue\n", " if _library and _library!=library: continue\n", " # For each point. \n", " for point in r['points']:\n", " point_file_path = os.path.join(r['path'], 'ckp-%s.0001.json' % point)\n", " with open(point_file_path) as point_file:\n", " point_data_raw = json.load(point_file)\n", " characteristics_list = point_data_raw['characteristics_list']\n", " num_repetitions = len(characteristics_list)\n", " platform = model_to_id[point_data_raw['features']['platform']['platform']['model']]\n", " if _platform and _platform!=platform: continue\n", " env = point_data_raw['choices']['env']\n", " batch_size = np.int64(env.get('CK_BATCH_SIZE',-1))\n", " batch_count = np.int64(env.get('CK_BATCH_COUNT',-1))\n", " convolution_method_default = 1 if platform==hikey_id else 0\n", " convolution_method = convolution_method_to_name[np.int64(env.get('CK_CONVOLUTION_METHOD_HINT',env.get('CK_CONVOLUTION_METHOD', convolution_method_default)))]\n", " if library.startswith('tensorflow-'):\n", " multiplier = np.float64(env.get('CK_ENV_TENSORFLOW_MODEL_MOBILENET_MULTIPLIER',-1))\n", " resolution = np.int64(env.get('CK_ENV_TENSORFLOW_MODEL_MOBILENET_RESOLUTION',-1))\n", " else:\n", " multiplier = np.float64(env.get('CK_ENV_MOBILENET_WIDTH_MULTIPLIER',-1))\n", " resolution = np.int64(env.get('CK_ENV_MOBILENET_RESOLUTION',-1))\n", " model = 'v1-%.2f-%d' % (multiplier, resolution)\n", " if accuracy:\n", " data = [\n", " {\n", " # features\n", " 'platform': platform,\n", " 'library': library,\n", " # choices\n", " 'model': model,\n", " 'batch_size': batch_size,\n", " 'batch_count': batch_count,\n", " 'convolution_method': convolution_method,\n", " 'resolution': resolution,\n", " 'multiplier': multiplier,\n", " # statistical repetition\n", " 'repetition_id': repetition_id,\n", " # runtime characteristics\n", " 'success?': characteristics['run'].get('run_success', 'n/a'),\n", " 'accuracy_top1': characteristics['run'].get('accuracy_top1', 0),\n", " 'accuracy_top5': characteristics['run'].get('accuracy_top5', 0),\n", " 'frame_predictions': characteristics['run'].get('frame_predictions', []),\n", "# # recompute accuracy from frame_predictions (was incorrectly recorded in early experiments)\n", "# 'accuracy_top1_': len([\n", "# prediction for prediction in characteristics['run'].get('frame_predictions', [])\n", "# if prediction['accuracy_top1']=='yes'\n", "# ]) / np.float64(batch_count),\n", "# 'accuracy_top5_': len([\n", "# prediction for prediction in characteristics['run'].get('frame_predictions', [])\n", "# if prediction['accuracy_top5']=='yes'\n", "# ]) / np.float64(batch_count)\n", " }\n", " for (repetition_id, characteristics) in zip(range(num_repetitions), characteristics_list)\n", " ]\n", " else: # performance\n", " data = [\n", " {\n", " # features\n", " 'platform': platform,\n", " 'library': library,\n", " # choices\n", " 'model': model,\n", " 'batch_size': batch_size,\n", " 'batch_count': batch_count,\n", " 'convolution_method': convolution_method,\n", " 'resolution': resolution,\n", " 'multiplier': multiplier,\n", " # statistical repetition\n", " 'repetition_id': repetition_id,\n", " # runtime characteristics\n", " 'success?': characteristics['run'].get('run_success', 'n/a'),\n", " 'time_avg_ms': characteristics['run']['prediction_time_avg_s']*1e+3,\n", " 'time_total_ms': characteristics['run']['prediction_time_total_s']*1e+3,\n", " }\n", " for (repetition_id, characteristics) in zip(range(num_repetitions), characteristics_list)\n", " ]\n", " index = [\n", " 'platform', 'library', 'model', 'multiplier', 'resolution', 'batch_size', 'convolution_method', 'repetition_id'\n", " ]\n", " # Construct a DataFrame.\n", " df = pd.DataFrame(data)\n", " df = df.set_index(index)\n", " # Append to the list of similarly constructed DataFrames.\n", " dfs.append(df)\n", " if dfs:\n", " # Concatenate all thus constructed DataFrames (i.e. stack on top of each other).\n", " result = pd.concat(dfs)\n", " result.sort_index(ascending=True, inplace=True)\n", " else:\n", " # Construct a dummy DataFrame the success status of which can be safely checked.\n", " result = pd.DataFrame(columns=['success?'])\n", " return result" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Merge performance and accuracy data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Return a new DataFrame with only the performance and accuracy metrics.\n", "def merge_performance_accuracy(df_performance, df_accuracy, \n", " reference_platform=None, reference_lib=None, reference_convolution_method='direct',\n", " performance_metric='time_avg_ms', accuracy_metric='accuracy_top1'):\n", " df = df_performance[[performance_metric]]\n", " accuracy_list = []\n", " for index, row in df.iterrows():\n", " (platform, lib, model, multiplier, resolution, batch_size, convolution_method) = index\n", " if reference_platform: platform = reference_platform\n", " try:\n", " accuracy = df_accuracy.loc[(platform, lib, model, multiplier, resolution, batch_size, convolution_method)][accuracy_metric]\n", " except:\n", " if reference_lib: lib = reference_lib\n", " convolution_method = reference_convolution_method\n", " accuracy = df_accuracy.loc[(platform, lib, model, multiplier, resolution, batch_size, convolution_method)][accuracy_metric]\n", " accuracy_list.append(accuracy)\n", " df = df.assign(accuracy_top1=accuracy_list) # FIXME: assign to the value of accuracy_metric\n", " return df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot experimental data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot performance (bar plot)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot_performance(df_raw, groupby_level='convolution_method', platform_id=hikey_id, gpu_mhz=id_to_gpu_mhz[hikey_id],\n", " performance_metric='time_avg_ms', title=None, figsize=None, rot=90):\n", " df_bar = pd.DataFrame(\n", " data=df_raw[performance_metric].values, columns=['ms'],\n", " index=pd.MultiIndex.from_tuples(\n", " tuples=[ (l,m[3:],c,r) for (p,l,m,_,_,_,c,r) in df_raw.index.values ],\n", " names=[ 'library', 'model', 'convolution_method', 'repetition_id' ]\n", " )\n", " )\n", " df_bar.columns.names = ['time']\n", " if groupby_level=='convolution_method':\n", " unstack_level = 'library'\n", " xlabel='(Model [channel multiplier - input resolution], Convolution Method)'\n", " colormap = cm.autumn\n", " elif groupby_level=='library':\n", " unstack_level = 'convolution_method'\n", " xlabel='(Library, Model [channel multiplier - input resolution])'\n", " colormap = cm.summer\n", " # Set default style.\n", " ylabel='Image recognition time (ms)'\n", " if not title: title = '%s (GPU: %s @ %d MHz)' % (id_to_name[platform_id], id_to_gpu[platform_id], gpu_mhz)\n", " if not figsize: figsize = [default_figwidth, 8]\n", " # Plot \n", " mean = df_bar.groupby(level=df_bar.index.names[:-1]).mean().unstack(unstack_level)\n", " std = df_bar.groupby(level=df_bar.index.names[:-1]).std().unstack(unstack_level)\n", " axes = mean.groupby(level=groupby_level) \\\n", " .plot(yerr=std, kind='bar', grid=True, width=0.8, rot=rot, figsize=figsize,\n", " fontsize=default_fontsize, colormap=colormap)\n", " for ax in axes:\n", " # Title.\n", " ax.set_title(title)\n", " # X label.\n", " ax.set_xlabel(xlabel)\n", " # Y axis.\n", " ax.set_ylabel(ylabel)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot performance (violin plot)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot_performance_violin(df_raw, groupby_level='convolution_method', platform_id=hikey_id, gpu_mhz=id_to_gpu_mhz[hikey_id],\n", " performance_metric='time_avg_ms', title=None, figsize=None, fontscale=1.75):\n", " df_violin = pd.DataFrame(\n", " data=df_raw[performance_metric].values, columns=['ms'],\n", " index=pd.MultiIndex.from_tuples(\n", " tuples=[ (l,m[3:],c,r) for (p,l,m,_,_,_,c,r) in df_raw.index.values ],\n", " names=[ 'library', 'model', 'convolution_method', 'repetition_id' ]\n", " )\n", " )\n", " if groupby_level=='convolution_method':\n", " df_violin = df_violin.swaplevel('convolution_method', 'library')\n", " hue_level = 'library'\n", " palette = 'autumn'\n", " elif groupby_level=='library':\n", " hue_level = 'convolution_method'\n", " palette = 'summer'\n", " num_model_values = len(df_violin.index.get_level_values(level='model').unique())\n", " # Set default style.\n", " xlabel='Model [channel multiplier - input resolution]'\n", " ylabel='Image recognition time (ms)'\n", " if not title: title = '%s (GPU: %s @ %d MHz)' % (id_to_name[platform_id], id_to_gpu[platform_id], gpu_mhz)\n", " if not figsize: figsize = (num_model_values*1.5, 12)\n", " sb.set_style('whitegrid')\n", " sb.set_palette(palette)\n", " # For each unique groupby value.\n", " groupby_values = df_violin.index.get_level_values(level=groupby_level).unique()\n", " for groupby_value in groupby_values:\n", " fig = plt.figure(figsize=figsize, dpi=default_figdpi)\n", " ax = fig.gca()\n", " df_violin_loc = df_violin.loc[groupby_value].reset_index()\n", " sb.violinplot(ax=ax, data=df_violin_loc, x='model', y='ms', hue=hue_level,\n", " fontscale=fontscale, inner='point', split=False, saturation=0.8)\n", " # Title.\n", " groupby_title = '%s: %s=%s' % (title, groupby_level, groupby_value)\n", " ax.set_title(groupby_title, fontsize=default_fontsize*fontscale)\n", " # X axis.\n", " ax.set_xlabel(xlabel, fontsize=default_fontsize*fontscale)\n", " # Y axis.\n", " ystep = 10\n", " ymin = np.int64(df_violin_loc['ms'].min())\n", " ymax = np.int64(df_violin_loc['ms'].max()) // ystep * ystep + ystep + 1\n", " ax.set_ylim([ymin, ymax])\n", " ax.set_yticks(range(0, ymax, ystep))\n", " ax.set_ylabel(ylabel, fontsize=default_fontsize*fontscale)\n", " # Vertical lines between groups of violins.\n", " for x in ax.get_xticks():\n", " ax.vlines(x=x+0.5, ymin=0, ymax=ymax, linestyles='dotted', colors='purple')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot performance vs. accuracy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot(df_performance_accuracy, libs=None, platform_id=hikey_id, gpu_mhz=id_to_gpu_mhz[hikey_id],\n", " performance_metric='time_avg_ms', accuracy_metric='accuracy_top1',\n", " xmin=0.0, xmax=75.1, xstep=5.0, ymin=0.4, ymax=0.751, ystep=0.05,\n", " title=None, save_fig=False, save_fig_name='mobilenets-default'):\n", " fig = plt.figure(figsize=(8,4), dpi=default_figdpi)\n", " ax = fig.gca()\n", " \n", " lib_to_color = { \n", " 'armcl-17.12' : 'red',\n", " 'armcl-18.01' : 'yellow',\n", " 'armcl-18.03' : 'orange',\n", " 'armcl-dv/dt' : 'green',\n", " 'armcl-18.05' : 'purple',\n", " 'tensorflow-1.7' : 'cyan',\n", " 'tensorflow-1.8' : 'blue',\n", " }\n", " multiplier_to_marker = {\n", " 'gemm' : { 1.00 : '*', 0.75 : 'D', 0.50: 'v', 0.25 : '8' },\n", " 'direct' : { 1.00 : 'p', 0.75 : 's', 0.50: '^', 0.25 : 'o' },\n", " 'winograd' : { 1.00 : 'P', 0.75 : 'X', 0.50: '<', 0.25 : '.' },\n", " }\n", "\n", " if libs==None: libs = df_performance_accuracy.index.levels[1].tolist()\n", " df = df_performance_accuracy.loc[platform_id].loc[libs]\n", " for index, row in df.iterrows():\n", " (lib, model, multiplier, resolution, batch_size, convolution_method) = index\n", " performance = row[performance_metric]\n", " accuracy = row[accuracy_metric]\n", " \n", " # Mark Pareto-optimal points.\n", " is_on_pareto = True\n", " for index1, row1 in df.iterrows():\n", " is_faster = row1[performance_metric] < row[performance_metric]\n", " is_no_less_accurate = row1[accuracy_metric] >= row[accuracy_metric]\n", " if is_faster and is_no_less_accurate:\n", " is_on_pareto = False\n", " break\n", "\n", " # GEMM-based convolution should be exactly the same in '18.03' and 'dv/dt', so plot\n", " # the minimum execution time of '18.03' and 'dv/dt' as '18.03'.\n", " if 'armcl-dv/dt' in libs and convolution_method=='gemm' and (lib=='armcl-dv/dt' or lib=='armcl-18.03'):\n", " performance_dv_dt = df.loc[('armcl-dv/dt', model, multiplier, resolution, batch_size, convolution_method)][performance_metric]\n", " performance_18_03 = df.loc[('armcl-18.03', model, multiplier, resolution, batch_size, convolution_method)][performance_metric]\n", " if lib=='armcl-18.03':\n", " if (performance_dv_dt < performance_18_03):\n", " continue\n", " if lib=='armcl-dv/dt':\n", " if (performance_dv_dt < performance_18_03):\n", " lib = 'armcl-18.03' # change color\n", " else:\n", " continue\n", " \n", " # Select size, color and marker.\n", " size = resolution / 16\n", " color = lib_to_color[lib]\n", " marker = multiplier_to_marker[convolution_method][multiplier]\n", "\n", " # Plot.\n", " ax.plot(performance, accuracy, marker, markerfacecolor=color, markersize=size)\n", "\n", " # Mark Pareto-optimal points with scaled black pluses.\n", " if is_on_pareto:\n", " ax.plot(performance, accuracy, 'k+', markersize=size)\n", "\n", " # Title.\n", " if not title: title = '%s (GPU: %s @ %d MHz)' % (id_to_name[platform_id], id_to_gpu[platform_id], gpu_mhz)\n", " ax.set_title(title)\n", " # X axis.\n", " xlabel='Image recognition time (ms)' if performance_metric=='time_avg_ms' else ''\n", " ax.set_xlabel(xlabel)\n", " ax.set_xlim(xmin, xmax)\n", " ax.set_xticks(np.arange(xmin, xmax, xstep))\n", " for xtick in ax.xaxis.get_major_ticks(): xtick.label.set_fontsize(12)\n", " # Y axis.\n", " ylabel='Image recognition accuracy (top %s)' % accuracy_metric[-1]\n", " ax.set_ylabel(ylabel)\n", " ax.set_ylim(ymin, ymax)\n", " ax.set_yticks(np.arange(ymin, ymax, ystep))\n", " for ytick in ax.yaxis.get_major_ticks(): ytick.label.set_fontsize(12)\n", " # Legend.\n", " handles = [ \n", " mp.patches.Patch(color=color, label=lib)\n", " for (lib, color) in sorted(lib_to_color.items())\n", " if lib in libs\n", " ]\n", " plt.legend(title='Library', handles=handles[::-1], loc='lower right')\n", " # Show with grid on.\n", " plt.grid(True)\n", " plt.show()\n", " # Save figure.\n", " if save_fig:\n", " save_fig_path = os.path.join(save_fig_dir, '%s.%s' % (save_fig_name, save_fig_ext))\n", " plt.savefig(save_fig_path, dpi=default_figdpi, bbox_inches='tight')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set options for saving figures/tables" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_paper_dir(module_uoa='dissemination.publication', data_uoa='08da9685582866a0'):\n", " r = ck.access({'action':'find','module_uoa':module_uoa,'data_uoa':data_uoa})\n", " if r['return']>0:\n", " print('Warning: %s' % r['error'])\n", " paper_dir = os.path.curdir\n", " else:\n", " paper_dir = r['path']\n", " return paper_dir" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "save_fig_ext = 'pdf'\n", "save_fig_dir = os.path.join(get_paper_dir(), 'figures')\n", "if not os.path.exists(save_fig_dir):\n", " os.makedirs(save_fig_dir)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "save_tab = False\n", "save_tab_ext = 'tex'\n", "save_tab_dir = os.path.join(get_paper_dir(), 'tables')\n", "if not os.path.exists(save_tab_dir):\n", " os.makedirs(save_tab_dir)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## TensorFlow experiments on HiKey" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow performance (latency)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_tensorflow_performance_raw = get_experimental_results(repo_uoa=tensorflow_performance_repo_uoa,\n", " tags='explore-mobilenets-performance', accuracy=False)\n", "# Take the minimum execution time out of several repetitions.\n", "df_tensorflow_performance = \\\n", " df_tensorflow_performance_raw.groupby(level=df_tensorflow_performance_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_tensorflow_performance)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by convolution method" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_tensorflow_performance_raw, platform_id=hikey_id, groupby_level='convolution_method')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_tensorflow_performance_raw, platform_id=hikey_id, groupby_level='convolution_method')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by library" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_tensorflow_performance_raw, platform_id=hikey_id, groupby_level='library')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_tensorflow_performance_raw, platform_id=hikey_id, groupby_level='library')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow accuracy on 500 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_tensorflow_accuracy_500_raw = get_experimental_results(repo_uoa=tensorflow_accuracy_500_repo_uoa,\n", " tags='explore-mobilenets-accuracy', accuracy=True)\n", "# Extract frame predictions.\n", "df_tensorflow_predictions_500 = df_tensorflow_accuracy_500_raw[['frame_predictions']]\n", "# Reduce the repetition_id index dimension (only 1 repetition anyway).\n", "df_tensorflow_accuracy_500 = \\\n", " df_tensorflow_accuracy_500_raw[['accuracy_top1', 'accuracy_top5']] \\\n", " .groupby(level=df_tensorflow_accuracy_500_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_tensorflow_accuracy_500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow accuracy on 50,000 images (measured)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_tensorflow_accuracy_50000_raw = get_experimental_results(repo_uoa=tensorflow_accuracy_50000_repo_uoa,\n", " tags='explore-mobilenets-accuracy', accuracy=True)\n", "# Extract frame predictions.\n", "df_tensorflow_predictions_50000 = df_tensorflow_accuracy_50000_raw[['frame_predictions']]\n", "# Reduce the repetition_id index dimension (only 1 repetition anyway).\n", "df_tensorflow_accuracy_50000 = \\\n", " df_tensorflow_accuracy_50000_raw[['accuracy_top1', 'accuracy_top5']] \\\n", " .groupby(level=df_tensorflow_accuracy_50000_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_tensorflow_accuracy_50000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow accuracy on 50,000 images (claimed)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TensorFlow accuracy reported with the MobileNets pretrained weights shared on 2017_06_14. Copied from:\n", "# https://github.com/tensorflow/models/blob/1630da3434974e9ad5a0b6d887ac716a97ce03d3/research/slim/nets/mobilenet_v1.md#pre-trained-models\n", "tensorflow_accuracy_50000_table = {\n", " 'v1-1.00-224':[569, 4.24, 70.7, 89.5],\n", " 'v1-1.00-192':[418, 4.24, 69.3, 88.9],\n", " 'v1-1.00-160':[291, 4.24, 67.2, 87.5],\n", " 'v1-1.00-128':[186, 4.24, 64.1, 85.3],\n", " 'v1-0.75-224':[317, 2.59, 68.4, 88.2],\n", " 'v1-0.75-192':[233, 2.59, 67.4, 87.3],\n", " 'v1-0.75-160':[162, 2.59, 65.2, 86.1],\n", " 'v1-0.75-128':[104, 2.59, 61.8, 83.6],\n", " 'v1-0.50-224':[150, 1.34, 64.0, 85.4],\n", " 'v1-0.50-192':[110, 1.34, 62.1, 84.0],\n", " 'v1-0.50-160':[77, 1.34, 59.9, 82.5],\n", " 'v1-0.50-128':[49, 1.34, 56.2, 79.6],\n", " 'v1-0.25-224':[41, 0.47, 50.6, 75.0],\n", " 'v1-0.25-192':[34, 0.47, 49.0, 73.6],\n", " 'v1-0.25-160':[21, 0.47, 46.0, 70.7],\n", " 'v1-0.25-128':[14, 0.47, 41.3, 66.2],\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_tensorflow_accuracy_50000_claimed = pd.DataFrame(\n", " index=['MACs (million)', 'Parameters (million)', 'accuracy_top1 (%)', 'accuracy_top5 (%)'],\n", " data=tensorflow_accuracy_50000_table,\n", ").T.sort_index()\n", "accuracy_top1 = df_tensorflow_accuracy_50000_claimed['accuracy_top1 (%)']/100\n", "accuracy_top5 = df_tensorflow_accuracy_50000_claimed['accuracy_top5 (%)']/100\n", "df_tensorflow_accuracy_50000_claimed = df_tensorflow_accuracy_50000_claimed.assign(accuracy_top1=accuracy_top1)\n", "df_tensorflow_accuracy_50000_claimed = df_tensorflow_accuracy_50000_claimed.assign(accuracy_top5=accuracy_top5)\n", "df_tensorflow_accuracy_50000_claimed.index = df_tensorflow_accuracy_50000.index\n", "display_in_full(df_tensorflow_accuracy_50000_claimed)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Diff measured as the fraction of correctly predicted images.\n", "df_tensorflow_accuracy_50000_diff = \\\n", " df_tensorflow_accuracy_50000_claimed[['accuracy_top1', 'accuracy_top5']] - \\\n", " df_tensorflow_accuracy_50000[['accuracy_top1', 'accuracy_top5']]\n", "display_in_full(df_tensorflow_accuracy_50000_diff)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Diff measured as the number of mispredicted images.\n", "df_tensorflow_accuracy_50000_diff_mispredicted = (df_tensorflow_accuracy_50000_diff) * 50000\n", "df_tensorflow_accuracy_50000_diff_mispredicted.columns = ['mispredicted_top1', 'mispredicted_top5']\n", "display_in_full(df_tensorflow_accuracy_50000_diff_mispredicted)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## ArmCL experiments on HiKey" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL performance (latency)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_performance_raw = get_experimental_results(repo_uoa=armcl_performance_repo_uoa,\n", " tags='explore-mobilenets-performance', accuracy=False)\n", "# Take the minimum execution time out of several repetitions.\n", "df_armcl_performance = df_armcl_performance_raw.groupby(level=df_armcl_performance_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_armcl_performance)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by convolution method" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_armcl_performance_raw, platform_id=hikey_id, groupby_level='convolution_method')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "plot_performance_violin(df_armcl_performance_raw, platform_id=hikey_id, groupby_level='convolution_method')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by library" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_armcl_performance_raw, platform_id=hikey_id, groupby_level='library')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_armcl_performance_raw, platform_id=hikey_id, groupby_level='library')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL accuracy on 500 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_accuracy_500_raw = get_experimental_results(repo_uoa=armcl_accuracy_500_repo_uoa,\n", " tags='explore-mobilenets-accuracy', accuracy=True)\n", "# Extract frame predictionsdf_armcl_accuracy_500_raw\n", "df_armcl_predictions_500 = df_armcl_accuracy_500_raw[['frame_predictions']]\n", "# Reduce the repetition_id index dimension (only 1 repetition anyway).\n", "df_armcl_accuracy_500 = \\\n", " df_armcl_accuracy_500_raw[['accuracy_top1', 'accuracy_top5']] \\\n", " .groupby(level=df_armcl_accuracy_500_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_armcl_accuracy_500)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Identical accuracy for \"18.03\" and \"dv/dt\".\n", "(df_armcl_accuracy_500.loc[hikey_id,'armcl-18.03'] - df_armcl_accuracy_500.loc[hikey_id,'armcl-dv/dt'] == 0).all()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Identical accuracy for \"18.03\" and \"18.01\".\n", "(df_armcl_accuracy_500.loc[hikey_id,'armcl-18.03'] - df_armcl_accuracy_500.loc[hikey_id,'armcl-18.01'] == 0).all()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_accuracy_500.loc[hikey_id,'armcl-18.03'] - df_armcl_accuracy_500.loc[hikey_id,'armcl-17.12']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TODO: Outline into a function for comparing ArmCL and TensorFlow predictions.\n", "df_armcl_predictions = df_armcl_predictions_500\n", "df_tensorflow_predictions = df_tensorflow_predictions_500\n", "\n", "tensorflow_lib = 'tensorflow-1.7'\n", "tensorflow_convolution_method = 'direct'\n", "\n", "for index, row in df_armcl_predictions.iterrows():\n", " (platform, lib, model, multiplier, resolution, batch_size, convolution_method, repetition_id) = index\n", " # For now, only check mispredictions for '18.03' and 'v1-1.00-224'.\n", " if not lib=='armcl-18.03' or not model=='v1-1.00-224': continue\n", " tensorflow_index = (platform, tensorflow_lib, model, multiplier, resolution, batch_size, tensorflow_convolution_method, repetition_id)\n", " # Extract frame predictions.\n", " armcl_predictions = row['frame_predictions']\n", " tensorflow_predictions = df_tensorflow_predictions.loc[tensorflow_index]['frame_predictions']\n", " # At the very minimum, the frame predictions should be of the same length.\n", " if len(armcl_predictions) != len(tensorflow_predictions):\n", " print('[Warning] ArmCL and TensorFlow predictions have different length! Skipping...')\n", " continue\n", " # Iterate over the frame predictions.\n", " for (armcl_prediction, tensorflow_prediction) in zip(armcl_predictions, tensorflow_predictions):\n", " if(armcl_prediction['accuracy_top1'] != tensorflow_prediction['accuracy_top1']):\n", " print(index)\n", " print('ArmCL: '+str(armcl_prediction))\n", " print('TensorFlow: '+str(tensorflow_prediction))\n", " print('')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL accuracy on 50,000 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_accuracy_50000_raw = get_experimental_results(repo_uoa=armcl_accuracy_50000_repo_uoa,\n", " tags='explore-mobilenets-accuracy', accuracy=True)\n", "# Extract frame predictions.\n", "df_armcl_predictions_50000 = df_armcl_accuracy_50000_raw[['frame_predictions']]\n", "# Reduce the repetition_id index dimension (only 1 repetition anyway).\n", "df_armcl_accuracy_50000 = \\\n", " df_armcl_accuracy_50000_raw[['accuracy_top1', 'accuracy_top5']] \\\n", " .groupby(level=df_armcl_accuracy_50000_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_armcl_accuracy_50000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot top 1 accuracy on 50,000 images (using the 'dv/dt' fork as reference lib) vs. performance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "accuracy_metric = 'accuracy_top1'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_performance_accuracy_50000 = merge_performance_accuracy(df_armcl_performance, df_armcl_accuracy_50000,\n", " reference_lib='armcl-dv/dt',\n", " reference_convolution_method='direct')\n", "display_in_full(df_armcl_performance_accuracy_50000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Only \"18.03\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_armcl_performance_accuracy_50000, libs=['armcl-18.03'], accuracy_metric=accuracy_metric,\n", " save_fig_name='%s-%s-50000-18_03' % (hikey_id, accuracy_metric+'_'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"dv/dt\" vs. \"18.03\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_armcl_performance_accuracy_50000, libs=['armcl-18.03','armcl-dv/dt'], accuracy_metric=accuracy_metric,\n", " save_fig_name='%s-%s-50000-dv_dt__18_03' % (hikey_id, accuracy_metric+'_'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"dv/dt\" vs. \"18.03\" vs. \"18.01\" vs. \"17.12\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_armcl_performance_accuracy_50000, accuracy_metric=accuracy_metric,\n", " save_fig_name='%s-%s-50000-dv_dt__18_03__18_01__17_12' % (hikey_id, accuracy_metric+'_'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot top 1 accuracy on 500 images (using the 'dv/dt' fork as the reference lib) vs. performance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_performance_accuracy_500 = merge_performance_accuracy(df_armcl_performance, df_armcl_accuracy_500,\n", " reference_lib='armcl-dv/dt',\n", " reference_convolution_method='direct')\n", "display_in_full(df_armcl_performance_accuracy_500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"dv/dt\" vs. \"18.03\" vs. \"18.01\" vs. \"17.12\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_armcl_performance_accuracy_500, accuracy_metric=accuracy_metric,\n", " save_fig_name='%s-%s-500-dv_dt__18_03__18_01__17_12' % (hikey_id, accuracy_metric+'_'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot top 1 accuracy on 500 images vs. performance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_performance_accuracy_500 = merge_performance_accuracy(df_armcl_performance, df_armcl_accuracy_500)\n", "display_in_full(df_armcl_performance_accuracy_500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"dv/dt\" vs. \"18.03\" vs. \"18.01\" vs. \"17.12\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_armcl_performance_accuracy_500, accuracy_metric=accuracy_metric,\n", " save_fig_name='%s-%s-500-dv_dt__18_03__18_01__17_12' % (hikey_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## ArmCL vs. TensorFlow on HiKey" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_accuracy_50000 = pd.DataFrame(\n", " data=[\n", " df_armcl_accuracy_50000['accuracy_top1'].values,\n", " df_tensorflow_accuracy_50000['accuracy_top1'].values,\n", " df_tensorflow_accuracy_50000_claimed['accuracy_top1'].values, \n", " ],\n", " index=[\n", " 'ArmCL 18.03 (measured)',\n", " 'TensorFlow 1.7 (measured)',\n", " 'TensorFlow 1.x (claimed)',\n", " ],\n", " columns=df_tensorflow_accuracy_50000_claimed.index.get_level_values(level='model').values\n", ").T.sort_index(ascending=False)\n", "# df_accuracy_50000.index.name = 'model'\n", "if save_tab:\n", " save_tab_name = 'accuracy_top1-50000'\n", " save_tab_path = os.path.join(save_tab_dir, '%s.%s' % (save_tab_name, save_tab_ext))\n", " with open(save_tab_path, 'w') as f: f.write(df_accuracy_50000.to_latex())\n", "display_in_full(df_accuracy_50000)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_performance = pd.concat([df_armcl_performance, df_tensorflow_performance])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot top 1 accuracy on 500 images vs. performance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_accuracy_500 = pd.concat([df_armcl_accuracy_500, df_tensorflow_accuracy_500])\n", "df_performance_accuracy_500 = merge_performance_accuracy(df_performance, df_accuracy_500)\n", "plot(df_performance_accuracy_500, accuracy_metric=accuracy_metric, save_fig=True,\n", " save_fig_name='%s-%s-500-dv_dt__18_03__18_01__17_12__tf' % (hikey_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot top 1 accuracy on 50,000 images vs. performance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_accuracy_50000 = pd.concat([df_armcl_accuracy_50000, df_tensorflow_accuracy_50000])\n", "df_performance_accuracy_50000 = merge_performance_accuracy(df_performance, df_accuracy_50000,\n", " reference_lib='armcl-dv/dt',\n", " reference_convolution_method='direct')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"dv/dt\" vs. \"18.03\" vs. \"18.01\" vs. \"17.12\" vs. \"tf\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "plot(df_performance_accuracy_50000, accuracy_metric=accuracy_metric, save_fig=True,\n", " save_fig_name='%s-%s-50000-dv_dt__18_03__18_01__17_12__tf' % (hikey_id, accuracy_metric+'_'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"dv/dt\" vs. \"18.03\" vs. \"17.12\" vs. \"tf\" (no \"18.01\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_performance_accuracy_50000, libs=['armcl-17.12','armcl-18.03','armcl-dv/dt','tensorflow-1.7'],\n", " accuracy_metric=accuracy_metric,\n", " save_fig_name='%s-%s-50000-dv_dt__18_03__17_12__tf' % (hikey_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## TensorFlow experiments on Firefly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow accuracy on 500 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_tensorflow_accuracy_500_raw = get_experimental_results(repo_uoa=firefly_tensorflow_accuracy_500_repo_uoa,\n", " tags='explore-mobilenets-accuracy', accuracy=True)\n", "# Extract frame predictions.\n", "df_firefly_tensorflow_predictions_500 = df_firefly_tensorflow_accuracy_500_raw[['frame_predictions']]\n", "# Reduce the repetition_id index dimension (only 1 repetition anyway).\n", "df_firefly_tensorflow_accuracy_500 = \\\n", " df_firefly_tensorflow_accuracy_500_raw[['accuracy_top1', 'accuracy_top5']] \\\n", " .groupby(level=df_firefly_tensorflow_accuracy_500_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_firefly_tensorflow_accuracy_500)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Check whether TensorFlow accuracy on Firefly is the same as on HiKey. (It's not!)\n", "df_firefly_tensorflow_accuracy_500.loc[firefly_id].loc['tensorflow-1.7'] - \\\n", "df_tensorflow_accuracy_500.loc[hikey_id].loc['tensorflow-1.7']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorFlow performance (latency)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_tensorflow_performance_raw = get_experimental_results(repo_uoa=firefly_tensorflow_performance_repo_uoa,\n", " tags='explore-mobilenets-performance', accuracy=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Take the minimum execution time out of several repetitions.\n", "df_firefly_tensorflow_performance_min = \\\n", " df_firefly_tensorflow_performance_raw.groupby(level=df_firefly_tensorflow_performance_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "# display_in_full(df_firefly_tensorflow_performance_min)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Take the maximum execution time out of several repetitions.\n", "df_firefly_tensorflow_performance_max = \\\n", " df_firefly_tensorflow_performance_raw.groupby(level=df_firefly_tensorflow_performance_raw.index.names[:-1]).max()\n", "# Set 'convolution_method' to 'gemm' for all rows to reuse the available plotting functionality.\n", "df_firefly_tensorflow_performance_max.index = \\\n", " df_firefly_tensorflow_performance_max.index \\\n", " .set_levels(levels=['gemm']*df_firefly_tensorflow_performance_max.index.size, level='convolution_method')\n", "# Display all rows and columns.\n", "# display_in_full(df_firefly_tensorflow_performance_max)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"tf-1.7\" vs \"tf-1.8\" (min/max)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_tensorflow_performance = pd.concat([df_firefly_tensorflow_performance_min, df_firefly_tensorflow_performance_max])\n", "# TODO: Use df_firefly_armcl_accuracy_500 and df_firefly_tensorflow_accuracy_500.\n", "df_accuracy_500 = pd.concat([df_armcl_accuracy_500, df_tensorflow_accuracy_500])\n", "df_firefly_performance_accuracy_500 = merge_performance_accuracy(df_firefly_tensorflow_performance, df_accuracy_500,\n", " reference_platform=hikey_id,\n", " reference_lib='tensorflow-1.7')\n", "plot(df_firefly_performance_accuracy_500, accuracy_metric=accuracy_metric, platform_id=firefly_id,\n", " xmin=0, xmax=150.1, xstep=10, save_fig_name='%s-%s-500-tf-min_max' % (firefly_id, accuracy_metric))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# plot(df_firefly_performance_accuracy_500, platform_id=firefly_id, title=firefly_name,\n", "# xmin=10., xmax=190.1, xstep=10, ymin=0.35, ymax=.801,\n", "# accuracy_metric=accuracy_metric, save_fig_name='%s-%s-500-tf-min_max-complete' % (firefly_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### \"tf-1.7\" vs \"tf-1.8\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_performance = df_firefly_tensorflow_performance_min\n", "df_firefly_accuracy_500 = df_firefly_tensorflow_accuracy_500\n", "df_firefly_performance_accuracy_500 = merge_performance_accuracy(df_firefly_performance, df_firefly_accuracy_500)\n", "plot(df_firefly_performance_accuracy_500, platform_id=firefly_id, title=firefly_name, xmin=0, xmax=150.1, xstep=10,\n", " accuracy_metric=accuracy_metric, save_fig_name='%s-%s-500-tf' % (firefly_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by convolution method" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_firefly_tensorflow_performance_raw, platform_id=firefly_id, groupby_level='convolution_method')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_firefly_tensorflow_performance_raw, platform_id=firefly_id, groupby_level='convolution_method')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by library" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_firefly_tensorflow_performance_raw, platform_id=firefly_id, groupby_level='library')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_firefly_tensorflow_performance_raw, platform_id=firefly_id, groupby_level='library')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## ArmCL experiments on Firefly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL performance (latency)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_armcl_performance_raw = get_experimental_results(repo_uoa=firefly_armcl_performance_repo_uoa,\n", " tags='explore-mobilenets-performance', accuracy=False)\n", "# Take the minimum execution time out of several repetitions.\n", "df_firefly_armcl_performance = \\\n", " df_firefly_armcl_performance_raw .groupby(level=df_firefly_armcl_performance_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_firefly_armcl_performance)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by convolution method" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_firefly_armcl_performance_raw, platform_id=firefly_id, groupby_level='convolution_method')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_firefly_armcl_performance_raw, platform_id=firefly_id, groupby_level='convolution_method')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plot by library" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance(df_firefly_armcl_performance_raw, platform_id=firefly_id, groupby_level='library')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_firefly_armcl_performance_raw, platform_id=firefly_id, groupby_level='library')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ArmCL accuracy on 500 images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_armcl_accuracy_500_raw = get_experimental_results(repo_uoa=firefly_armcl_accuracy_500_repo_uoa,\n", " tags='explore-mobilenets-accuracy', accuracy=True)\n", "# Extract frame predictions.\n", "# df_firefly_armcl_predictions_500 = df_firefly_armcl_accuracy_500_raw[['frame_predictions']]\n", "# Reduce the repetition_id index dimension (only 1 repetition anyway).\n", "df_firefly_armcl_accuracy_500 = \\\n", " df_firefly_armcl_accuracy_500_raw[['accuracy_top1', 'accuracy_top5']] \\\n", " .groupby(level=df_firefly_armcl_accuracy_500_raw.index.names[:-1]).min()\n", "# Display all rows and columns.\n", "display_in_full(df_firefly_armcl_accuracy_500)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Identical accuracy for \"18.03\" and \"18.01\".\n", "(df_firefly_armcl_accuracy_500.loc[firefly_id,'armcl-18.03'] - df_firefly_armcl_accuracy_500.loc[firefly_id,'armcl-18.01'] == 0).all()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Non-identical accuracy for \"18.03\" and \"17.12\".\n", "df_firefly_armcl_accuracy_500.loc[firefly_id,'armcl-18.03'] - df_firefly_armcl_accuracy_500.loc[firefly_id,'armcl-17.12']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Non-identical accuracy for \"18.03\" on HiKey and \"18.03\" on Firefly.\n", "df_armcl_accuracy_500.loc[hikey_id,'armcl-18.03'] - df_firefly_armcl_accuracy_500.loc[firefly_id,'armcl-18.03']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_armcl_performance_accuracy_500 = merge_performance_accuracy(df_firefly_armcl_performance, df_firefly_armcl_accuracy_500)\n", "display_in_full(df_firefly_armcl_performance_accuracy_500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot top 1 accuracy on 500 images vs. performance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot(df_firefly_armcl_performance_accuracy_500, accuracy_metric=accuracy_metric, platform_id=firefly_id,\n", " xmin=0, xmax=150.1, xstep=10, save_fig_name='%s-%s-500-18_03__18_01__17_12' % (firefly_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## ArmCL vs. TensorFlow on Firefly" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_firefly_tensorflow_performance = df_firefly_tensorflow_performance_min.loc[firefly_id,['tensorflow-1.7'],:]\n", "df_firefly_accuracy_500 = pd.concat([df_firefly_armcl_accuracy_500, df_firefly_tensorflow_accuracy_500])\n", "df_firefly_performance = pd.concat([df_firefly_armcl_performance, df_firefly_tensorflow_performance])\n", "df_firefly_performance_accuracy_500 = merge_performance_accuracy(df_firefly_performance, df_firefly_accuracy_500)\n", "plot(df_firefly_performance_accuracy_500, accuracy_metric=accuracy_metric, platform_id=firefly_id, save_fig=True,\n", " xmin=0, xmax=150.1, xstep=10, save_fig_name='%s-%s-500-18_03__18_01__17_12__tf' % (firefly_id, accuracy_metric))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## New experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 20180709 - 178 MHz" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_armcl_performance_178mhz = get_experimental_results(repo_uoa='ck-request-asplos18-mobilenets-armcl-opencl-20180709-178mhz',\n", " tags='explore-mobilenets-performance', accuracy=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_armcl_performance_178mhz, platform_id=hikey_id, gpu_mhz=178, groupby_level='convolution_method')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_performance_violin(df_armcl_performance_178mhz, platform_id=hikey_id, gpu_mhz=178, groupby_level='library')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 1 }