{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# [PUBLIC] Analysis of CLBlast client multiple sizes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"overview\"></a>\n",
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This Jupyter Notebook analyses the performance that CLBlast  (single configuaration) achieves across a range of sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"data\"></a>\n",
    "## Get the experimental data from DropBox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**NB:** Please ignore this section if you are not interested in re-running or modifying this notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The experimental data was collected on the experimental platform and archived as follows:\n",
    "```\n",
    "$ cd `ck find ck-math:script:<...>`\n",
    "$ python <...>.py\n",
    "$ ck zip local:experiment:* --archive_name=<...>.zip\n",
    "```\n",
    "\n",
    "It can be downloaded and extracted as follows:\n",
    "```\n",
    "$ wget <...>.zip\n",
    "$ ck add repo:<....> --zip=<....>.zip --quiet\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"code\"></a>\n",
    "## Data wrangling code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**NB:** Please ignore this section if you are not interested in re-running or modifying this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Includes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Standard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import json\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Scientific"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If some of the scientific packages are missing, please install them using:\n",
    "```\n",
    "# pip install jupyter pandas numpy matplotlib\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import IPython as ip\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib as mp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "print ('IPython version: %s' % ip.__version__)\n",
    "print ('Pandas version: %s' % pd.__version__)\n",
    "print ('NumPy version: %s' % np.__version__)\n",
    "print ('Seaborn version: %s' % sns.__version__) # apt install python-tk\n",
    "print ('Matplotlib version: %s' % mp.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import cm\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "from IPython.core.display import HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Collective Knowledge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If CK is not installed, please install it using:\n",
    "```\n",
    "# pip install ck\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import ck.kernel as ck\n",
    "print ('CK version: %s' % ck.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Define helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Return the number of floating-point operations for C = alpha * A * B + beta * C,\n",
    "# where A is a MxK matrix and B is a KxN matrix.\n",
    "def xgemm_flops(alpha, beta, M, K, N):\n",
    "    flops_AB = 2*M*N*K if alpha!=0 else 0\n",
    "    flops_C = 2*M*N if beta!=0 else 0\n",
    "    flops = flops_AB + flops_C\n",
    "    return flops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Return GFLOPS (Giga floating-point operations per second) for a known kernel and -1 otherwise.\n",
    "def GFLOPS(kernel, run_characteristics, time_ms):\n",
    "    if kernel.lower().find('xgemm') != -1:\n",
    "        time_ms = np.float64(time_ms)\n",
    "        alpha = np.float64(run_characteristics['arg_alpha'])\n",
    "        beta  = np.float64(run_characteristics['arg_beta'])\n",
    "        M = np.int64(run_characteristics['arg_m'])\n",
    "        K = np.int64(run_characteristics['arg_k'])\n",
    "        N = np.int64(run_characteristics['arg_n'])\n",
    "        return (1e-9 * xgemm_flops(alpha, beta, M, K, N)) / (1e-3 * time_ms)\n",
    "    else:\n",
    "        return (-1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def convert2int(s):\n",
    "    if s[-1]=='K':\n",
    "        return np.int64(s[0:-1])*1024\n",
    "    else:\n",
    "        return np.int64(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def args_str(kernel, run):\n",
    "    args = ''\n",
    "    if kernel.lower().find('xgemm') != -1:\n",
    "        args = 'alpha=%s, beta=%s, M=%s, K=%s, N=%s' % \\\n",
    "            (run['arg_alpha'], run['arg_beta'], run['arg_m'], run['arg_k'], run['arg_n'])\n",
    "    return args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Access the experimental data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_experimental_results(repo_uoa='explore-matrix-size-acl-sgemm-opencl-odroid-xu3', tags=''):\n",
    "    module_uoa = 'experiment'\n",
    "    r = ck.access({'action':'search', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'tags':tags})\n",
    "    if r['return']>0:\n",
    "        print (\"Error: %s\" % r['error'])\n",
    "        exit(1)\n",
    "    experiments = r['lst']\n",
    "    \n",
    "    dfs = []\n",
    "    for experiment in experiments:\n",
    "        data_uoa = experiment['data_uoa']\n",
    "        r = ck.access({'action':'list_points', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'data_uoa':data_uoa})\n",
    "        if r['return']>0:\n",
    "            print (\"Error: %s\" % r['error'])\n",
    "            exit(1)\n",
    "        print experiment\n",
    "        for point in r['points']:\n",
    "            with open(os.path.join(r['path'], 'ckp-%s.0001.json' % point)) as point_file:\n",
    "                point_data_raw = json.load(point_file)\n",
    "                characteristics_list = point_data_raw['characteristics_list']\n",
    "                num_repetitions = len(characteristics_list)\n",
    "                print characteristics_list\n",
    "                # Obtain column data.\n",
    "                data = [\n",
    "                    {\n",
    "                        \n",
    "                        'repetition_id': repetition_id,\n",
    "                        'm': convert2int(characteristics['run']['m']),\n",
    "                        'n': convert2int(characteristics['run']['n']),\n",
    "                        'k': convert2int(characteristics['run']['k']),\n",
    "                        #'mnk': convert2int(characteristics['run']['m'][0]) * convert2int(characteristics['run']['n'][0]) * convert2int(characteristics['run']['k'][0]),\n",
    "                        'G': np.float32(characteristics['run']['GFLOPS_1'])\n",
    "                        #'strategy'  : tuner_output['strategy'],\n",
    "                        #'config_id': config_id,\n",
    "                        #'config' : config['parameters'],\n",
    "                        #'kernel' : config['kernel']\n",
    "                        #'args_id' : args_str(config['kernel'], characteristics['run']),\n",
    "                        #'ms' : np.float64(config['time']),\n",
    "                        #'GFLOPS' : GFLOPS(config['kernel'], characteristics['run'], config['time'])\n",
    "                    }\n",
    "                    for (repetition_id, characteristics) in zip(range(num_repetitions), characteristics_list) \n",
    "                    #for (m,n,k,G,) in characteristics['run']\n",
    "                    #for (config_id, config) in zip(range(len(tuner_output['result'])), tuner_output['result'])\n",
    "                ]\n",
    "            #print data    \n",
    "             #Construct a DataFrame.\n",
    "            df = pd.DataFrame(data)\n",
    "            # Set columns and index names.\n",
    "            df.columns.name = 'characteristics'\n",
    "            df.index.name = 'index'\n",
    "            df = df.set_index(['m', 'n', 'k', 'repetition_id'])\n",
    "            # Append to the list of similarly constructed DataFrames.\n",
    "            dfs.append(df)\n",
    "    # Concatenate all constructed DataFrames (i.e. stack on top of each other).\n",
    "    result = pd.concat(dfs)\n",
    "    return result.sortlevel(result.index.names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = get_experimental_results(tags='acl-sgemm-opencl')\n",
    "pd.options.display.max_columns = len(df.columns)\n",
    "pd.options.display.max_rows = len(df.index)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = df.sortlevel(df.index.names[3])\n",
    "#df.sort_value(level=df.index.names[3])\n",
    "#df = df.sort_values('mnk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#pd.options.display.max_columns=2\n",
    "#df = df.reset_index('mnk').sort_values('mnk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df_mean = df.groupby(level=df.index.names[:-1]).mean()\n",
    "df_std = df.groupby(level=df.index.names[:-1]).std()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df_mean.T \\\n",
    "    .plot(yerr=df_std.T, title='GFLOPS',\n",
    "          kind='bar', rot=0, ylim=[0,20], figsize=[20, 12], grid=True, legend=True, colormap=cm.autumn, fontsize=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "kernel = df.iloc[0].name[0]\n",
    "kernel"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}