{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"# [cknowledge.org](http://cknowledge.org): Community-driven benchmarking and optimization of computing systems - from classical to quantum\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[Quantum Computing](https://github.com/ctuning/ck-quantum/wiki)\n",
"* [CK-QISKit](https://github.com/ctuning/ck-qiskit) (IBM)\n",
"* [CK-Rigetti](https://github.com/ctuning/ck-rigetti) ([Rigetti Computing](https://rigetti.com/))\n",
"* [CK-ProjectQ](https://github.com/ctuning/ck-projectq) ([ProjectQ](https://projectq.ch/))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[Artificial Intelligence and Machine Learning](http://cknowledge.org/ai)\n",
"* [Reproducible Quality-Efficient Systems Tournaments](http://cknowledge.org/request) ([ReQuEST initiative](http://cknowledge.org/request.html#organizers))\n",
"* [AI artifacts](http://cknowledge.org/ai-artifacts) (cTuning foundation)\n",
"* [Android app](https://play.google.com/store/apps/details?id=openscience.crowdsource.video.experiments) (dividiti)\n",
"* [Desktop app](https://github.com/dividiti/ck-crowdsource-dnn-optimization) (dividiti)\n",
"* [CK-Caffe](https://github.com/dividiti/ck-caffe) (Berkeley)\n",
"* [CK-Caffe2](https://github.com/ctuning/ck-caffe2) (Facebook)\n",
"* [CK-CNTK](https://github.com/ctuning/ck-cntk) (Microsoft)\n",
"* [CK-KaNN](https://github.com/ctuning/ck-kann) (Kalray)\n",
"* [CK-MVNC](https://github.com/ctuning/ck-mvnc) (Movidius / Intel)\n",
"* [CK-MXNet](https://github.com/ctuning/ck-mxnet) (Apache)\n",
"* [CK-NNTest](https://github.com/ctuning/ck-nntest) (cTuning foundation)\n",
"* [CK-TensorFlow](https://github.com/ctuning/ck-tensorflow) (Google)\n",
"* [CK-TensorRT](https://github.com/ctuning/ck-tensorrt) (NVIDIA)\n",
"* etc."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"# Variational Quantum Eigensolver (VQE) on Rigetti machines\n",
"[Quantum Collective Knowledge Hackaton](https://www.eventbrite.co.uk/e/quantum-computing-hackathon-tickets-46441126660#), [Centre for Mathematical Science - University of Cambridge](http://www.cms.cam.ac.uk/), 15 June 2018"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. [Organisers](#organisers)\n",
"1. [References](#references)\n",
"1. [Time-to-solution metric](#metric)\n",
"1. [Setting up](#settingup)\n",
"1. [Running experiments](#running)\n",
"1. [Experimental data](#data)\n",
"1. [Data wrangling code](#code) (for developers)\n",
"1. [Analysis](#analysis)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Organisers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Rigetti Computing](https://rigetti.com): access to [Quantum Virtual Machine](http://pyquil.readthedocs.io/en/latest/qvm.html) (QVM) and [Quantum Processing Unit](http://pyquil.readthedocs.io/en/latest/qpu.html) (QPU).\n",
"- [River Lane Research](https://riverlane.io/): Steve Brierley, Oscar Higgott, Daochen Wang, Amy Flower\n",
"- [dividiti](http://dividiti.com/): Anton Lokhmotov, Leo Gordon, Flavio Vella, Grigori Fursin"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Rigetti's docs](http://grove-docs.readthedocs.io/en/latest/vqe.html)\n",
"- [\"A variational eigenvalue solver on a quantum processor\"](https://arxiv.org/abs/1304.3061) (2013)\n",
"- [\"The theory of variational hybrid quantum-classical algorithms\"](https://arxiv.org/abs/1509.04279) (2015)\n",
"- [\"Quantum optimization using variational algorithms\n",
"on near-term quantum devices\"](https://arxiv.org/abs/1710.01022) [2017]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Time-to-solution metric"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To compare solutions of participants we use a **time-to-solution** $T$ metric defined as follows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Definition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's assume that to find the ground state of a given molecule (e.g. [Helium](https://en.wikipedia.org/wiki/Helium)) a participant makes $N$ runs of their implementation (e.g. $N=3$). A run is considered _successful_ if the ground state found in this run is equal to the ground state known for the molecule to given precision $\\delta$ (e.g. $\\delta=0.1$). Assume that a single run takes $t$ samples (calls to the quantum computer) on average.\n",
"\n",
"Let $s$ be the probability of success of the participant's VQE implementation (i.e. the number of successful runs divided by the total number of runs $N$).\n",
"\n",
"Let $R$ be the number of runs required to find the ground state with given probability $p$ (e.g. $p=0.6$):\n",
"\\begin{equation}\n",
"R = {\\frac{\\log(1-p)}{\\log(1-s)}}.\n",
"\\end{equation}\n",
"\n",
"The **time-to-solution** $T$, defined as the total number of samples used throughout the whole optimisation procedure of VQE, is then calculated as:\n",
"\\begin{equation}\n",
"T = R \\times t.\n",
"\\end{equation}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Derivation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If the probability of success is $s$, then the probability of _failing to find_ the ground state after $R$ runs is $(1-s)^R$. Therefore, the probability of finding the ground state at least once after $R$ runs is $p =1 - (1-s)^R$. Therefore, the number of runs $R$ required to find the ground state at least once with probability $p$ can be found by solving $p=1-(1-s)^R$."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Uncertainties"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also calculate the standard error associated with the calculated time-to-solution $T$. \n",
"\n",
"From the [binomial distribution](Binomial_distribution), the uncertainty $\\sigma_s$ in the success probability $s$ is:\n",
"\\begin{equation}\n",
" \\sigma_s=\\sqrt{\\frac{s(1-s)}{N}}\n",
"\\end{equation}\n",
"where $N$ is the number of runs used to determine $s$.\n",
"\n",
"The uncertainty in the time taken per run $t$ is:\n",
"\\begin{equation}\n",
" \\sigma_t=\\frac{\\mathrm{std}}{\\sqrt{N}}\n",
"\\end{equation}\n",
"where $\\mathrm{std}$ is the standard deviation of the times taken by all $N$ runs.\n",
"\n",
"The uncertainty in total time taken is:\n",
"\\begin{equation}\n",
"\\sigma_T=\\sqrt{0.25 \\cdot (T(t+\\sigma_t, s) - T(t-\\sigma_t,s))^2 + (T(t, s + \\sigma_s) - T(t,s))^2}\n",
"\\end{equation}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Setting up"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Please follow instructions [here](https://github.com/ctuning/ck-quantum/blob/master/README.md)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Running experiments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"$ ck benchmark program:rigetti-vqe \\\n",
" --env.RIGETTI_QUANTUM_DEVICE= \\\n",
" --env.VQE_MINIMIZER_METHOD= \\\n",
" --env.VQE_SAMPLE_SIZE= \\\n",
" --env.VQE_MAX_ITERATIONS= \\\n",
" --record --record_repo=local --record_uoa=- \\\n",
" --tags=qck,hackathon-2018_06_15,,, \\\n",
" --repetitions=\n",
"```\n",
"where:\n",
"- `platform`: `8Q-Agave` or `QVM`;\n",
"- `minimizer_method`: `my_melder_nead` or `my_cobyla` or `my_minimizer` (as defined in [optimizers.py](https://github.com/ctuning/ck-quantum/blob/master/package/tool-hackathon/hackathon-src/hackathon/optimizers.py) installed under e.g. `$CK_TOOLS/hackathon-1.0-linux-64/lib/hackathon`);\n",
"- `sample_size`: e.g. `100` (or another resolution);\n",
"- `max_iterations`: e.g. `80` (or another cut-off point);\n",
"- `email`: a valid email address (later to be replaced with a team id e.g. `team-01`);\n",
"- `repetitions`: how many times to run the experiment with the given parameters: e.g. `3`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Get sample experimental data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The sample experimental data can be downloaded and registered with CK as follows:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"$ wget https://www.dropbox.com/s/a1odux4asze9zpd/ck-quantum-hackathon-20180615.zip\n",
"$ ck add repo --zip=ck-quantum-hackathon-20180615.zip\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"repo_uoa = 'ck-quantum-hackathon-20180615'\n",
"!ck list $repo_uoa:experiment:* --print_full | sort"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Data wrangling code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**NB:** Please ignore this section if you are not interested in re-running or modifying this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Includes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Standard"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"import json\n",
"import re"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Scientific"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If some of the scientific packages are missing, please install them using:\n",
"```\n",
"# pip install jupyter pandas numpy matplotlib\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import IPython as ip\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib as mp"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print ('IPython version: %s' % ip.__version__)\n",
"print ('Pandas version: %s' % pd.__version__)\n",
"print ('NumPy version: %s' % np.__version__)\n",
"print ('Matplotlib version: %s' % mp.__version__)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Image, display\n",
"def display_in_full(df):\n",
" pd.options.display.max_columns = len(df.columns)\n",
" pd.options.display.max_rows = len(df.index)\n",
" display(df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"from matplotlib import cm\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"default_colormap = cm.autumn\n",
"default_fontsize = 16\n",
"default_barwidth = 0.8\n",
"default_figwidth = 16\n",
"default_figheight = 8\n",
"default_figdpi = 200\n",
"default_figsize = [default_figwidth, default_figheight]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if mp.__version__[0]=='2': mp.style.use('classic')\n",
"mp.rcParams['figure.max_open_warning'] = 200\n",
"mp.rcParams['figure.dpi'] = default_figdpi\n",
"mp.rcParams['font.size'] = default_fontsize\n",
"mp.rcParams['legend.fontsize'] = 'medium'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Collective Knowledge"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If CK is not installed, please install it using:\n",
"```\n",
"# pip install ck\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import ck.kernel as ck\n",
"print ('CK version: %s' % ck.__version__)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# NB: Make sure the quantum hackathon tool is installed. (It should be if you have run any experiments.)\n",
"# $ ck install package --tags=ck-quantum,tool,hackathon,v1\n",
"r=ck.access({'action':'show', 'module_uoa':'env', 'tags':'tool,hackathon'})\n",
"if r['return']>0:\n",
" print (\"Error: %s\" % r['error'])\n",
" exit(1)\n",
" \n",
"# Get the path to the first returned environment entry.\n",
"tool_hackathon_dir=r['lst'][0]['meta']['env']['CK_ENV_LIB_HACKATHON_LIB']\n",
"sys.path.append(tool_hackathon_dir)\n",
"from hackathon.utils import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Access experimental data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def get_experimental_results(repo_uoa, tags='qck', module_uoa='experiment'):\n",
" r = ck.access({'action':'search', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'tags':tags})\n",
" if r['return']>0:\n",
" print('Error: %s' % r['error'])\n",
" exit(1)\n",
" experiments = r['lst']\n",
" \n",
" dfs = []\n",
" for experiment in experiments:\n",
" data_uoa = experiment['data_uoa']\n",
" r = ck.access({'action':'list_points', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'data_uoa':data_uoa})\n",
" if r['return']>0:\n",
" print('Error: %s' % r['error'])\n",
" exit(1)\n",
" tags = r['dict']['tags']\n",
"\n",
" skip = False\n",
" # Get team name (final data) or email (submission data).\n",
" team_tags = [ tag for tag in tags if tag.startswith('team-') ]\n",
" email_tags = [ tag for tag in tags if tag.find('@')!=-1 ]\n",
" if len(team_tags) > 0:\n",
" team = team_tags[0][0:7]\n",
" elif len(email_tags) > 0:\n",
" team = email_tags[0]\n",
" else:\n",
" print('[Warning] Cannot determine team name for experiment in: \\'%s\\'' % r['path'])\n",
" team = 'team-default'\n",
"\n",
" if skip:\n",
" print('[Warning] Skipping experiment with bad tags:')\n",
" print(tags)\n",
" continue\n",
" \n",
" # For each point. \n",
" for point in r['points']:\n",
" point_file_path = os.path.join(r['path'], 'ckp-%s.0001.json' % point)\n",
" with open(point_file_path) as point_file:\n",
" point_data_raw = json.load(point_file)\n",
" characteristics_list = point_data_raw['characteristics_list']\n",
" num_repetitions = len(characteristics_list)\n",
" data = [\n",
" {\n",
" # features\n",
" 'platform': characteristics['run'].get('vqe_input', {}).get('q_device_name', 'unknown').lower(),\n",
" # choices\n",
" 'minimizer_method': characteristics['run'].get('vqe_input', {}).get('minimizer_method', 'n/a'),\n",
" 'minimizer_options': characteristics['run'].get('vqe_input', {}).get('minimizer_options', {'maxfev':-1}),\n",
" 'minimizer_src': characteristics['run'].get('vqe_input', {}).get('minimizer_src', ''),\n",
" 'sample_number': characteristics['run'].get('vqe_input', {}).get('sample_number','n/a'),\n",
" # statistical repetition\n",
" 'repetition_id': repetition_id,\n",
" # runtime characteristics\n",
" 'run': characteristics['run'],\n",
" 'report': characteristics['run'].get('report', {}),\n",
" 'vqe_output': characteristics['run'].get('vqe_output', {}),\n",
" }\n",
" for (repetition_id, characteristics) in zip(range(num_repetitions), characteristics_list)\n",
" if len(characteristics['run']) > 0\n",
" ]\n",
" \n",
" for datum in data:\n",
" datum['team'] = team\n",
" datum['point'] = point\n",
" datum['success'] = datum.get('vqe_output',{}).get('success',False)\n",
" datum['nfev'] = np.int64(datum.get('vqe_output',{}).get('nfev',-1))\n",
" datum['nit'] = np.int64(datum.get('vqe_output',{}).get('nit',-1))\n",
" datum['fun'] = np.float64(datum.get('vqe_output',{}).get('fun',0))\n",
" datum['fun_validated'] = np.float64(datum.get('vqe_output',{}).get('fun_validated',0))\n",
" datum['fun_exact'] = np.float64(datum.get('vqe_output',{}).get('fun_exact',0))\n",
" datum['total_seconds'] = np.float64(datum.get('report',{}).get('total_seconds',0))\n",
" datum['total_q_seconds'] = np.float64(datum.get('report',{}).get('total_q_seconds',0))\n",
" datum['total_q_shots'] = np.int64(datum.get('report',{}).get('total_q_shots',0))\n",
" tmp_max_iterations = list(datum.get('minimizer_options',{'maxfev':-1}).values())\n",
" datum['max_iterations'] = tmp_max_iterations[0] if len(tmp_max_iterations)>0 else -1\n",
" index = [\n",
" 'platform', 'team', 'minimizer_method', 'sample_number', 'max_iterations', 'point', 'repetition_id'\n",
" ]\n",
" # Construct a DataFrame.\n",
" df = pd.DataFrame(data)\n",
" df = df.set_index(index)\n",
" # Append to the list of similarly constructed DataFrames.\n",
" dfs.append(df)\n",
" if dfs:\n",
" # Concatenate all thus constructed DataFrames (i.e. stack on top of each other).\n",
" result = pd.concat(dfs)\n",
" result.sort_index(ascending=True, inplace=True)\n",
" else:\n",
" # Construct a dummy DataFrame the success status of which can be safely checked.\n",
" result = pd.DataFrame(columns=['success'])\n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Merge experimental results from the same team with the same parameters\n",
"# (minimizer_method, sample_number, max_iterations) and minimizer source.\n",
"def merge_experimental_results(df):\n",
" dfs = []\n",
" df_prev = None\n",
" for index, row in df.iterrows():\n",
" # Construct a DataFrame.\n",
" df_curr = pd.DataFrame(row).T\n",
" # Check if this row is similar to the previous row.\n",
" if df_prev is not None: # if not the very first row\n",
" if df_prev.index.levels[:-2]==df_curr.index.levels[:-2]: # if the indices match for all but the last two levels\n",
" if df_prev.index.levels[-2]!=df_curr.index.levels[-2]: # if the experiments are different\n",
" if df_prev['minimizer_src'].values==df_curr['minimizer_src'].values: # if the minimizer source is the same\n",
" print('[Info] Merging experiment:')\n",
" print(df_curr.index.levels)\n",
" print('[Info] into:')\n",
" print(df_prev.index.levels)\n",
" print('[Info] as:')\n",
" # df_curr.index = df_prev.index.copy() # TODO: increment repetition_id\n",
" df_curr.index = pd.MultiIndex.from_tuples([(x[0],x[1],x[2],x[3],x[4],x[5],x[6]+1) for x in df_prev.index])\n",
" print(df_curr.index.levels)\n",
" print\n",
" else:\n",
" print('[Warning] Cannot merge experiments as the minimizer source is different:')\n",
" # print('------------------------------------------------------------------------')\n",
" print(df_prev.index.levels)\n",
" # print(df_prev['minimizer_src'].values[0])\n",
" # print\n",
" # print('------------------------------------------------------------------------')\n",
" print(df_curr.index.levels)\n",
" # print(df_curr['minimizer_src'].values[0])\n",
" print\n",
" # else:\n",
" # print('[Info] Keeping experiments separate:')\n",
" # print(df_prev.index.levels)\n",
" # print(df_curr.index.levels)\n",
" # print\n",
" # Append to the list of similarly constructed DataFrames.\n",
" dfs.append(df_curr)\n",
" # Prepare for next iteration.\n",
" df_prev = df_curr\n",
"\n",
" # Concatenate all thus constructed DataFrames (i.e. stack on top of each other).\n",
" result = pd.concat(dfs)\n",
" result.index.names = df.index.names\n",
" result.sort_index(ascending=True, inplace=True)\n",
" \n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def get_metrics(df, delta=0.1, prob=0.5, which_fun_key='fun_exact', which_time_key='total_q_shots'):\n",
" dfs = []\n",
" names_no_repetitions = df.index.names[:-1]\n",
" for index, group in df.groupby(level=names_no_repetitions):\n",
" # Compute metrics.\n",
" classical_energy, minimizer_method, minimizer_src, n_succ, T_ave, T_err, t_ave, t_err, s, s_err = \\\n",
" benchmark_list_of_runs(group['run'], verbose=False, delta=delta, prob=prob,\n",
" which_fun_key=which_fun_key, which_time_key=which_time_key)\n",
" # Construct a DataFrame from the metrics.\n",
" data = {\n",
" # Time to solution.\n",
" 'T_ave' : T_ave,\n",
" 'T_err' : T_err,\n",
" # Time metric (seconds or shots).\n",
" 't_ave' : t_ave,\n",
" 't_err' : t_err,\n",
" # Tries metric.\n",
" 's' : s,\n",
" 's_err' : s_err\n",
" }\n",
" data.update({ k : v for (k, v) in zip(names_no_repetitions, index) })\n",
" data['num_repetitions'] = len(group)\n",
" # NB: index must be something.\n",
" df_ = pd.DataFrame(data=data, index=[0])\n",
" df_ = df_.set_index(names_no_repetitions)\n",
" # Append to the list of similarly constructed DataFrames.\n",
" dfs.append(df_)\n",
" if dfs:\n",
" # Concatenate all thus constructed DataFrames (i.e. stack on top of each other).\n",
" result = pd.concat(dfs).dropna()\n",
" result.sort_index(ascending=True, inplace=True)\n",
" return result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot experimental data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot(df, platform_set=None, minimizer_method_set=None, sample_number_set=None, max_iterations_set=None,\n",
" markersize_divisor=20,\n",
" xmin=0.0, xmax=85.01, xstep=5.00,\n",
" ymin=-3.0, ymax=-0.49, ystep=0.25,\n",
" figsize=(18,9), dpi=200, legend_loc='lower right'):\n",
" \n",
" platform_set = platform_set or df.index.get_level_values(level='platform').unique()\n",
" minimizer_method_set = minimizer_method_set or df.index.get_level_values(level='minimizer_method').unique()\n",
" sample_number_set = sample_number_set or df.index.get_level_values(level='sample_number').unique()\n",
" max_iterations_set = max_iterations_set or df.index.get_level_values(level='max_iterations').unique()\n",
"\n",
" # Options.\n",
" minimizer_method_to_color = {\n",
" 'my_cobyla' : 'orange',\n",
" 'my_nelder_mead' : 'green',\n",
" 'my_minimizer' : 'blue'\n",
" }\n",
" platform_to_marker = {\n",
" '8q-agave' : '8', # octagon\n",
" 'qvm' : 's', # square\n",
" 'local_qasm_simulator' : 'p' # pentagon\n",
" }\n",
" \n",
" last_marker_size = 10\n",
" last_marker_color = 'black'\n",
" last_marker_success_true = '^'\n",
" last_marker_success_false = 'v'\n",
"\n",
" fig = plt.figure(figsize=figsize, dpi=dpi)\n",
" ax = fig.gca()\n",
" for index, row in df.iterrows():\n",
" (platform, team, minimizer_method, sample_number, max_iterations, point, repetition_id) = index\n",
" if platform not in platform_set: continue\n",
" if sample_number not in sample_number_set: continue\n",
" if minimizer_method not in minimizer_method_set: continue\n",
" # NB: This uses 'fun', not 'fun_exact' or 'fun_validated'.\n",
" energies = [ iteration['energy'] for iteration in row['report']['iterations'] ]\n",
" marker=platform_to_marker[platform]\n",
" markersize=sample_number/markersize_divisor\n",
" color=minimizer_method_to_color.get(minimizer_method, 'red')\n",
" markerfacecolor=color\n",
" linestyle='-'\n",
" ax.plot(range(len(energies)), energies, marker=marker, color=color, linestyle=linestyle,\n",
" markerfacecolor=markerfacecolor, markersize=markersize)\n",
" # Mark last function evaluation.\n",
" last_energy = energies[-1]\n",
" last_fev = row['nfev']-1 if minimizer_method=='my_cobyla' or 'my_nelder_mead' else row['nfev']\n",
" last_marker = last_marker_success_true if row['success'] else last_marker_success_false\n",
" ax.plot(last_fev, last_energy, color=last_marker_color, marker=last_marker, markersize=last_marker_size)\n",
"\n",
" # Horizontal line for the known ground state.\n",
" plt.axhline(y=-2.80778395754, color='red', linestyle='--')\n",
" # Vertical lines for max_iterations.\n",
" for max_iterations in max_iterations_set:\n",
" plt.axvline(x=max_iterations, color='black')\n",
" # Grid.\n",
" plt.grid()\n",
" # Title.\n",
" title = 'Variational Quantum Eigensolver (VQE)'\n",
" ax.set_title(title)\n",
" # X axis.\n",
" xlabel='Function evaluation'\n",
" ax.set_xlabel(xlabel)\n",
" ax.set_xlim(xmin, xmax)\n",
" ax.set_xticks(np.arange(xmin, xmax, xstep))\n",
" # Y axis.\n",
" ylabel='Energy'\n",
" ax.set_ylabel(ylabel)\n",
" ax.set_ylim(ymin, ymax)\n",
" ax.set_yticks(np.arange(ymin, ymax, ystep))\n",
" # Legend. https://matplotlib.org/users/legend_guide.html\n",
" handles = [\n",
" mp.lines.Line2D([], [], label='platform=\"%s\",minimizer_method=\"%s\"' % (p,m), color=minimizer_method_to_color.get(m, 'red'),\n",
" marker=platform_to_marker[p], markersize=last_marker_size)\n",
" for p in sorted(platform_set)\n",
" for m in sorted(minimizer_method_set)\n",
" ]\n",
" handles.append(mp.lines.Line2D([],[], label='ground state', color='red', linestyle='--'))\n",
" plt.legend(handles=handles, title='platform,minimizer_method', loc=legend_loc)\n",
" # Save figure.\n",
"# plt.savefig('vqe.energy.png')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_metric(df, metric='total_q_seconds'):\n",
" df.columns.name='metric'\n",
" # \"df.index.names[:-1]\" means reduce along 'repetition_id' (statistical variation).\n",
" df_mean = df[[metric]].groupby(level=df.index.names[:-1]).mean().unstack('platform')\n",
" df_std = df[[metric]].groupby(level=df.index.names[:-1]).std().unstack('platform')\n",
" ax = df_mean.plot(kind='bar', yerr=df_std, grid=True, legend=True, rot=45,\n",
" fontsize=default_fontsize, figsize=default_figsize, colormap=default_colormap)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### All experimental data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = get_experimental_results(repo_uoa=repo_uoa)\n",
"display_in_full(df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Merge experimental results from different runs with the same parameters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = merge_experimental_results(df)\n",
"display_in_full(df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Compute the time-to-solution metric etc."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics = get_metrics(df, delta=0.1, prob=0.5, which_fun_key='fun_exact', which_time_key='total_q_shots')\n",
"df_metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The (unexpected) winner"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Somewhat unexpectedly, the winner obtained the exact answer (!) with `sample_number=1` (!!) on real hardware (!!!). Please see below an explanation why this was the case."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idxmin1 = df_metrics['T_ave'].idxmin()\n",
"idxmin1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics.loc[[idxmin1]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[idxmin1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, minimizer Function, number of Samples, number of function eValuations, Experiment, Repetition\n",
"(p,t,f,s,v,e) = idxmin1\n",
"# Plot the winner.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[idxmin1]))]],\n",
" xmax=7, xstep=1, ymin=-3.00, ymax=0.00+0.01, legend_loc='upper center')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Exclude the winner.\n",
"df_metrics = df_metrics.drop(idxmin1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Explanation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The [Hamiltonian](https://en.wikipedia.org/wiki/Hamiltonian_(quantum_mechanics)) of [Helium](https://en.wikipedia.org/wiki/Helium) in the [STO-3G](https://en.wikipedia.org/wiki/STO-nG_basis_sets) basis is given by:\n",
"\n",
"\\begin{equation}\n",
" H = -1.6678202144537553 + 0.7019459893849936 \\cdot Z_0 + 0.7019459893849936 \\cdot Z_1 + 0.263928235683768058 \\cdot Z_0 \\cdot Z_1\n",
"\\end{equation}\n",
"\n",
"Since the Hamiltonian consists of a sum of commuting operators ($Z_0$, $Z_1$, $Z_0\\cdot Z1$), there exists a simultaneous eigenbasis (including the ground state) on which the value of each of the operators is $+1$ or $-1$. \n",
"\n",
"When `sample_number=1`, measuring an operator once in any state also results in $+1$ or $-1$. This implies that one get \"lucky\" with _any_ mininizer method even on noisy hardware. (Indeed, we have been able to reproduce this result with `my_nelder_mead`.)\n",
"\n",
"For more complex molecules, the Hamiltonian is unlikely to consist of a sum of commuting operators, hence picking up a good optimizer and its parameters will be crucial for success."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The (conditional) runner-up"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The runner-up also used `sample_number=1` but only a single run (hence, it's a \"conditional\" runner-up, as we can determine the error only from multiple runs)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idxmin2 = df_metrics['T_ave'].idxmin()\n",
"idxmin2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics.loc[[idxmin2]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[idxmin2]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, minimizer Function, number of Samples, number of function eValuations, Experiment, Repetition\n",
"(p,t,f,s,v,e) = idxmin2\n",
"# Plot the winner.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[idxmin2]))]],\n",
" xmax=7, xstep=1, ymin=-3.00, ymax=0.00+0.01, legend_loc='upper center')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the minimizer source.\n",
"print(df.loc[(p,t,f,s,v,e,0)]['minimizer_src'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Exclude the conditional runner-up.\n",
"df_metrics = df_metrics.drop(idxmin2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The QVM runner-up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"platform_qvm = 'qvm'\n",
"df_metrics_qvm = df_metrics.loc[platform_qvm]\n",
"idxmin_qvm = df_metrics_qvm['T_ave'].idxmin()\n",
"idxmin_qvm"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics.loc[platform_qvm].loc[[idxmin_qvm]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[platform_qvm].loc[idxmin_qvm]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, minimizer Function, number of Samples, number of function eValuations, Experiment, Repetition\n",
"p = platform_qvm\n",
"(t,f,s,v,e) = idxmin_qvm\n",
"# Plot the runner up.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[(p,t,f,s,v,e)]))]],\n",
" xmax=9, xstep=1, ymin=-3.00, ymax=0.00+0.01, legend_loc='upper right')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the minimizer source.\n",
"print(df.loc[(p,t,f,s,v,e,0)]['minimizer_src'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Exclude the QVM runner-up.\n",
"df_metrics = df_metrics.drop((p,t,f,s,v,e))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The QPU runner-up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"platform_qpu = '8q-agave'\n",
"df_metrics_qpu = df_metrics.loc[platform_qpu]\n",
"idxmin_qpu = df_metrics_qpu['T_ave'].idxmin()\n",
"idxmin_qpu"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics_qpu.loc[[idxmin_qpu]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[platform_qpu].loc[idxmin_qpu]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, Experiment, minimizer Function, number of Samples, number of function eValuations, Repetition\n",
"p = platform_qpu\n",
"(t,f,s,v,e) = idxmin_qpu\n",
"# Plot the QPU runner-up.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[(p,t,f,s,v,e)]))]],\n",
" xmax=7, xstep=1, ymin=-3.00, ymax=-0.00+0.01, legend_loc='upper right')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the minimizer source.\n",
"print(df.loc[(p,t,f,s,v,e,0)]['minimizer_src'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Exclude the QPU runner-up.\n",
"df_metrics = df_metrics.drop((p,t,f,s,v,e))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The best entry with 100% convergence (prob=0.999)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics_prob100 = get_metrics(df, delta=0.1, prob=0.999, which_fun_key='fun_exact', which_time_key='total_q_shots')\n",
"df_metrics_prob100 = df_metrics_prob100[(df_metrics_prob100['s']==1) & (df_metrics_prob100['num_repetitions']>1)]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idxmin_prob100 = df_metrics_prob100['T_ave'].idxmin()\n",
"idxmin_prob100"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics_prob100.loc[[idxmin_prob100]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[idxmin_prob100]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, Experiment, minimizer Function, number of Samples, number of function eValuations, Repetition\n",
"(p,t,f,s,v,e) = idxmin_prob100\n",
"# Plot.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[(p,t,f,s,v,e)]))]],\n",
" xmax=30, xstep=1, ymin=-3.00, ymax=-0.00+0.01, legend_loc='upper right')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The worst entry with 100% convergence (prob=0.999)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idxmax_prob100 = df_metrics_prob100['T_ave'].idxmax()\n",
"idxmax_prob100"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics_prob100.loc[[idxmax_prob100]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[idxmax_prob100]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, Experiment, minimizer Function, number of Samples, number of function eValuations, Repetition\n",
"(p,t,f,s,v,e) = idxmax_prob100\n",
"# Plot.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[(p,t,f,s,v,e)]))]],\n",
" xmax=31, xstep=1, ymin=-3.00, ymax=-0.00+0.01, legend_loc='upper right')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# The ratio of the worst of the best with 100% convergence.\n",
"df_metrics_prob100.loc[idxmax_prob100]['T_ave'] / df_metrics_prob100.loc[idxmin_prob100]['T_ave']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Exclude the best entry with 100% convergence.\n",
"df_metrics_prob100 = df_metrics_prob100.drop(idxmin_prob100)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The most accurate entry (also the runner-up with 100% convergence!)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics_delta0 = get_metrics(df, delta=0.01, prob=0.999, which_fun_key='fun_exact', which_time_key='total_q_shots')\n",
"df_metrics_delta0 = df_metrics_delta0[(df_metrics_delta0['num_repetitions']>1)]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idxmin_delta0 = df_metrics_delta0['T_ave'].idxmin()\n",
"idxmin_delta0"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Also, the runner-up entry with 100% convergence!\n",
"idxmin_prob100 = df_metrics_prob100['T_ave'].idxmin()\n",
"idxmin_prob100"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df_metrics_delta0.loc[[idxmin_delta0]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.loc[idxmin_delta0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Platform, Team, Experiment, minimizer Function, number of Samples, number of function eValuations, Repetition\n",
"(p,t,f,s,v,e) = idxmin_delta0\n",
"# Plot.\n",
"plot(df.loc[[(p,t,f,s,v,e,k) for k in range(len(df.loc[(p,t,f,s,v,e)]))]],\n",
" xmax=9, xstep=1, ymin=-3.00, ymax=-0.00+0.01, legend_loc='upper right')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the minimizer source.\n",
"print(df.loc[(p,t,f,s,v,e,0)]['minimizer_src'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot convergence"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# # Plot all.\n",
"# plot(df, legend_loc='center')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Plot QPU only.\n",
"plot(df, platform_set=[platform_qpu], markersize_divisor=10,\n",
" xmin=0, xmax=34+0.01, xstep=1, ymin=-2.80, ymax=-1.80-0.01, ystep=0.05, legend_loc='lower left')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Plot COBYLA only.\n",
"plot(df, minimizer_method_set=['my_cobyla'], sample_number_set=[50], markersize_divisor=5,\n",
" xmin=5, xmax=32+0.01, xstep=1, ymin=-2.89, ymax=-1.79+0.01, ystep=0.05, legend_loc='upper right')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot execution metrics"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# plot_metric(df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# plot_metric(df, metric='total_q_shots')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 1
}