08_OptimizeClustersHA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizing HA separately"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There were greatest differences in HA objective values when compared results with and without surrogates. Thats why I would like to study this objective more thoroughly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    " %matplotlib inline\n",
    "import seaborn\n",
    "import matplotlib.pyplot as plt\n",
    "from kmeans import kmeans, randomsample\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import os\n",
    "from BorealWeights import BorealWeightedProblem\n",
    "from pyomo.opt import SolverFactory\n",
    "from gradutil import *\n",
    "seed = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now form the clusters by only using HA values, so we can see if the original problem was in the cluster forming part, or if there is something more peculiar in this HA objective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "_,carbon,_,ha = init_boreal()\n",
    "X = normalize(ha.values)\n",
    "randomcenters = randomsample(X, 50)\n",
    "centers, xtoc, dist = kmeans(X,\n",
    "                             randomcenters,\n",
    "                             delta=.00001,\n",
    "                             maxiter=100,\n",
    "                             metric='cosine',\n",
    "                             verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "C = centers.copy()\n",
    "weights = np.array([sum(xtoc==i) for i in range(len(C))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "clustProblemHA = BorealWeightedProblem(C,weights)\n",
    "opt = SolverFactory('glpk')\n",
    "resClustHA = opt.solve(clustProblemHA.model, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "HASurrogateList = res_to_list(clustProblemHA.model)\n",
    "resultSurrogateHA = cluster_to_value(C, HASurrogateList, weights)\n",
    "print(\"(iv) Combined Habitat {:.0f}\".format(resultSurrogateHA))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "resultOriginHA = clusters_to_origin(X, xtoc, HASurrogateList)\n",
    "print(\"(iv) Combined Habitat {:.0f}\".format(resultOriginHA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the original optimization the result was:\n",
    "- (iv) Combined Habitat 10327\n",
    "\n",
    "Which is almost the same than here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this I  conclude, that the problem is clustering when using all the objectives. So lets try doing everything as here before, but use more data for clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The solution could be normalizing all the objectives to 0-1 scale, so there would be no weighting differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "orig_hc_x = np.concatenate((ha, carbon), axis=1)\n",
    "clust_hc_x = normalize(orig_hc_x)\n",
    "no_nan_hc_x = orig_hc_x.copy()\n",
    "hc_inds = np.where(np.isnan(no_nan_hc_x))\n",
    "no_nan_hc_x[hc_inds] = np.take(np.nanmin(no_nan_hc_x, axis=0) - np.nanmax(no_nan_hc_x, axis=0), hc_inds[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "randomcenters = randomsample(clust_hc_x, 50)\n",
    "hc_centers, hc_xtoc, hc_dist = kmeans(clust_hc_x,\n",
    "                             randomcenters,\n",
    "                             delta=.00001,\n",
    "                             maxiter=100,\n",
    "                             metric='cosine',\n",
    "                             verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "hc_C = np.array([no_nan_hc_x[hc_xtoc == i].mean(axis=0) for i in range(len(hc_centers))])\n",
    "\n",
    "hc_weights = np.array([sum(hc_xtoc==i) for i in range(len(hc_C))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "clustProblem_hc_ha = BorealWeightedProblem(hc_C[:,:7],hc_weights)\n",
    "opt = SolverFactory('glpk')\n",
    "resClust_hc_ha = opt.solve(clustProblem_hc_ha.model, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "hc_HASurrogateList = res_to_list(clustProblem_hc_ha.model)\n",
    "hc_resultSurrogateHA = cluster_to_value(hc_C[:,:7], hc_HASurrogateList, hc_weights)\n",
    "print(\"(iv) Combined Habitat {:.0f}\".format(hc_resultSurrogateHA))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "hc_resultOriginHA = clusters_to_origin(orig_hc_x[:,:7], hc_xtoc, hc_HASurrogateList)\n",
    "print(\"(iv) Combined Habitat {:.0f}\".format(hc_resultOriginHA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when clustering with carbon, wes see that the difference is quite big: pretty much the same than what it was when using all the data to do the clustering. So the problem really is here. We just should decide what to do with this..."
   ]
  }
 ],
 "metadata": {
  "git": {
   "suppress_outputs": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}