keras_blog.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation of libFM in Keras\n",
    "\n",
    "This notebook shows how to implement libfm in Keras, and how to use it in the [Talking Data competition on Kaggle](https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection).  The [companion post](https://www.ibm.com/developerworks/community/blogs/jfp/entry/Implementing_Libfm_in_Keras?lang=en) provides context and explanation of how this works.  \n",
    "\n",
    "I recommend people also read the seminal paper on Factorization Machines:\n",
    "\n",
    "Steffen Rendle (2010): <i><a href=\"https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf\">Factorization Machines</a></i>, in Proceedings of the 10th IEEE International Conference on Data Mining (ICDM 2010), Sydney, Australia <a href=\"https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf\">PDF</a>\n",
    "\n",
    "First, let's do some imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jfpuget/anaconda3/envs/tf15/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import gc\n",
    "\n",
    "import mlcrate as mlc\n",
    "import pickle as pkl\n",
    "\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Input, Embedding, Dense, Flatten, Concatenate, Dot, Reshape, Add, Subtract\n",
    "from keras import objectives\n",
    "from keras import backend as K\n",
    "from keras import regularizers \n",
    "from keras.optimizers import Adam\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "from keras.regularizers import l2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's load some data.  I had concatenated the original train and test_supplement data, and saved it as a feather file, using the mlcrate library.  Any other way will do, as long as data can be loaded in memory!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = mlc.load('../data/data_small.feather')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['app', 'channel', 'click_id', 'device', 'ip', 'is_attributed', 'os',\n",
       "       'second'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I often use the following helper function to initialise TensorFlow sessions in order to get (almost) reproducible results. I write almost because CuDNN library is not deterministic, hence results are not totally reproducible.  But fixing all random seeds removes most of the variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import random as rn\n",
    "\n",
    "# The below is necessary in Python 3.2.3 onwards to\n",
    "# have reproducible behavior for certain hash-based operations.\n",
    "# See these references for further details:\n",
    "# https://docs.python.org/3.4/using/cmdline.html#envvar-PYTHONHASHSEED\n",
    "# https://github.com/fchollet/keras/issues/2280#issuecomment-306959926\n",
    "\n",
    "import os\n",
    "\n",
    "def init_seeds(seed):\n",
    "    os.environ['PYTHONHASHSEED'] = '0'\n",
    "\n",
    "    # The below is necessary for starting Numpy generated random numbers\n",
    "    # in a well-defined initial state.\n",
    "\n",
    "    np.random.seed(seed)\n",
    "\n",
    "    # The below is necessary for starting core Python generated random numbers\n",
    "    # in a well-defined state.\n",
    "\n",
    "    rn.seed(seed)\n",
    "\n",
    "    # Force TensorFlow to use single thread.\n",
    "    # Multiple threads are a potential source of\n",
    "    # non-reproducible results.\n",
    "    # For further details, see: https://stackoverflow.com/questions/42022950/which-seeds-have-to-be-set-where-to-realize-100-reproducibility-of-training-res\n",
    "\n",
    "    session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
    "\n",
    "    from keras import backend as K\n",
    "\n",
    "    # The below tf.set_random_seed() will make random number generation\n",
    "    # in the TensorFlow backend have a well-defined initial state.\n",
    "    # For further details, see: https://www.tensorflow.org/api_docs/python/tf/set_random_seed\n",
    "\n",
    "    tf.set_random_seed(seed)\n",
    "\n",
    "    sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
    "    K.set_session(sess)\n",
    "    return sess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will focus on the interactions between os, device and app categories.  We compute an upper bound on the values in each category to prepare for the deep leanring model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[957, 4228, 769]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = ['os', 'device', 'app']\n",
    "f_size  = [int(data[f].max()) + 1 for f in features]\n",
    "f_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First step is to count the number of interaction for each feature combination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "os  device  app\n",
       "0   0       0         604\n",
       "            19     385493\n",
       "            46      16545\n",
       "            49        792\n",
       "            50       1089\n",
       "Name: click_id, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = data.groupby(features)['click_id'].count()\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to compliment this data set with the combinations for which there are no interactions.  One way to do it is to unstack, then stack the result back. \n",
    "\n",
    "Let's do it step by step.  unstack() moves the last index (app) to column names.  Missing entries are then filled with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "      <th>768</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>os</th>\n",
       "      <th>device</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>604.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 769 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "app          0    1    2    3    4    5    6    7    8    9   ...   759  760  \\\n",
       "os device                                                     ...              \n",
       "0  0       604.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "   4         0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "   6         0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "   8         0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "   12        0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "\n",
       "app        761  762  763  764  765  766  767  768  \n",
       "os device                                          \n",
       "0  0       0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "   4       0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "   6       0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "   8       0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "   12      0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "\n",
       "[5 rows x 769 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = X.unstack().fillna(0)\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "stack() moves column names back to index. This gives us the dataset we wanted.  We cast the result so that it takes less memory than if using 64 bits floats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "os  device  app\n",
       "0   0       0      604.0\n",
       "            1        0.0\n",
       "            2        0.0\n",
       "            3        0.0\n",
       "            4        0.0\n",
       "dtype: float32"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = X.stack().astype('float32')\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that missing values now appear as 0.  Last data preparation step is to take the log of the counts to make their distribution less skewed towards large values.  We also move the index to columns via reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>os</th>\n",
       "      <th>device</th>\n",
       "      <th>app</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6.405229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   os  device  app         0\n",
       "0   0       0    0  6.405229\n",
       "1   0       0    1  0.000000\n",
       "2   0       0    2  0.000000\n",
       "3   0       0    3  0.000000\n",
       "4   0       0    4  0.000000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = np.log1p(X).reset_index()\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We rename the count column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>os</th>\n",
       "      <th>device</th>\n",
       "      <th>app</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6.405229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   os  device  app       num\n",
       "0   0       0    0  6.405229\n",
       "1   0       0    1  0.000000\n",
       "2   0       0    2  0.000000\n",
       "3   0       0    3  0.000000\n",
       "4   0       0    4  0.000000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.columns=features + ['num']\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now get the training data for our deep learning model.  The target is the num column.  Note that we do not use the original problem target here, we use interaction counts.  This is a form of unsupervised learning akin to other matrix factorizaiton techniques like PCA or tSVD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = [X[f].values for f in features]\n",
    "y_train = (X[['num']].values).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset is highly imbalanced with less than 1% of non zero entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.00698991895404317"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(X.num > 0).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to cope with that imbalance we will use sample weights. We assign a much larger weight to non negative rows (31 vs 1).  The choice of the positive sample weight is a bit arbitrary, and could be tuned.  \n",
    "\n",
    "Using large weights can lead to exploding gradient.  One way to deal with it is to clip the optimizer norm when compiling the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "w_train = (30 * (y_train > 0).astype('float32') + 1).ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create our model.  The model is a direct implementaiotn of libfm, with a first embedding layer, then a smart computation of all pairwise dot products.  The smartness is not mine, it comes direclty from Rendle paper.  I generalized a bit his model to allow for non categorical input, in which case I use a dense layer to be consistent with categories embeddings.  More explanation can be found on my blog.\n",
    "\n",
    "The other modification I made is to create a secondary model that outputs the embeddings.  We will use it to retrieve the embeddings once the main model will be trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "k_latent = 2\n",
    "embedding_reg = 0.0002\n",
    "kernel_reg = 0.1\n",
    "\n",
    "def get_embed(x_input, x_size, k_latent):\n",
    "    if x_size > 0: #category\n",
    "        embed = Embedding(x_size, k_latent, input_length=1, \n",
    "                          embeddings_regularizer=l2(embedding_reg))(x_input)\n",
    "        embed = Flatten()(embed)\n",
    "    else:\n",
    "        embed = Dense(k_latent, kernel_regularizer=l2(embedding_reg))(x_input)\n",
    "    return embed\n",
    "\n",
    "def build_model_1(X, f_size):\n",
    "    dim_input = len(f_size)\n",
    "    \n",
    "    input_x = [Input(shape=(1,)) for i in range(dim_input)] \n",
    "     \n",
    "    biases = [get_embed(x, size, 1) for (x, size) in zip(input_x, f_size)]\n",
    "    \n",
    "    factors = [get_embed(x, size, k_latent) for (x, size) in zip(input_x, f_size)]\n",
    "    \n",
    "    s = Add()(factors)\n",
    "    \n",
    "    diffs = [Subtract()([s, x]) for x in factors]\n",
    "    \n",
    "    dots = [Dot(axes=1)([d, x]) for d,x in zip(diffs, factors)]\n",
    "    \n",
    "    x = Concatenate()(biases + dots)\n",
    "    x = BatchNormalization()(x)\n",
    "    output = Dense(1, activation='relu', kernel_regularizer=l2(kernel_reg))(x)\n",
    "    model = Model(inputs=input_x, outputs=[output])\n",
    "    opt = Adam(clipnorm=0.5)\n",
    "    model.compile(optimizer=opt, loss='mean_squared_error')\n",
    "    output_f = factors + biases\n",
    "    model_features = Model(inputs=input_x, outputs=output_f)\n",
    "    return model, model_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the main model architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_3 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 1, 2)         1914        input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 1, 2)         8456        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 1, 2)         1538        input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "flatten_4 (Flatten)             (None, 2)            0           embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_5 (Flatten)             (None, 2)            0           embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_6 (Flatten)             (None, 2)            0           embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, 2)            0           flatten_4[0][0]                  \n",
      "                                                                 flatten_5[0][0]                  \n",
      "                                                                 flatten_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1, 1)         957         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 1, 1)         4228        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 1, 1)         769         input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "subtract_1 (Subtract)           (None, 2)            0           add_1[0][0]                      \n",
      "                                                                 flatten_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "subtract_2 (Subtract)           (None, 2)            0           add_1[0][0]                      \n",
      "                                                                 flatten_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "subtract_3 (Subtract)           (None, 2)            0           add_1[0][0]                      \n",
      "                                                                 flatten_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 1)            0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_2 (Flatten)             (None, 1)            0           embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_3 (Flatten)             (None, 1)            0           embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dot_1 (Dot)                     (None, 1)            0           subtract_1[0][0]                 \n",
      "                                                                 flatten_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dot_2 (Dot)                     (None, 1)            0           subtract_2[0][0]                 \n",
      "                                                                 flatten_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dot_3 (Dot)                     (None, 1)            0           subtract_3[0][0]                 \n",
      "                                                                 flatten_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 6)            0           flatten_1[0][0]                  \n",
      "                                                                 flatten_2[0][0]                  \n",
      "                                                                 flatten_3[0][0]                  \n",
      "                                                                 dot_1[0][0]                      \n",
      "                                                                 dot_2[0][0]                      \n",
      "                                                                 dot_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 6)            24          concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 1)            7           batch_normalization_1[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 17,893\n",
      "Trainable params: 17,881\n",
      "Non-trainable params: 12\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model, model_features = build_model_1(X_train, f_size)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the secondary model architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_3 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 1, 2)         1914        input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 1, 2)         8456        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 1, 2)         1538        input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1, 1)         957         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 1, 1)         4228        input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 1, 1)         769         input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "flatten_4 (Flatten)             (None, 2)            0           embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_5 (Flatten)             (None, 2)            0           embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_6 (Flatten)             (None, 2)            0           embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 1)            0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_2 (Flatten)             (None, 1)            0           embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_3 (Flatten)             (None, 1)            0           embedding_3[0][0]                \n",
      "==================================================================================================\n",
      "Total params: 17,862\n",
      "Trainable params: 17,862\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_features.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can train the main model now.  I tuned the batch size and found that a fairly large size of 2¹⁷ was best.  I know, this would be ridiculous with other deep leanring models, but here it gives the best loss.  This was traing on a 1080 Ti NVIDIA GPU.  We stop training as soon as it does not improve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "131072\n",
      "Train on 5045409 samples, validate on 5045409 samples\n",
      "Epoch 1/100\n",
      "5045409/5045409 [==============================] - 4s 1us/step - loss: 2.3507 - val_loss: 0.3435\n",
      "Epoch 2/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 1.4834 - val_loss: 0.2961\n",
      "Epoch 3/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 1.1209 - val_loss: 0.2832\n",
      "Epoch 4/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.8814 - val_loss: 0.2525\n",
      "Epoch 5/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.6741 - val_loss: 0.2372\n",
      "Epoch 6/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.5501 - val_loss: 0.2279\n",
      "Epoch 7/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.4848 - val_loss: 0.2139\n",
      "Epoch 8/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.4441 - val_loss: 0.1976\n",
      "Epoch 9/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.4147 - val_loss: 0.1842\n",
      "Epoch 10/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3923 - val_loss: 0.1730\n",
      "Epoch 11/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3751 - val_loss: 0.1627\n",
      "Epoch 12/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3608 - val_loss: 0.1561\n",
      "Epoch 13/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3486 - val_loss: 0.1499\n",
      "Epoch 14/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3384 - val_loss: 0.1435\n",
      "Epoch 15/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3290 - val_loss: 0.1394\n",
      "Epoch 16/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3206 - val_loss: 0.1351\n",
      "Epoch 17/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3127 - val_loss: 0.1315\n",
      "Epoch 18/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.3056 - val_loss: 0.1270\n",
      "Epoch 19/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2991 - val_loss: 0.1246\n",
      "Epoch 20/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2929 - val_loss: 0.1215\n",
      "Epoch 21/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2875 - val_loss: 0.1177\n",
      "Epoch 22/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2821 - val_loss: 0.1135\n",
      "Epoch 23/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2772 - val_loss: 0.1124\n",
      "Epoch 24/100\n",
      "5045409/5045409 [==============================] - 4s 1us/step - loss: 0.2726 - val_loss: 0.1093\n",
      "Epoch 25/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2685 - val_loss: 0.1065\n",
      "Epoch 26/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2647 - val_loss: 0.1052\n",
      "Epoch 27/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2613 - val_loss: 0.1025\n",
      "Epoch 28/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2584 - val_loss: 0.1008\n",
      "Epoch 29/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2552 - val_loss: 0.0983\n",
      "Epoch 30/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2523 - val_loss: 0.0976\n",
      "Epoch 31/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2497 - val_loss: 0.0968\n",
      "Epoch 32/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2470 - val_loss: 0.0949\n",
      "Epoch 33/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2447 - val_loss: 0.0944\n",
      "Epoch 34/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2425 - val_loss: 0.0929\n",
      "Epoch 35/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2404 - val_loss: 0.0924\n",
      "Epoch 36/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2384 - val_loss: 0.0903\n",
      "Epoch 37/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2363 - val_loss: 0.0893\n",
      "Epoch 38/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2345 - val_loss: 0.0885\n",
      "Epoch 39/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2326 - val_loss: 0.0885\n",
      "Epoch 40/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2309 - val_loss: 0.0871\n",
      "Epoch 41/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2292 - val_loss: 0.0851\n",
      "Epoch 42/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2277 - val_loss: 0.0845\n",
      "Epoch 43/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2261 - val_loss: 0.0836\n",
      "Epoch 44/100\n",
      "5045409/5045409 [==============================] - 3s 1us/step - loss: 0.2245 - val_loss: 0.0837\n",
      "Epoch 00044: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f844bfe98d0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_epochs = 100\n",
    "P = 17\n",
    "try:\n",
    "    del sess\n",
    "except:\n",
    "    pass\n",
    "sess = init_seeds(0)\n",
    "\n",
    "batch_size = 2**P\n",
    "print(batch_size)\n",
    "model, model_features = build_model_1(X_train, f_size)\n",
    "earlystopper = EarlyStopping(patience=0, verbose=1)\n",
    "\n",
    "model.fit(X_train,  y_train, \n",
    "          epochs=n_epochs, batch_size=batch_size, verbose=1, shuffle=True, \n",
    "          validation_data=(X_train, y_train), \n",
    "          sample_weight=w_train,\n",
    "          callbacks=[earlystopper],\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the main model trained we can get the embeddings by preidcitng the secondary model.  This is one beautiful trick if you think of it: we shared layers between models, and we don't directly use the model we trained for predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_pred = model_features.predict(X_train, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should get 1 factors + 1 bias per feature, i.e. 6 in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now retrieve the factors and the biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>os</th>\n",
       "      <th>device</th>\n",
       "      <th>app</th>\n",
       "      <th>num</th>\n",
       "      <th>os_fm_factor_0</th>\n",
       "      <th>os_fm_factor_1</th>\n",
       "      <th>device_fm_factor_0</th>\n",
       "      <th>device_fm_factor_1</th>\n",
       "      <th>app_fm_factor_0</th>\n",
       "      <th>app_fm_factor_1</th>\n",
       "      <th>os_fm_bias</th>\n",
       "      <th>device_fm_bias</th>\n",
       "      <th>app_fm_bias</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6.405229</td>\n",
       "      <td>0.170492</td>\n",
       "      <td>-0.083319</td>\n",
       "      <td>0.047872</td>\n",
       "      <td>-0.093722</td>\n",
       "      <td>-0.146895</td>\n",
       "      <td>0.175355</td>\n",
       "      <td>0.012788</td>\n",
       "      <td>-0.300488</td>\n",
       "      <td>0.022133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.170492</td>\n",
       "      <td>-0.083319</td>\n",
       "      <td>0.047872</td>\n",
       "      <td>-0.093722</td>\n",
       "      <td>0.457369</td>\n",
       "      <td>0.176898</td>\n",
       "      <td>0.012788</td>\n",
       "      <td>-0.300488</td>\n",
       "      <td>0.170095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.170492</td>\n",
       "      <td>-0.083319</td>\n",
       "      <td>0.047872</td>\n",
       "      <td>-0.093722</td>\n",
       "      <td>0.511771</td>\n",
       "      <td>0.186729</td>\n",
       "      <td>0.012788</td>\n",
       "      <td>-0.300488</td>\n",
       "      <td>0.263752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.170492</td>\n",
       "      <td>-0.083319</td>\n",
       "      <td>0.047872</td>\n",
       "      <td>-0.093722</td>\n",
       "      <td>0.511416</td>\n",
       "      <td>0.196360</td>\n",
       "      <td>0.012788</td>\n",
       "      <td>-0.300488</td>\n",
       "      <td>0.275957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.170492</td>\n",
       "      <td>-0.083319</td>\n",
       "      <td>0.047872</td>\n",
       "      <td>-0.093722</td>\n",
       "      <td>0.275622</td>\n",
       "      <td>0.166596</td>\n",
       "      <td>0.012788</td>\n",
       "      <td>-0.300488</td>\n",
       "      <td>0.056378</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   os  device  app       num  os_fm_factor_0  os_fm_factor_1  \\\n",
       "0   0       0    0  6.405229        0.170492       -0.083319   \n",
       "1   0       0    1  0.000000        0.170492       -0.083319   \n",
       "2   0       0    2  0.000000        0.170492       -0.083319   \n",
       "3   0       0    3  0.000000        0.170492       -0.083319   \n",
       "4   0       0    4  0.000000        0.170492       -0.083319   \n",
       "\n",
       "   device_fm_factor_0  device_fm_factor_1  app_fm_factor_0  app_fm_factor_1  \\\n",
       "0            0.047872           -0.093722        -0.146895         0.175355   \n",
       "1            0.047872           -0.093722         0.457369         0.176898   \n",
       "2            0.047872           -0.093722         0.511771         0.186729   \n",
       "3            0.047872           -0.093722         0.511416         0.196360   \n",
       "4            0.047872           -0.093722         0.275622         0.166596   \n",
       "\n",
       "   os_fm_bias  device_fm_bias  app_fm_bias  \n",
       "0    0.012788       -0.300488     0.022133  \n",
       "1    0.012788       -0.300488     0.170095  \n",
       "2    0.012788       -0.300488     0.263752  \n",
       "3    0.012788       -0.300488     0.275957  \n",
       "4    0.012788       -0.300488     0.056378  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "factors = X_pred[:len(features)]\n",
    "\n",
    "biases = X_pred[len(features):2*len(features)]\n",
    "\n",
    "for f, X_p in zip(features, factors):\n",
    "    for i in range(k_latent):\n",
    "        X['%s_fm_factor_%d' % (f, i)] = X_p[:,i]\n",
    "\n",
    "for f, X_p in zip(features, biases):\n",
    "    X['%s_fm_bias' % (f)] = X_p[:,0]\n",
    "\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now save this result.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlc.save(X, '../data/os_device_app_fm_2.feather')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "I then joined it with the original data, using the feature columns as keys.  Adding this and other matrix factorizaiton greatly improved my model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5045409, 2)\n",
      "(5045409, 2)\n",
      "(5045409, 2)\n",
      "(5045409, 1)\n",
      "(5045409, 1)\n",
      "(5045409, 1)\n"
     ]
    }
   ],
   "source": [
    "for pred in X_pred:\n",
    "    print(pred.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tf15]",
   "language": "python",
   "name": "conda-env-tf15-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}