{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Time-series prediction with Keras `SimpleRNN` class\n",
    "### Dr. Tirthajyoti Sarkar, Fremont, CA 94536 ([LinkedIn](https://www.linkedin.com/in/tirthajyoti-sarkar-2127aa7/), [Github](https://tirthajyoti.github.io))\n",
    "\n",
    "For more tutorial-style notebooks on deep learning, **[here is my Github repo](https://github.com/tirthajyoti/Deep-learning-with-Python)**.\n",
    "\n",
    "For more tutorial-style notebooks on general machine learning, **[here is my Github repo](https://github.com/tirthajyoti/Machine-Learning-with-Python)**.\n",
    "\n",
    "---\n",
    "### What is this Notebook about?\n",
    "In this notebook, we show a building simple recurrent neural network (RNN) using Keras.\n",
    "\n",
    "We will generate some synthetic time-series data by multiplying two periodic/ sinusoidal signals and adding some stochasticity (Gaussian noise). Then, we will take a small fraction of the data and train a simple RNN model with it and try to predict the rest of the data and see how the predictions match up with the ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From f:\\KTU\\Neuroninių tinklų metodai\\venv\\Lib\\site-packages\\keras\\src\\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, SimpleRNN\n",
    "from keras.optimizers import RMSprop\n",
    "from keras.callbacks import Callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.561333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.203835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.207246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.486425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.239823</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Data\n",
       "0 -0.561333\n",
       "1  0.203835\n",
       "2 -0.207246\n",
       "3 -0.486425\n",
       "4  0.239823"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total time points\n",
    "N = 3000\n",
    "# Time point to partition train/test splits\n",
    "Tp = 750\n",
    "\n",
    "t=np.arange(0,N)\n",
    "x=(2*np.sin(0.02*t)*np.sin(0.003*t))+0.5*np.random.normal(size=N)\n",
    "df = pd.DataFrame(x, columns=['Data'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 70328 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(15,4))\n",
    "plt.plot(df,c='blue')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the values in train and test\n",
    "\n",
    "So, we took only 25% of the data as training samples and set aside the rest of the data for testing. \n",
    "\n",
    "Looking at the time-series plot, we think **it is not easy for a standard model to come up with correct trend predictions.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "values=df.values\n",
    "train,test = values[0:Tp,:], values[Tp:N,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data length: (754,)\n",
      "Test data length: (2254,)\n"
     ]
    }
   ],
   "source": [
    "print(\"Train data length:\", train.shape)\n",
    "print(\"Test data length:\", test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 76048 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "index = df.index.values\n",
    "plt.figure(figsize=(15,4))\n",
    "plt.plot(index[0:Tp],train,c='blue')\n",
    "plt.plot(index[Tp:N],test,c='orange',alpha=0.7)\n",
    "plt.legend(['Train','Test'])\n",
    "plt.axvline(df.index[Tp], c=\"r\")\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step (or _embedding_)\n",
    "RNN model requires a step value that contains n number of elements as an input sequence.\n",
    "\n",
    "Suppose x = {1,2,3,4,5,6,7,8,9,10}\n",
    "\n",
    "for step=1, x input  and its y prediction become:\n",
    "\n",
    "| x  | y  |\n",
    "|---|---|\n",
    "| 1  | 2  |\n",
    "| 2  | 3  |\n",
    "| 3  | 4  |\n",
    "| ...  | ...  |\n",
    "| 9  | 10  |\n",
    "\n",
    "for step=3, x and y contain:\n",
    "\n",
    "| x  | y  |\n",
    "|---|---|\n",
    "| 1,2,3  | 4  |\n",
    "| 2,3,4  | 5  |\n",
    "| 3,4,5  | 6  |\n",
    "| ...  | ...  |\n",
    "| 7,8,9  | 10  |\n",
    "\n",
    "Here, we choose `step=4`. In more complex RNN and in particular for text processing, this is also called _embedding size_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "step = 4\n",
    "\n",
    "# add step elements into train and test\n",
    "test = np.append(test,np.repeat(test[-1,],step))\n",
    "train = np.append(train,np.repeat(train[-1,],step))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data length: (754,)\n",
      "Test data length: (2254,)\n"
     ]
    }
   ],
   "source": [
    "print(\"Train data length:\", train.shape)\n",
    "print(\"Test data length:\", test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting to a multi-dimensional array\n",
    "Next, we'll convert test and train data into the matrix with step value as it has shown above example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convertToMatrix(data, step):\n",
    "    X, Y =[], []\n",
    "    for i in range(len(data)-step):\n",
    "        d=i+step\n",
    "        X.append(data[i:d,])\n",
    "        Y.append(data[d,])\n",
    "    return np.array(X), np.array(Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainX,trainY =convertToMatrix(train,step)\n",
    "testX,testY =convertToMatrix(test,step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
    "testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (750, 1, 4) ,  (750,)\n",
      "Test data shape: (2250, 1, 4) ,  (2250,)\n",
      "[ 0.75878716  1.50069712  0.54320651 ... -0.12409353 -0.12409353\n",
      " -0.12409353]\n"
     ]
    }
   ],
   "source": [
    "print(\"Training data shape:\", trainX.shape,', ',trainY.shape)\n",
    "print(\"Test data shape:\", testX.shape,', ',testY.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras model with `SimpleRNN` layer\n",
    "\n",
    "- 256 neurons in the RNN layer\n",
    "- 32 denurons in the densely connected layer\n",
    "- a single neuron for the output layer\n",
    "- ReLu activation\n",
    "- learning rate: 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_simple_rnn(num_units=128, embedding=4,num_dense=32,lr=0.001):\n",
    "    \"\"\"\n",
    "    Builds and compiles a simple RNN model\n",
    "    Arguments:\n",
    "              num_units: Number of units of a the simple RNN layer\n",
    "              embedding: Embedding length\n",
    "              num_dense: Number of neurons in the dense layer followed by the RNN layer\n",
    "              lr: Learning rate (uses RMSprop optimizer)\n",
    "    Returns:\n",
    "              A compiled Keras model.\n",
    "    \"\"\"\n",
    "    model = Sequential()\n",
    "    model.add(SimpleRNN(units=num_units, input_shape=(1,embedding), activation=\"relu\"))\n",
    "    model.add(Dense(num_dense, activation=\"relu\"))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(loss='mean_squared_error', optimizer=RMSprop(lr=lr),metrics=['mse'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_simple_rnn() # Taking the defaults"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " simple_rnn_1 (SimpleRNN)    (None, 128)               17024     \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 32)                4128      \n",
      "                                                                 \n",
      " dense_3 (Dense)             (None, 1)                 33        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 21,185\n",
      "Trainable params: 21,185\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A simple callback class to show a message every 50 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyCallback(Callback):\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        if (epoch+1) % 50 == 0 and epoch>0:\n",
    "            print(\"Epoch number {} done\".format(epoch+1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size=16\n",
    "num_epochs = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch number 50 done\n",
      "Epoch number 100 done\n",
      "Epoch number 150 done\n",
      "Epoch number 200 done\n",
      "Epoch number 250 done\n",
      "Epoch number 300 done\n",
      "Epoch number 350 done\n",
      "Epoch number 400 done\n",
      "Epoch number 450 done\n",
      "Epoch number 500 done\n",
      "Epoch number 550 done\n",
      "Epoch number 600 done\n",
      "Epoch number 650 done\n",
      "Epoch number 700 done\n",
      "Epoch number 750 done\n",
      "Epoch number 800 done\n",
      "Epoch number 850 done\n",
      "Epoch number 900 done\n",
      "Epoch number 950 done\n",
      "Epoch number 1000 done\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fe68057d760>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(trainX,trainY,\n",
    "          epochs=num_epochs,\n",
    "          batch_size=batch_size,\n",
    "          callbacks=[MyCallback()],verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 50292 characters]",
      "text/plain": [
       "<Figure size 700x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7,5))\n",
    "plt.title(\"RMSE loss over epochs\",fontsize=16)\n",
    "plt.plot(np.sqrt(model.history.history['loss']),c='k',lw=2)\n",
    "plt.grid(True)\n",
    "plt.xlabel(\"Epochs\",fontsize=14)\n",
    "plt.ylabel(\"Root-mean-squared error\",fontsize=14)\n",
    "plt.xticks(fontsize=14)\n",
    "plt.yticks(fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions\n",
    "Note that the model was fitted only with the `trainX` and `trainY` data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38432 characters]",
      "text/plain": [
       "<Figure size 500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,4))\n",
    "plt.title(\"This is what the model saw\",fontsize=18)\n",
    "plt.plot(trainX[:,0][:,0],c='blue')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24/24 [==============================] - 0s 546us/step\n",
      "71/71 [==============================] - 0s 487us/step\n"
     ]
    }
   ],
   "source": [
    "trainPredict = model.predict(trainX)\n",
    "testPredict= model.predict(testX)\n",
    "predicted=np.concatenate((trainPredict,testPredict),axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 66236 characters]",
      "text/plain": [
       "<Figure size 1000x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,4))\n",
    "plt.title(\"This is what the model predicted\",fontsize=18)\n",
    "plt.plot(testPredict,c='orange')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing it with the ground truth (test set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 139756 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "index = df.index.values\n",
    "plt.figure(figsize=(15,4))\n",
    "plt.title(\"Ground truth and prediction together\",fontsize=18)\n",
    "plt.plot(index,df,c='blue')\n",
    "plt.plot(index,predicted,c='orange',alpha=0.75)\n",
    "plt.legend(['True data','Predicted'],fontsize=15)\n",
    "plt.axvline(df.index[Tp], c=\"r\")\n",
    "plt.grid(True)\n",
    "plt.xticks(fontsize=14)\n",
    "plt.yticks(fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How are the errors distributed?\n",
    "The errors, or residuals, as they are called in a regression problem, can be plotted to see if they follow any specific distribution. In the generation process, we injected Gaussian noise, so we expect the error to follow the same pattern, _if the model has been able to fit to the real data correctly_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "error=predicted[Tp:N]-df[Tp:N]\n",
    "error=np.array(error).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 6844 characters]",
      "text/plain": [
       "<Figure size 504x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(7,5))\n",
    "plt.hist(error,bins=25,edgecolor='k',color='orange')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 50708 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(15,4))\n",
    "plt.plot(error,c='blue',alpha=0.75)\n",
    "plt.hlines(y=0,xmin=-50,xmax=2400,color='k',lw=3)\n",
    "plt.xlim(-50,2350)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make the model better\n",
    "\n",
    "Note, for running these experiments reasonably fast, we will fix the model size to be smaller than the model above. We will use a RNN layer with 32 neurons followed by a densely connected layer of 8 neurons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Varying the embedding/step size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predictions(model,trainX,testX):\n",
    "    trainPredict = model.predict(trainX)\n",
    "    testPredict= model.predict(testX)\n",
    "    predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
    "\n",
    "    return predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_compare(predicted):\n",
    "    index = df.index.values\n",
    "    plt.figure(figsize=(15,4))\n",
    "    plt.title(\"Ground truth and prediction together\",fontsize=18)\n",
    "    plt.plot(index,df,c='blue')\n",
    "    plt.plot(index,predicted,c='orange',alpha=0.75)\n",
    "    plt.legend(['True data','Predicted'],fontsize=15)\n",
    "    plt.axvline(df.index[Tp], c=\"r\")\n",
    "    plt.grid(True)\n",
    "    plt.xticks(fontsize=14)\n",
    "    plt.yticks(fontsize=14)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_data(step=4):\n",
    "    values=df.values\n",
    "    train,test = values[0:Tp,:], values[Tp:N,:]\n",
    "    test = np.append(test,np.repeat(test[-1,],step))\n",
    "    train = np.append(train,np.repeat(train[-1,],step))\n",
    "    trainX,trainY =convertToMatrix(train,step)\n",
    "    testX,testY =convertToMatrix(test,step)\n",
    "    trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
    "    testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
    "\n",
    "    return trainX,testX,trainY,testY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.RMSprop.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (750, 1, 2) ,  (750,)\n",
      "Test data shape: (2250, 1, 2) ,  (2250,)\n",
      "24/24 [==============================] - 0s 712us/step\n",
      "71/71 [==============================] - 0s 630us/step\n",
      "Embedding size: 2\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 136892 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training data shape: (750, 1, 4) ,  (750,)\n",
      "Test data shape: (2250, 1, 4) ,  (2250,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.RMSprop.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24/24 [==============================] - 0s 696us/step\n",
      "71/71 [==============================] - 0s 600us/step\n",
      "Embedding size: 4\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 138128 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training data shape: (750, 1, 6) ,  (750,)\n",
      "Test data shape: (2250, 1, 6) ,  (2250,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.RMSprop.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24/24 [==============================] - 0s 739us/step\n",
      "71/71 [==============================] - 0s 643us/step\n",
      "Embedding size: 6\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 135340 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training data shape: (750, 1, 8) ,  (750,)\n",
      "Test data shape: (2250, 1, 8) ,  (2250,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.RMSprop.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24/24 [==============================] - 0s 609us/step\n",
      "71/71 [==============================] - 0s 600us/step\n",
      "Embedding size: 8\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 130472 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training data shape: (750, 1, 10) ,  (750,)\n",
      "Test data shape: (2250, 1, 10) ,  (2250,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.RMSprop.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24/24 [==============================] - 0s 652us/step\n",
      "71/71 [==============================] - 0s 629us/step\n",
      "Embedding size: 10\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 130848 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training data shape: (750, 1, 12) ,  (750,)\n",
      "Test data shape: (2250, 1, 12) ,  (2250,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.RMSprop.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24/24 [==============================] - 0s 652us/step\n",
      "71/71 [==============================] - 0s 600us/step\n",
      "Embedding size: 12\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 130560 characters]",
      "text/plain": [
       "<Figure size 1500x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for s in [2,4,6,8,10,12]:\n",
    "    trainX,testX,trainY,testY = prepare_data(s)\n",
    "    print(\"Training data shape:\", trainX.shape,', ',trainY.shape)\n",
    "    print(\"Test data shape:\", testX.shape,', ',testY.shape)\n",
    "\n",
    "    model = build_simple_rnn(num_units=32,num_dense=8,embedding=s)\n",
    "    batch_size=16\n",
    "    num_epochs = 100\n",
    "    model.fit(trainX,trainY,\n",
    "          epochs=num_epochs,\n",
    "          batch_size=batch_size,\n",
    "          verbose=0)\n",
    "    preds = predictions(model,trainX,testX)\n",
    "    print(\"Embedding size: {}\".format(s))\n",
    "    print(\"-\"*100)\n",
    "    plot_compare(preds)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ran for 100 epochs\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 85300 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran for 200 epochs\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 85748 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran for 300 epochs\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 85148 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran for 400 epochs\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 88116 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran for 500 epochs\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 86228 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for e in [100,200,300,400,500]:\n",
    "    trainX,testX,trainY,testY = prepare_data(8)\n",
    "    model = build_simple_rnn(num_units=32,num_dense=8,embedding=8)\n",
    "    batch_size=16\n",
    "    num_epochs = e\n",
    "    model.fit(trainX,trainY,\n",
    "          epochs=num_epochs,\n",
    "          batch_size=batch_size,\n",
    "          verbose=0)\n",
    "    preds = predictions(model,trainX,testX)\n",
    "    print(\"Ran for {} epochs\".format(e))\n",
    "    print(\"-\"*100)\n",
    "    plot_compare(preds)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ran with batch size: 4\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 87064 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran with batch size: 8\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 85896 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran with batch size: 16\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 85968 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran with batch size: 32\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 87316 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Ran with batch size: 64\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 86884 characters]",
      "text/plain": [
       "<Figure size 1080x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for b in [4,8,16,32,64]:\n",
    "    trainX,testX,trainY,testY = prepare_data(8)\n",
    "    model = build_simple_rnn(num_units=32,num_dense=8,embedding=8)\n",
    "    batch_size=b\n",
    "    num_epochs = 250\n",
    "    model.fit(trainX,trainY,\n",
    "          epochs=num_epochs,\n",
    "          batch_size=batch_size,\n",
    "          verbose=0)\n",
    "    preds = predictions(model,trainX,testX)\n",
    "    print(\"Ran with batch size: {}\".format(b))\n",
    "    print(\"-\"*100)\n",
    "    plot_compare(preds)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "Clearly, following trends were observed,\n",
    "\n",
    "- Too small embedding size is not useful but a very long embedding is also not effective. An embedding of 8 looks good for this data.\n",
    "- More epochs are always better.\n",
    "- A batch size of 8 or 16 looks optimal.\n",
    "\n",
    "Ultimately, an exhaustive hyperparameter tuning is needed for the best overall performance. Also, the predictive power is not well-defined as we are judging the quality of the prediction mostly visually here but a neumerical metric (or a few of them) would be a better approach."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
