{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5ece52ab-b4d0-4408-8065-17ac6098b85e",
   "metadata": {},
   "source": [
    "# Vienasluoksniai tinklai\n",
    "\n",
    "Mašininio mokymosi ir giliojo mokymosi srityse dažnai susiduriama su dviejų rūšių problemomis - regresija ir klasifikacija. Regresijos uždaviniuose prognozuojate funkcijos reikšmes, o klasifikavimo uždaviniuose prognozuojate skirtingas objektų klases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "89d78f8a-6bd6-4e33-99a4-09ed5a6a524a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7522d8d6-5394-4e91-b175-09ce1bed015e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 12772 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_samples = 1000\n",
    "\n",
    "X0 = np.random.normal(loc=[-1,4], scale=[1,0.5], size=(int(n_samples/2), 2))\n",
    "X1 = np.random.normal(loc=[2,4], scale=[0.5,2], size=(int(n_samples/2), 2))\n",
    "y0 = np.zeros((int(n_samples/2),1))\n",
    "y1 = np.ones((int(n_samples/2),1))\n",
    "X = np.vstack([X0, X1])\n",
    "yhat = np.vstack([y0, y1])\n",
    "\n",
    "X = np.array([[0,0],[0,1],[1,0],[1,1]])\n",
    "yhat = np.array([[0],[1],[1],[1]])\n",
    "\n",
    "plt.scatter(*X[yhat[:,0]==0].T, label='0', alpha=0.4); \n",
    "plt.scatter(*X[yhat[:,0]==1].T, label='1', alpha=0.4);\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd43d0a2-39c0-4829-8bbb-e130e5be224a",
   "metadata": {},
   "source": [
    "Naudosime sigmoidinę aktyvacijos funkciją:\n",
    "\n",
    "$$ f(z) = \\frac{1}{1+e^{-z}} $$\n",
    "\n",
    "$$ \\frac{\\mathrm{d}f}{\\mathrm{d}z} = f(z) \\cdot (1-f(z)) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "93cb4f7c-c9e4-407d-9af0-29b2fc3473e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def dfdz(z):\n",
    "    return f(z) * (1 - f(z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0dafd9a9-c75b-4509-be46-bd8ef36991bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 31616 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-5,5)\n",
    "plt.plot(z, f(z), label='$f(z)$'); \n",
    "plt.plot(z, dfdz(z), label='$df\\, /\\,dz$'); \n",
    "plt.legend();\n",
    "plt.title('Sigmoidinė funkcija');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33ff2ff9-34d4-4223-8c5c-525fe998a325",
   "metadata": {},
   "source": [
    "## Tinklo struktūra\n",
    "\n",
    "Neuronas susijęs su įvesties duomenų masyvu $x_i$ per svorio vektorių $w_i$. Atliekame skaliarinę $w$ ir $x$ sandaugą ir pritaikius šiam skaičiui aktyvacijos funkciją $f$, gauname neurono išvestį $y$:\n",
    "\n",
    "$$ \\mathrm{output} = f(w \\cdot x + b) \\equiv f(z) $$\n",
    "čia apibrėžiame, kad $z = w\\cdot x$+b.\n",
    "\n",
    "Jei turime daug sluoksnių, ši išvestis perduodama kaip įvestis kitam neuronų sluoksniui. Šiame pavyzdyje naudosime sigmoidinę funkciją. \n",
    "\n",
    "Mūsų neuroninis tinklas turės tik vieną išvestį - $y$. Išvesties reikšmė yra tarp 0 ir 1, nes tai yra sigmoidinės funkcijos rezultatas. Tikrosios reikšmės bus žymimos $\\hat y$.\n",
    "\n",
    "* Išvestis: $y$, $0<y<1$\n",
    "* Tikroji reikšmė: $\\hat y$ (1 arba 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1324728f-bb43-4440-9d4a-9f5e887f8617",
   "metadata": {},
   "source": [
    "## Nuostolių funkcija\n",
    "\n",
    "Neuroninio tinklo apmokymo metu minimizuosime kryžminės entropijos nuostolių funkciją:\n",
    "\n",
    "$$ J(y, \\hat y) = −(\\hat y\\log(y) + (1−\\hat y)\\log(1−y)) $$\n",
    "\n",
    "$$ \\frac{\\partial J}{\\partial y} = \\frac{1-\\hat y}{1-y} -\\frac{\\hat y}{y} $$\n",
    "\n",
    "Atkreipkite dėmesį į tai, kad čia skaičiuojamas neuroninio tinklo išvesties logaritmas bei dalyba iš jo. Tikėtina, kad ši reikšmė bus lygi arba artimos 0 arba 1 reikšmėms. Jei priartėsime prie dalybos iš 0 arba logaritmo iš 0, mūsų algoritme gali atsirasti skaitiniai nestabilumai. Dėl šios priežasties, pridedame nedidelę konstantą `eps`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6b25b250-30e7-4578-b34b-4f742fa9283a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def J(y, yhat):\n",
    "    eps = 1e-8\n",
    "    return -(yhat * np.log(y+eps) + (1-yhat) * np.log(1-y+eps))\n",
    "\n",
    "def dJdy(y, yhat):\n",
    "    eps = 1e-8\n",
    "    return (1-yhat)/(1-y+eps) - yhat/(y+eps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b7267bb7-edf5-4f22-b2c2-29a49727dfab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32580 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = np.linspace(0.0001,0.9999)\n",
    "plt.plot(y, J(y,0), label='$J(y,0)$'); \n",
    "plt.plot(y, J(y,1), label='$J(y,1)$'); \n",
    "plt.legend();\n",
    "plt.title('Kryžminės entropijos nuostolių funkcija');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695e3638-82b0-422c-9f2e-cedc21c72ca2",
   "metadata": {},
   "source": [
    "## Prognozavimas\n",
    "\n",
    "* Įvestis $x_i$ ($i=0,...,n$)\n",
    "* Svoriai $w_{i}$ ($i=0,...,n$) ir $b$\n",
    "* Išvestis $y$\n",
    "\n",
    "$$y = f\\left(\\sum_i w_i x_i + b\\right) = f(w \\cdot x + b)$$\n",
    "\n",
    "Išvesties skaičiavimas atliekant prognozę pagal įėjimų ir svorių rinkinį vadinamas tiesioginės sklaidos perdavimu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1259,
   "id": "70663f08-a53e-4491-bb2e-7d72c27f110a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward(x, w, b):\n",
    "    return f(np.matmul(x,w)+b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09da4f79-fda5-4dff-a68b-8a2b9d1386ae",
   "metadata": {},
   "source": [
    "## Tinklo apmokymas\n",
    "\n",
    "Norėdami išsiaiškinti, kokie turėtų būti svoriai, juos atnaujinsime stochastinio gradientinio nusileidimo metodu (po vieną žingsnį kiekvienam duomenų rinkinio elementui) naudodami kryžminės entropijos nuostolių funkciją. Šiuo atveju, reikia judėti priešinga gradiento kryptimi žemyn kiekvieno svorio $w_i$ ir $b$ kryptimis, darydami $\\alpha$ dydžio žingsnius. Atnaujinimo taisyklė atrodo taip:\n",
    "\n",
    "$$ w_i \\rightarrow w_i - \\alpha \\frac{\\partial J}{\\partial w_i}$$\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{\\partial J}{\\partial w_i} &= \\frac{\\partial J(y, \\hat y)}{\\partial y} \\frac{\\partial y(z)}{\\partial w_i}\\\\\n",
    " &= \\frac{\\partial J(y, \\hat y)}{\\partial y} \\frac{\\partial y(z)}{\\partial z} \\frac{ \\partial z(w,x)}{\\partial w_i} \n",
    "\\end{align*}\n",
    "\n",
    "\n",
    "Kadangi $z = \\sum x_i w_i+b$, gauname $\\partial z / \\partial w_i = x_i$. Taip pat, $y(z) = f(z)$. Gauname:\n",
    "\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{\\partial J}{\\partial w_i} &= \\left[\\frac{1-\\hat y}{1-y(z)} - \\frac{\\hat y}{y(z)} \\right]\\left[ y(z)(1-y(z)) \\right]\\left[ x_i \\right]\\\\\n",
    "&= \\left(y(1-\\hat y) - \\hat y(1-y) \\right)x_i\n",
    "\\end{align*}\n",
    "\n",
    "Tuomet atnaujinimo taisyklė:\n",
    "\n",
    "$$ w_i \\rightarrow w_i - \\alpha \\left(y(1-\\hat y) - \\hat y(1-y) \\right)x_i $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1260,
   "id": "e4f730e8-2356-47ea-a2ad-cae537caa184",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Option one: use the given derivatives (clearer)\n",
    "def backward1(x, w, b, y, yhat, alpha):\n",
    "    z = np.dot(x, w)\n",
    "    delta = dJdy(y, yhat) * dfdz(z) \n",
    "    \n",
    "    dw = np.reshape(delta*x, (2, -1))\n",
    "    db = delta\n",
    "    \n",
    "    w -= alpha*dw\n",
    "    b -= alpha*db\n",
    "    \n",
    "    return w, b\n",
    "\n",
    "## Option two: use the explicit calculted form (faster)\n",
    "def backward2(x, w, b, y, yhat, alpha):  # w not needed for this form\n",
    "    delta = ( y*(1-yhat) - yhat*(1-y) ) \n",
    "    \n",
    "    dw = np.reshape(delta*x, (2, -1))\n",
    "    db = delta\n",
    "    \n",
    "    w -= alpha*dw\n",
    "    b -= alpha*db\n",
    "    \n",
    "    return w, b\n",
    "\n",
    "def backward3(X, w, b, y, yhat, alpha):  # w not needed for this form\n",
    "    delta = ( y*(1-yhat) - yhat*(1-y) ) \n",
    "\n",
    "    dw = np.reshape(np.mean(delta * X, axis=0), (2, -1))\n",
    "    db = np.mean(delta)\n",
    "    \n",
    "    w -= alpha*dw\n",
    "    b -= alpha*db\n",
    "    \n",
    "    return w, b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b06e8738-1f04-4dce-91fb-167319378f2f",
   "metadata": {},
   "source": [
    "Neuroninio tinklo apmokymo eiga:\n",
    "\n",
    "1. Kiekvienam imties elementui $x$ sudarome $y$ prognozę.\n",
    "2. Apskaičiuojame svorių $w$ ir $b$ atnaujinimo žingsnius (atgalinis ėjimas) ir svorius atnaujiname\n",
    "\n",
    "Keletą kartų pereiname per mokymo aibę, kiekvienam pavyzdžiui atlikdami nedidelius $w$ ir $b$ atnaujinimus. Kiekvienas atskiras pilnas ciklas kurio metu pilnai pereinama mokymo aibė vadinamas epocha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1281,
   "id": "cf8a0bac-af71-4087-9b93-b1711e6130e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(X, yhat, n_epoch, alpha):\n",
    "    costs = np.zeros(shape=(n_epoch,))\n",
    "\n",
    "    np.random.seed(1)\n",
    "    w = np.random.normal(0, 0.01, size=(X.shape[1],1)) \n",
    "    b = np.random.normal(0, 0.01)\n",
    "\n",
    "    for epoch in range(n_epoch):\n",
    "        \n",
    "        #y = forward(X, w, b)\n",
    "        #w, b = backward3(X, w, b, y, yhat, alpha)\n",
    "        \n",
    "        for i, x in enumerate(X):\n",
    "            y = forward(x, w, b)  # prediction for one sample\n",
    "            w, b = backward2(x, w, b, y, yhat[i], alpha)  # take step\n",
    "            \n",
    "        # Calculate total cost after epoch\n",
    "        predictions = forward(X, w, b)  # predictions for entire set\n",
    "        costs[epoch] = np.mean(J(predictions, yhat))  # mean cost per sample\n",
    "\n",
    "        # report progress\n",
    "        if ((epoch % 10) == 0) or (epoch == (n_epoch - 1)):\n",
    "            accuracy = np.mean(predictions.round() == yhat)  # current accuracy on entire set\n",
    "            print('Training accuracy after epoch {}: {:.4%}'.format(epoch, accuracy))\n",
    "            \n",
    "    return w, b, costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1285,
   "id": "df66636b-9f59-40fd-ac22-f4c93814f6b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy after epoch 0: 75.0000%\n",
      "Training accuracy after epoch 10: 75.0000%\n",
      "Training accuracy after epoch 20: 75.0000%\n",
      "Training accuracy after epoch 30: 75.0000%\n",
      "Training accuracy after epoch 40: 100.0000%\n",
      "Training accuracy after epoch 50: 100.0000%\n",
      "Training accuracy after epoch 60: 100.0000%\n",
      "Training accuracy after epoch 70: 100.0000%\n",
      "Training accuracy after epoch 80: 100.0000%\n",
      "Training accuracy after epoch 90: 100.0000%\n",
      "Training accuracy after epoch 100: 100.0000%\n",
      "Training accuracy after epoch 110: 100.0000%\n",
      "Training accuracy after epoch 120: 100.0000%\n",
      "Training accuracy after epoch 130: 100.0000%\n",
      "Training accuracy after epoch 140: 100.0000%\n",
      "Training accuracy after epoch 150: 100.0000%\n",
      "Training accuracy after epoch 160: 100.0000%\n",
      "Training accuracy after epoch 170: 100.0000%\n",
      "Training accuracy after epoch 180: 100.0000%\n",
      "Training accuracy after epoch 190: 100.0000%\n",
      "Training accuracy after epoch 200: 100.0000%\n",
      "Training accuracy after epoch 210: 100.0000%\n",
      "Training accuracy after epoch 220: 100.0000%\n",
      "Training accuracy after epoch 230: 100.0000%\n",
      "Training accuracy after epoch 240: 100.0000%\n",
      "Training accuracy after epoch 250: 100.0000%\n",
      "Training accuracy after epoch 260: 100.0000%\n",
      "Training accuracy after epoch 270: 100.0000%\n",
      "Training accuracy after epoch 280: 100.0000%\n",
      "Training accuracy after epoch 290: 100.0000%\n",
      "Training accuracy after epoch 300: 100.0000%\n",
      "Training accuracy after epoch 310: 100.0000%\n",
      "Training accuracy after epoch 320: 100.0000%\n",
      "Training accuracy after epoch 330: 100.0000%\n",
      "Training accuracy after epoch 340: 100.0000%\n",
      "Training accuracy after epoch 350: 100.0000%\n",
      "Training accuracy after epoch 360: 100.0000%\n",
      "Training accuracy after epoch 370: 100.0000%\n",
      "Training accuracy after epoch 380: 100.0000%\n",
      "Training accuracy after epoch 390: 100.0000%\n",
      "Training accuracy after epoch 399: 100.0000%\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 31212 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "w, b, costs = train(X, yhat, 400, 0.1)   \n",
    "\n",
    "plt.plot(costs); \n",
    "plt.title('Vidutinės nuostolių funkcijos reikšmės kiekvienai epochai');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e5a398-26a7-4f63-8aac-80149edbbc70",
   "metadata": {},
   "source": [
    "Kadangi dirbame 2D erdvėje, funkciją, kurią išmoko mūsų neuroninis tinklas, galime pavaizduoti grafike."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1286,
   "id": "94a0ea27-aa70-4c62-aca7-ef5fbd004efc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 69164 characters]\n",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x1 = np.linspace(-6,6,250)\n",
    "x2 = np.flip(np.linspace(-6,12,250))\n",
    "\n",
    "fun_map = np.empty((x1.size, x2.size))\n",
    "for n,i in enumerate(x1):\n",
    "    for m,j in enumerate(x2):\n",
    "        val = forward([i,j], w, b)\n",
    "        fun_map[m,n] = val\n",
    "\n",
    "plt.figure(figsize=(10,5))\n",
    "plt.imshow(fun_map, extent=[x1.min(),x1.max(),x2.min(),x2.max()], vmin=0, vmax=1, aspect='auto')\n",
    "plt.colorbar()\n",
    "plt.scatter(*X[yhat[:,0]==0].T, label='0', alpha=0.4); \n",
    "plt.scatter(*X[yhat[:,0]==1].T, label='1', alpha=0.4);\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90747560-f9a3-4ad7-af93-982198e682b0",
   "metadata": {},
   "source": [
    "## Netiesinė sprendimo riba\n",
    "\n",
    "Matome, kad mūsų tinklo išmokta riba yra tik linija. Tai nestebina, nes iš esmės atliekame tik logistinę regresiją. Kas nutiks su sudėtingesniu duomenų rinkiniu?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1278,
   "id": "6636ca49-dcc8-4a74-b60b-8b2f04f7da54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 83552 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_samples = 1000\n",
    "X0 = np.random.normal(loc=[0,0], scale=[2,0.5], size=(int(n_samples/2), 2))\n",
    "X11 = np.random.normal(loc=[0,3.5], scale=[0.5,1], size=(int(n_samples/4), 2))\n",
    "X12 = np.random.normal(loc=[0,-3.5], scale=[0.5,1], size=(int(n_samples/4), 2))\n",
    "X1 = np.vstack([X11, X12])\n",
    "y0 = np.zeros(shape=(int(n_samples/2), 1))\n",
    "y1 = np.ones(shape=(int(n_samples/2), 1))\n",
    "X = np.vstack([X0, X1])\n",
    "yhat = np.vstack([y0, y1])\n",
    "\n",
    "plt.scatter(*X[yhat[:,0]==0].T, label='0', alpha=0.4); \n",
    "plt.scatter(*X[yhat[:,0]==1].T, label='1', alpha=0.4);\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1282,
   "id": "6081045c-6261-4595-8b31-8fe773e67fc1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy after epoch 0: 64.9000%\n",
      "Training accuracy after epoch 10: 63.7000%\n",
      "Training accuracy after epoch 20: 63.7000%\n",
      "Training accuracy after epoch 30: 63.7000%\n",
      "Training accuracy after epoch 40: 63.7000%\n",
      "Training accuracy after epoch 50: 63.7000%\n",
      "Training accuracy after epoch 60: 63.7000%\n",
      "Training accuracy after epoch 70: 63.7000%\n",
      "Training accuracy after epoch 80: 63.7000%\n",
      "Training accuracy after epoch 90: 63.7000%\n",
      "Training accuracy after epoch 100: 63.7000%\n",
      "Training accuracy after epoch 110: 63.7000%\n",
      "Training accuracy after epoch 120: 63.7000%\n",
      "Training accuracy after epoch 130: 63.7000%\n",
      "Training accuracy after epoch 140: 63.7000%\n",
      "Training accuracy after epoch 150: 63.7000%\n",
      "Training accuracy after epoch 160: 63.7000%\n",
      "Training accuracy after epoch 170: 63.7000%\n",
      "Training accuracy after epoch 180: 63.7000%\n",
      "Training accuracy after epoch 190: 63.7000%\n",
      "Training accuracy after epoch 200: 63.7000%\n",
      "Training accuracy after epoch 210: 63.7000%\n",
      "Training accuracy after epoch 220: 63.7000%\n",
      "Training accuracy after epoch 230: 63.7000%\n",
      "Training accuracy after epoch 240: 63.7000%\n",
      "Training accuracy after epoch 249: 63.7000%\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 25164 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "w, b, costs = train(X, yhat, 250, 0.01)    \n",
    "\n",
    "plt.plot(costs); \n",
    "plt.title('Vidutinės nuostolių funkcijos reikšmės kiekvienai epochai');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1283,
   "id": "f2860ddc-a852-4b6c-b1f7-ce494fa05309",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f2ca5f44b50>"
      ]
     },
     "execution_count": 1283,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 97916 characters]\n",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x1 = np.linspace(-8,8,250)\n",
    "x2 = np.flip(np.linspace(-10,10,250))\n",
    "fun_map = np.empty((x1.size, x2.size))\n",
    "for n,i in enumerate(x1):\n",
    "    for m,j in enumerate(x2):\n",
    "        fun_map[m,n] = forward([i,j], w, b)\n",
    "\n",
    "plt.figure(figsize=(10,5))\n",
    "plt.imshow(fun_map, extent=[x1.min(),x1.max(),x2.min(),x2.max()], vmin=0, vmax=1, aspect='auto')\n",
    "plt.colorbar()\n",
    "plt.scatter(*X[yhat[:,0]==0].T, label='0', alpha=0.4); \n",
    "plt.scatter(*X[yhat[:,0]==1].T, label='1', alpha=0.4);\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcec77e4-e3f6-4770-beba-f656b0271c92",
   "metadata": {},
   "source": [
    "Matome, kad rezultatai yra blogi (kartais net blogesni nei atsitiktinio spėjimo), o mokymas beveik iš karto sustoja. Be to, išmoktos funkcijos gradientas yra labai nedidelis. Tai galime suprasti kaip modelio netikrumą dėl to, ką ji turėtų išvesti."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
