{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"\n",
    "    border: 2px solid #4CAF50; \n",
    "    padding: 15px; \n",
    "    background-color: #f4f4f4; \n",
    "    border-radius: 10px; \n",
    "    align-items: center;\">\n",
    "\n",
    "<h1 style=\"margin: 0; color: #4CAF50;\">Supervised ML Modelle: K-Nearest-Neighbour</h1>\n",
    "<h2 style=\"margin: 5px 0; color: #555;\">DSAI</h2>\n",
    "<h3 style=\"margin: 5px 0; color: #555;\">Jakob Eggl</h3>\n",
    "\n",
    "<div style=\"flex-shrink: 0;\">\n",
    "    <img src=\"https://www.htl-grieskirchen.at/wp/wp-content/uploads/2022/11/logo_bildschirm-1024x503.png\" alt=\"Logo\" style=\"width: 250px; height: auto;\"/>\n",
    "</div>\n",
    "<p1> © 2024/25 Jakob Eggl. Nutzung oder Verbreitung nur mit ausdrücklicher Genehmigung des Autors.</p1>\n",
    "</div>\n",
    "<div style=\"flex: 1;\">\n",
    "</div>   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Teile (adaptiert) von *AI Inside Seminar KI Mag. Otto Reichel*!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# k-Nearest Neighbour Klassifizierer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wir sehen uns nun den $k$-Nearest Neighbor *Klassifizierer* an! Dieser ist ein sehr einfacher und leicht verständlicher Machine Learning Algorithmus, welcher aber dennoch in vielen Fällen eine erstaunlich gute Performance liefert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Der *kNN*-Klassifizierer $\\ldots$\n",
    "* $\\ldots$ wird bei uns verwendet zur *Klassifizierung*\n",
    "* $\\ldots$ wird durch sehr wenige **Hyperparameter** parametrisiert ($k$: Die Anazahl der Nachbarn; $d(x,y)$ Funktion zur Distanzmessung zwischen Datenpunkten $x$ und $y$.)\n",
    "* $\\ldots$ zieht zur Klassifizierung die Labels der nächsten Nachbarn in Betracht "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Info:** Ein **Hyperparameter** ist ein Parameter, welcher am Anfang vom Benutzer (von uns) eingestellt wird und das Model bzw. ggf. den Lernvorgang parametrisiert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beschreibung des Algorithmus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![KNN-Classifier_for_k=3](../resources/knn_classifier.png)\n",
    "\n",
    "(von https://mlarchive.com/machine-learning/k-nearest-neighbor-knn-explained/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Die Beschreibung dieses Models wird nun anhand von einem Beispiel erklärt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ein Beispiel als Erklärung"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wir gehen davon aus, dass ein Datensatz über **Hunde** und **Pferde** vorliegen, von denen jeweils **Größe** und **Gewicht** bekannt ist. An Hand dieser beiden Eingangsfeatures soll klassifiziert werden, ob es sich um einen Hund oder ein Pferd handelt. Trägt man die Features in ein Diagramm ein so könnte sich das folgende Bild ergeben."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![KNN_Hund_vs_Katze](../resources/knn_hund_vs_katze.png)\n",
    "\n",
    "(von AI Inside Seminar KI Mag. Otto Reichel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Möchte man nun für einen neuen Datenpunkt $x=(a\\textrm{ [lbs]}, b\\textrm{ [ft]})$ feststellen, ob es sich um einen Hund oder ein Pferd handelt, kann man die $k$ nächsten Nachbarn betrachten. Gibt es unter diesen Nachbarn mehr Hunde, so wird der neue Datenpunkt als Hund klassifiziert, andernfalls als Pferd."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dabei wird folgendermaßen vorgegangen:\n",
    "1) Man berechnet die Distanz von $x$ zu allen anderen gegebenen Datenpunkten\n",
    "2) Man sortiert die Punkte nach zunehmender Distanz zu $x$\n",
    "3) Man bestimmt unter den ersten $k$ der sortierten Punkte das am häufigsten auftretende Label. Dies ist nun auch das Label für den neuen Datenpunkt $x$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zusammenfassend kann man also für den **Lernvorgang** und die **Vorhersage** folgendes sagen:\n",
    "\n",
    "Der Lernprozess (Trainingsphase) für den kNN-Klassifizierer ist sehr einfach und beinhaltet nur indirekt \"Lernen\".\n",
    "\n",
    "Die Trainingsdaten $X$ (samt Label $y$) werden im Model effizient gespeichert. Bei der Prediction von einem neuen Datenpunkt können so die nächsten Nachbarn schnell gefunden werden. Dies wird auch *Lazy Learning* genannt.\n",
    "\n",
    "Wie wird eine Vorhersage (Prediction) gemacht?\n",
    "\n",
    "* Je nach gewählter Abstandfunktion wird *der Abstand* zu den Datenpunkten *berechnet*.\n",
    "* Nun werden die Datenpunkte sortiert, sodass die Datenpunkte mit einem geringen Abstand am Anfang gereiht sind."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Die Wahl von $k$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![kNN_Choice_of_k_matters](../resources/knn_3_vs_7.png)\n",
    "\n",
    "(von https://ai.plainenglish.io/introduction-to-k-nearest-neighbors-knn-algorithm-e8617a448fa8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insgesamt ist die Wahl von $k$ frei wählbar. Es gibt aber trotzdem ein paar Dinge, die es zu beachten gibt!\n",
    "\n",
    "* Ein gerader Wert von $k$ könnte zu einem Gleichstand führen, in solchen Fällen wird entweder ein zufälliger Wert verwendet, oder nähere Nachbarn werden stärker gewichtet.\n",
    "* Es kann aber auch bei ungeradem $k$ ein Gleichstand entstehen. Beispiel: Labels 0, 1, 2 für Katze, Hund, Pferd. Nun verwenden wir $k=3$ und erhalten möglicherweise jedes Label einmal.\n",
    "* *Faustregel:* Beim ersten Durchlauf kann $k\\approx\\sqrt{\\vert X_{\\text{train}}\\vert}$ verwendet werden, wobei $\\vert X_{\\text{train}}\\vert$ die Anzahl der Datenpunkte im Trainingsset ist.\n",
    "* Falls *viele Ausreißer* und möglicherweise *Fehler* in den Daten vorhanden sind, ist ein *größerer Wert* für $k$ vorteilhaft, da dies dann diese Punkte etwas ausgleicht, indem mehr \"normale\" Punkte in Betracht gezogen werden."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![kNN_Choice_k_underfitting_overfitting](../resources/knn_overfitting_underfitting.png)\n",
    "\n",
    "(von https://medium.com/@jwbtmf/classifying-customer-churn-using-k-nearest-neighbour-supervised-machine-learning-algorithm-3254441ec7be)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcher dieser Fälle beschreibt **overfitting** und welcher dieser Fälle beschreibt **underfitting**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Die Wahl der Distanz Funktion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neben dem Wert $k$ als Anzahl der Nachbarn kann man auch entscheiden, welche Funktion als Distanzfunktion verwendet werden soll.\n",
    "\n",
    "Für numerische Features wird standardmäßig die **euklidische Distanz** $d(x,y)$ verwendet. Sie berechnet sich genauso wie der Betrag von einem Vektor. Für Daten mit $n$ Features, haben wir also\n",
    "\\begin{equation*}\n",
    "    d(x,y) = \\sqrt{(x_1-y_1)^2 + (x_2-y_2)^2 + \\ldots + (x_n-y_n)^2}.\n",
    "\\end{equation*}\n",
    "Hier ist $y$ der neue Datenpunkt und $x$ steht für einen Datenpunkt in den (Trainings)daten!\n",
    "\n",
    "Je nach Anwendungsfall kann aber auch eine andere Distanzfunktion verwendet werden. So gibt es auch zum Beispiel noch die **Manhattan**-Distanz.\n",
    "\n",
    "Die Manhattan Distanz ist folgendermaßen definiert\n",
    "\\begin{equation*}\n",
    "    d(x,y) = \\vert x_1-y_1 \\vert + \\vert x_2 - y_2 \\vert + \\ldots + \\vert x_n - y_n \\vert.\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Befinden sich in den Daten **kateogrische** (categorical) Features oder **ordinale** **Features**, so muss eine andere Distanzfunktion (zum Beispiel *Hamming* (Referat?)) verwendet werden, *oder* es werden die Daten zuvor mit einem Ordinalencoder oder einem OneHot-Encoder encodiert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalisieren der Daten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Da beim Berechnen der Distanz die Größenordnung einen riesen Einfluss hat, müssen wir vor der Verwendung eines kNN-Klassifizierers sicherstellen, dass die Daten einen vernünftigen Bereich haben (alle in etwa den selben Bereich).\n",
    "\n",
    "Warum ist das so?\n",
    "\n",
    "Wir können die Daten alle auf einen gleichen Bereich bringen, indem wir eine $z$-score Normalisierung machen.\n",
    "\n",
    "Zur Wiederholung, die $z$-score Normalisierung eines Features $X$ sieht folgendermaßen aus\n",
    "\\begin{equation*}\n",
    "    z_i = \\frac{x_i-\\mu}{\\sigma},\n",
    "\\end{equation*}\n",
    "wobei $\\mu$ und $\\sigma$ der Mittelwert bzw. die Standardabweichung des Features $X$ sind. $z_i$ ist der standardisierte Wert vom ursprünglichen Wert $x_i$ und $i$ steht für den Index des Datenpunktes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wäre eine Min-Max Normalisierung auch möglich?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vorteile und Nachteile von kNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Vorteile**:\n",
    "* Sehr einfach\n",
    "* Extrem gut nachvollziehbar\n",
    "* Funktioniert in vielen Fällen sehr gut\n",
    "* Nur wenige Parameter ($k$ und Distanzfunktion)\n",
    "\n",
    "**Nachteile**:\n",
    "* Performance nimmt mit der Anzahl der Features ab (In hohen Dimensionen sind alle Punkte etwa gleich weit voneinander entfernt. Dieses Phänomen nennt man **Curse of Dimensionality**)\n",
    "* Stark abhängig von gewählter Distanzfunktion und gewähltem $k$\n",
    "* Ohne Normalisieren ($z$-Score Normalisierung) würden die einzelnen Features falsch (sprich stark unterschiedlich) gewichtet werden\n",
    "* Wird bei großen Datasets sehr rechenintensiv und somit langsam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Info:** Als Beispiel für den **Curse of Dimensionality** kann man sich folgendes vorstellen: \n",
    "\n",
    "Annahme du vergleichst die Attribute: Wohnadresse, Name des Vaters, Name der Mutter und Telefonnummer mit deinem Bruder/deiner Schwester. In diesem Fall seit du und dein/deine Bruder/Schwester sehr ähnlich. Bis auf Telefonnummer sind alle Attribute gleich und im Vergleich zu anderen Personen (zum Beispiel in der Klasse) seit ihr beide euch am nächsten. Nun erweitern wir die gegebenen Features pro Person mit den Features: \"Lieblingsfarbe, Alter, Schuhgröße, Ausbildung, Name, Einkommen, $\\ldots$ etc.\". Mit diesen Features sind plötzlich du und deine Schwester/dein Bruder nicht mehr wirklich ähnlich. Genauer gesagt gibt es jetzt genügend weitere Personen die gleich viele Attribute gleich/ähnlich haben."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tipps und Tricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Wir können bei nicht allzu großen Datasets auch mehrere $k$-Werte nacheinander probieren und dann den besten Wert verwenden\n",
    "* Features müssen normalisiert werden ($z$-score normalisiert (=standardisiert))\n",
    "* Dimension verringern bei schlechter Performance (eventuell mit Korrelationsmatrix bzw. offensichtlich unbrauchbare Features zur Klassifikation)\n",
    "* Falls ein Label viel öfter vorkommt als ein anderes (unbalanciert), dann könnte man diese Nachbarn höher gewichten (kann in Python/Scikit-Learn einfach gemacht werden, indem ein zusätzlicher Wert übergeben wird.)\n",
    "* Die $k$ nächsten Nachbarn können auch verwendet werden, um die fehlenden Werte zu ersetzen. Sprich wenn zum Beispiel das Feature \"Alter\" fehlt, dann können wir die $k$ nächsten Datenpunkte ansehen und schauen, wie das Alter bei diesen Punkte ist. Anschließend können wir zum Beispiel den Durschnitt bilden."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## kNN in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import fetch_openml\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler, LabelEncoder\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         id  gender   age  hypertension  heart_disease ever_married  \\\n",
      "0      9046    Male  67.0             0              1          Yes   \n",
      "1     51676  Female  61.0             0              0          Yes   \n",
      "2     31112    Male  80.0             0              1          Yes   \n",
      "3     60182  Female  49.0             0              0          Yes   \n",
      "4      1665  Female  79.0             1              0          Yes   \n",
      "...     ...     ...   ...           ...            ...          ...   \n",
      "5105  18234  Female  80.0             1              0          Yes   \n",
      "5106  44873  Female  81.0             0              0          Yes   \n",
      "5107  19723  Female  35.0             0              0          Yes   \n",
      "5108  37544    Male  51.0             0              0          Yes   \n",
      "5109  44679  Female  44.0             0              0          Yes   \n",
      "\n",
      "          work_type Residence_type  avg_glucose_level   bmi   smoking_status  \\\n",
      "0           Private          Urban             228.69  36.6  formerly smoked   \n",
      "1     Self-employed          Rural             202.21   NaN     never smoked   \n",
      "2           Private          Rural             105.92  32.5     never smoked   \n",
      "3           Private          Urban             171.23  34.4           smokes   \n",
      "4     Self-employed          Rural             174.12  24.0     never smoked   \n",
      "...             ...            ...                ...   ...              ...   \n",
      "5105        Private          Urban              83.75   NaN     never smoked   \n",
      "5106  Self-employed          Urban             125.20  40.0     never smoked   \n",
      "5107  Self-employed          Rural              82.99  30.6     never smoked   \n",
      "5108        Private          Rural             166.29  25.6  formerly smoked   \n",
      "5109       Govt_job          Urban              85.28  26.2          Unknown   \n",
      "\n",
      "      stroke  \n",
      "0          1  \n",
      "1          1  \n",
      "2          1  \n",
      "3          1  \n",
      "4          1  \n",
      "...      ...  \n",
      "5105       0  \n",
      "5106       0  \n",
      "5107       0  \n",
      "5108       0  \n",
      "5109       0  \n",
      "\n",
      "[5110 rows x 12 columns]\n"
     ]
    }
   ],
   "source": [
    "# Load dataset (Vehicle Silhouettes dataset)\n",
    "ds = pd.read_csv(\"../../_data/stroke.csv\") # ggf. etwas anders als unser bisheriges Dataset\n",
    "print(ds)\n",
    "ds = ds.dropna()\n",
    "y = ds.stroke\n",
    "X = ds.drop('stroke', axis=1)\n",
    "X = X.select_dtypes(include=[np.number])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "id",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "age",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "hypertension",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "heart_disease",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "avg_glucose_level",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "bmi",
         "rawType": "float64",
         "type": "float"
        }
       ],
       "conversionMethod": "pd.DataFrame",
       "ref": "4a023f6d-2f15-4c73-bc34-1cc2357abf53",
       "rows": [
        [
         "0",
         "9046",
         "67.0",
         "0",
         "1",
         "228.69",
         "36.6"
        ],
        [
         "2",
         "31112",
         "80.0",
         "0",
         "1",
         "105.92",
         "32.5"
        ],
        [
         "3",
         "60182",
         "49.0",
         "0",
         "0",
         "171.23",
         "34.4"
        ],
        [
         "4",
         "1665",
         "79.0",
         "1",
         "0",
         "174.12",
         "24.0"
        ],
        [
         "5",
         "56669",
         "81.0",
         "0",
         "0",
         "186.21",
         "29.0"
        ],
        [
         "6",
         "53882",
         "74.0",
         "1",
         "1",
         "70.09",
         "27.4"
        ],
        [
         "7",
         "10434",
         "69.0",
         "0",
         "0",
         "94.39",
         "22.8"
        ],
        [
         "9",
         "60491",
         "78.0",
         "0",
         "0",
         "58.57",
         "24.2"
        ],
        [
         "10",
         "12109",
         "81.0",
         "1",
         "0",
         "80.43",
         "29.7"
        ],
        [
         "11",
         "12095",
         "61.0",
         "0",
         "1",
         "120.46",
         "36.8"
        ],
        [
         "12",
         "12175",
         "54.0",
         "0",
         "0",
         "104.51",
         "27.3"
        ],
        [
         "14",
         "5317",
         "79.0",
         "0",
         "1",
         "214.09",
         "28.2"
        ],
        [
         "15",
         "58202",
         "50.0",
         "1",
         "0",
         "167.41",
         "30.9"
        ],
        [
         "16",
         "56112",
         "64.0",
         "0",
         "1",
         "191.61",
         "37.5"
        ],
        [
         "17",
         "34120",
         "75.0",
         "1",
         "0",
         "221.29",
         "25.8"
        ],
        [
         "18",
         "27458",
         "60.0",
         "0",
         "0",
         "89.22",
         "37.8"
        ],
        [
         "20",
         "70630",
         "71.0",
         "0",
         "0",
         "193.94",
         "22.4"
        ],
        [
         "21",
         "13861",
         "52.0",
         "1",
         "0",
         "233.29",
         "48.9"
        ],
        [
         "22",
         "68794",
         "79.0",
         "0",
         "0",
         "228.7",
         "26.6"
        ],
        [
         "23",
         "64778",
         "82.0",
         "0",
         "1",
         "208.3",
         "32.5"
        ],
        [
         "24",
         "4219",
         "71.0",
         "0",
         "0",
         "102.87",
         "27.2"
        ],
        [
         "25",
         "70822",
         "80.0",
         "0",
         "0",
         "104.12",
         "23.5"
        ],
        [
         "26",
         "38047",
         "65.0",
         "0",
         "0",
         "100.98",
         "28.2"
        ],
        [
         "28",
         "54827",
         "69.0",
         "0",
         "1",
         "195.23",
         "28.3"
        ],
        [
         "30",
         "43717",
         "57.0",
         "1",
         "0",
         "212.08",
         "44.2"
        ],
        [
         "31",
         "33879",
         "42.0",
         "0",
         "0",
         "83.41",
         "25.4"
        ],
        [
         "32",
         "39373",
         "82.0",
         "1",
         "0",
         "196.92",
         "22.2"
        ],
        [
         "33",
         "54401",
         "80.0",
         "0",
         "1",
         "252.72",
         "30.5"
        ],
        [
         "34",
         "14248",
         "48.0",
         "0",
         "0",
         "84.2",
         "29.7"
        ],
        [
         "35",
         "712",
         "82.0",
         "1",
         "1",
         "84.03",
         "26.5"
        ],
        [
         "36",
         "47269",
         "74.0",
         "0",
         "0",
         "219.72",
         "33.7"
        ],
        [
         "37",
         "24977",
         "72.0",
         "1",
         "0",
         "74.63",
         "23.1"
        ],
        [
         "38",
         "47306",
         "58.0",
         "0",
         "0",
         "92.62",
         "32.0"
        ],
        [
         "39",
         "62602",
         "49.0",
         "0",
         "0",
         "60.91",
         "29.9"
        ],
        [
         "40",
         "4651",
         "78.0",
         "0",
         "0",
         "78.03",
         "23.9"
        ],
        [
         "41",
         "1261",
         "54.0",
         "0",
         "0",
         "71.22",
         "28.5"
        ],
        [
         "42",
         "61960",
         "82.0",
         "0",
         "1",
         "144.9",
         "26.4"
        ],
        [
         "44",
         "7937",
         "60.0",
         "1",
         "0",
         "213.03",
         "20.2"
        ],
        [
         "45",
         "19824",
         "76.0",
         "1",
         "0",
         "243.58",
         "33.6"
        ],
        [
         "47",
         "47472",
         "58.0",
         "0",
         "0",
         "107.26",
         "38.6"
        ],
        [
         "48",
         "35626",
         "81.0",
         "0",
         "0",
         "99.33",
         "33.7"
        ],
        [
         "49",
         "36338",
         "39.0",
         "1",
         "0",
         "58.09",
         "39.2"
        ],
        [
         "52",
         "59190",
         "79.0",
         "0",
         "1",
         "127.29",
         "27.7"
        ],
        [
         "53",
         "47167",
         "77.0",
         "1",
         "0",
         "124.13",
         "31.4"
        ],
        [
         "55",
         "25831",
         "63.0",
         "0",
         "1",
         "196.71",
         "36.5"
        ],
        [
         "56",
         "38829",
         "82.0",
         "0",
         "0",
         "59.32",
         "33.2"
        ],
        [
         "58",
         "58631",
         "73.0",
         "1",
         "0",
         "194.99",
         "32.8"
        ],
        [
         "59",
         "5111",
         "54.0",
         "1",
         "0",
         "180.93",
         "27.7"
        ],
        [
         "60",
         "10710",
         "56.0",
         "0",
         "0",
         "185.17",
         "40.4"
        ],
        [
         "61",
         "55927",
         "80.0",
         "1",
         "0",
         "74.9",
         "22.2"
        ]
       ],
       "shape": {
        "columns": 6,
        "rows": 4909
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>age</th>\n",
       "      <th>hypertension</th>\n",
       "      <th>heart_disease</th>\n",
       "      <th>avg_glucose_level</th>\n",
       "      <th>bmi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9046</td>\n",
       "      <td>67.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>228.69</td>\n",
       "      <td>36.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>31112</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>105.92</td>\n",
       "      <td>32.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60182</td>\n",
       "      <td>49.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>171.23</td>\n",
       "      <td>34.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1665</td>\n",
       "      <td>79.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>174.12</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>56669</td>\n",
       "      <td>81.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>186.21</td>\n",
       "      <td>29.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5104</th>\n",
       "      <td>14180</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>103.08</td>\n",
       "      <td>18.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5106</th>\n",
       "      <td>44873</td>\n",
       "      <td>81.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>125.20</td>\n",
       "      <td>40.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5107</th>\n",
       "      <td>19723</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>82.99</td>\n",
       "      <td>30.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5108</th>\n",
       "      <td>37544</td>\n",
       "      <td>51.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>166.29</td>\n",
       "      <td>25.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5109</th>\n",
       "      <td>44679</td>\n",
       "      <td>44.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>85.28</td>\n",
       "      <td>26.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4909 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id   age  hypertension  heart_disease  avg_glucose_level   bmi\n",
       "0      9046  67.0             0              1             228.69  36.6\n",
       "2     31112  80.0             0              1             105.92  32.5\n",
       "3     60182  49.0             0              0             171.23  34.4\n",
       "4      1665  79.0             1              0             174.12  24.0\n",
       "5     56669  81.0             0              0             186.21  29.0\n",
       "...     ...   ...           ...            ...                ...   ...\n",
       "5104  14180  13.0             0              0             103.08  18.6\n",
       "5106  44873  81.0             0              0             125.20  40.0\n",
       "5107  19723  35.0             0              0              82.99  30.6\n",
       "5108  37544  51.0             0              0             166.29  25.6\n",
       "5109  44679  44.0             0              0              85.28  26.2\n",
       "\n",
       "[4909 rows x 6 columns]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4909, 6)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       1\n",
       "2       1\n",
       "3       1\n",
       "4       1\n",
       "5       1\n",
       "       ..\n",
       "5104    0\n",
       "5106    0\n",
       "5107    0\n",
       "5108    0\n",
       "5109    0\n",
       "Name: stroke, Length: 4909, dtype: int64"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, ..., 0, 0, 0])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.33465284,  1.07013796, -0.31806673,  4.38196829,  2.77769839,\n",
       "         0.98134488],\n",
       "       [-0.28353855,  1.64656262, -0.31806673,  4.38196829,  0.0138418 ,\n",
       "         0.45926914],\n",
       "       [ 1.10121139,  0.27201152, -0.31806673, -0.22820795,  1.48413156,\n",
       "         0.70120668],\n",
       "       ...,\n",
       "       [-0.82605376, -0.34875349, -0.31806673, -0.22820795, -0.50236926,\n",
       "         0.21733161],\n",
       "       [ 0.02284987,  0.36069224, -0.31806673, -0.22820795,  1.37291993,\n",
       "        -0.41934612],\n",
       "       [ 0.36272572,  0.05030973, -0.31806673, -0.22820795, -0.45081569,\n",
       "        -0.34294479]])"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_values = range(1, 31)\n",
    "accuracies = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in k_values:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    accuracies.append(accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44080 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot accuracy vs. k\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(k_values, accuracies, marker='o', linestyle='dashed', color='b')\n",
    "plt.xlabel('Number of Neighbors (k)')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title('kNN Classifier Performance on Breast Cancer Dataset')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aufgabe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Verwende nun die folgenden Datasets und versuche die bestmögliche Performance zu erreichen.\n",
    "* Lade dazu das Dataset mit den bekannten Methoden (Laden mit Hilfe von `pd.read_csv`)\n",
    "* Überlege, wie du bei schlechter Performance diese verbessern kannst. Zum Beipsiel: Normalisieren, Ausreißer entfernen etc.\n",
    "* Müssen wir ggf. Features entfernen?\n",
    "* Gehören ggf. Features mit einem Ordinal-Encoder oder mit einem Onehot-Encoder encodiert?\n",
    "* Verwende für jedes Dataset eigene Code-Zellen und dokumentiere für die verschiedenen Durchläufe die Ergebnisse (zBsp. Accuracy, Confusion Matrix oder den MSE)\n",
    "\n",
    "**Datasets:**\n",
    "* Breast Cancer `breast_cancer.csv` (verwendet von https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic)\n",
    "* Diabetes `diabetes.csv` (verwendet von https://www.kaggle.com/uciml/pima-indians-diabetes-database)\n",
    "* Stroke Prediction `stroke.csv` (verwendet von https://www.kaggle.com/datasets/fedesoriano/stroke-prediction-dataset)\n",
    "\n",
    "*Hinweis:* Überlege dir stets, welchen Problemtyp du verwendest und verwende dementsprechend das richtige Model! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     radius1  texture1  perimeter1   area1  smoothness1  compactness1  \\\n",
      "0      17.99     10.38      122.80  1001.0      0.11840       0.27760   \n",
      "1      20.57     17.77      132.90  1326.0      0.08474       0.07864   \n",
      "2      19.69     21.25      130.00  1203.0      0.10960       0.15990   \n",
      "3      11.42     20.38       77.58   386.1      0.14250       0.28390   \n",
      "4      20.29     14.34      135.10  1297.0      0.10030       0.13280   \n",
      "..       ...       ...         ...     ...          ...           ...   \n",
      "564    21.56     22.39      142.00  1479.0      0.11100       0.11590   \n",
      "565    20.13     28.25      131.20  1261.0      0.09780       0.10340   \n",
      "566    16.60     28.08      108.30   858.1      0.08455       0.10230   \n",
      "567    20.60     29.33      140.10  1265.0      0.11780       0.27700   \n",
      "568     7.76     24.54       47.92   181.0      0.05263       0.04362   \n",
      "\n",
      "     concavity1  concave_points1  symmetry1  fractal_dimension1  ...  \\\n",
      "0       0.30010          0.14710     0.2419             0.07871  ...   \n",
      "1       0.08690          0.07017     0.1812             0.05667  ...   \n",
      "2       0.19740          0.12790     0.2069             0.05999  ...   \n",
      "3       0.24140          0.10520     0.2597             0.09744  ...   \n",
      "4       0.19800          0.10430     0.1809             0.05883  ...   \n",
      "..          ...              ...        ...                 ...  ...   \n",
      "564     0.24390          0.13890     0.1726             0.05623  ...   \n",
      "565     0.14400          0.09791     0.1752             0.05533  ...   \n",
      "566     0.09251          0.05302     0.1590             0.05648  ...   \n",
      "567     0.35140          0.15200     0.2397             0.07016  ...   \n",
      "568     0.00000          0.00000     0.1587             0.05884  ...   \n",
      "\n",
      "     texture3  perimeter3   area3  smoothness3  compactness3  concavity3  \\\n",
      "0       17.33      184.60  2019.0      0.16220       0.66560      0.7119   \n",
      "1       23.41      158.80  1956.0      0.12380       0.18660      0.2416   \n",
      "2       25.53      152.50  1709.0      0.14440       0.42450      0.4504   \n",
      "3       26.50       98.87   567.7      0.20980       0.86630      0.6869   \n",
      "4       16.67      152.20  1575.0      0.13740       0.20500      0.4000   \n",
      "..        ...         ...     ...          ...           ...         ...   \n",
      "564     26.40      166.10  2027.0      0.14100       0.21130      0.4107   \n",
      "565     38.25      155.00  1731.0      0.11660       0.19220      0.3215   \n",
      "566     34.12      126.70  1124.0      0.11390       0.30940      0.3403   \n",
      "567     39.42      184.60  1821.0      0.16500       0.86810      0.9387   \n",
      "568     30.37       59.16   268.6      0.08996       0.06444      0.0000   \n",
      "\n",
      "     concave_points3  symmetry3  fractal_dimension3  Diagnosis  \n",
      "0             0.2654     0.4601             0.11890          M  \n",
      "1             0.1860     0.2750             0.08902          M  \n",
      "2             0.2430     0.3613             0.08758          M  \n",
      "3             0.2575     0.6638             0.17300          M  \n",
      "4             0.1625     0.2364             0.07678          M  \n",
      "..               ...        ...                 ...        ...  \n",
      "564           0.2216     0.2060             0.07115          M  \n",
      "565           0.1628     0.2572             0.06637          M  \n",
      "566           0.1418     0.2218             0.07820          M  \n",
      "567           0.2650     0.4087             0.12400          M  \n",
      "568           0.0000     0.2871             0.07039          B  \n",
      "\n",
      "[569 rows x 31 columns]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 51744 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load dataset (Vehicle Silhouettes dataset)\n",
    "ds = pd.read_csv(\"../../_data/breast_cancer.csv\") # ggf. etwas anders als unser bisheriges Dataset\n",
    "print(ds)\n",
    "ds = ds.dropna()\n",
    "y = ds.Diagnosis\n",
    "X = ds.drop('Diagnosis', axis=1)\n",
    "X = X.select_dtypes(include=[np.number])\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)\n",
    "k_values = range(1, 31)\n",
    "accuracies = []\n",
    "for k in k_values:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    accuracies.append(accuracy_score(y_test, y_pred))\n",
    "# Plot accuracy vs. k\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(k_values, accuracies, marker='o', linestyle='dashed', color='b')\n",
    "plt.xlabel('Number of Neighbors (k)')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title('kNN Classifier Performance on Breast Cancer Dataset')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
      "0              6      148             72             35        0  33.6   \n",
      "1              1       85             66             29        0  26.6   \n",
      "2              8      183             64              0        0  23.3   \n",
      "3              1       89             66             23       94  28.1   \n",
      "4              0      137             40             35      168  43.1   \n",
      "..           ...      ...            ...            ...      ...   ...   \n",
      "763           10      101             76             48      180  32.9   \n",
      "764            2      122             70             27        0  36.8   \n",
      "765            5      121             72             23      112  26.2   \n",
      "766            1      126             60              0        0  30.1   \n",
      "767            1       93             70             31        0  30.4   \n",
      "\n",
      "     DiabetesPedigreeFunction  Age  Outcome  \n",
      "0                       0.627   50        1  \n",
      "1                       0.351   31        0  \n",
      "2                       0.672   32        1  \n",
      "3                       0.167   21        0  \n",
      "4                       2.288   33        1  \n",
      "..                        ...  ...      ...  \n",
      "763                     0.171   63        0  \n",
      "764                     0.340   27        0  \n",
      "765                     0.245   30        0  \n",
      "766                     0.349   47        1  \n",
      "767                     0.315   23        0  \n",
      "\n",
      "[768 rows x 9 columns]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 50120 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load dataset (Vehicle Silhouettes dataset)\n",
    "ds = pd.read_csv(\"../../_data/diabetes.csv\") # ggf. etwas anders als unser bisheriges Dataset\n",
    "print(ds)\n",
    "ds = ds.dropna()\n",
    "y = ds.Outcome\n",
    "X = ds.drop('Outcome', axis=1)\n",
    "X = X.select_dtypes(include=[np.number])\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)\n",
    "k_values = range(1, 31)\n",
    "accuracies = []\n",
    "for k in k_values:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    accuracies.append(accuracy_score(y_test, y_pred))\n",
    "# Plot accuracy vs. k\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(k_values, accuracies, marker='o', linestyle='dashed', color='b')\n",
    "plt.xlabel('Number of Neighbors (k)')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title('kNN Classifier Performance on Breast Cancer Dataset')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dsai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
