{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"\n",
    "    border: 2px solid #4CAF50; \n",
    "    padding: 15px; \n",
    "    background-color: #f4f4f4; \n",
    "    border-radius: 10px; \n",
    "    align-items: center;\">\n",
    "\n",
    "<h1 style=\"margin: 0; color: #4CAF50;\">Supervised ML Modelle: Support-Vector-Machines</h1>\n",
    "<h2 style=\"margin: 5px 0; color: #555;\">DSAI</h2>\n",
    "<h3 style=\"margin: 5px 0; color: #555;\">Jakob Eggl</h3>\n",
    "\n",
    "<div style=\"flex-shrink: 0;\">\n",
    "    <img src=\"https://www.htl-grieskirchen.at/wp/wp-content/uploads/2022/11/logo_bildschirm-1024x503.png\" alt=\"Logo\" style=\"width: 250px; height: auto;\"/>\n",
    "</div>\n",
    "<p1> © 2024/25 Jakob Eggl. Nutzung oder Verbreitung nur mit ausdrücklicher Genehmigung des Autors.</p1>\n",
    "</div>\n",
    "<div style=\"flex: 1;\">\n",
    "</div>   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Teile (adaptiert) von *AI Inside Seminar KI Mag. Otto Reichel*!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machines (SVM's)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nun betrachten wir den Support Vector Machine (SVM) Klassifizierer. Es ist eine der am meisten theoretisch fundierten Machine Learning Modelle.\n",
    "\n",
    "Die SVM $\\ldots$\n",
    "* $\\ldots$ wird bei uns verwendet zur Klassifizierung\n",
    "* $\\ldots$ gibt es in 2 Varianten: **Linear SVM** und **Kernel SVM**\n",
    "* $\\ldots$ hat wenige Hyperparameter ($C$ + Parameter für Kernel SVM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Preview_3_Classes](../resources/SVM_Preview.jpg)\n",
    "\n",
    "(von https://spotintelligence.com/2024/05/06/support-vector-machines-svm/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beschreibung des Algorithmus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SVM's sind Supervised Machine Learning Modelle, die versuchen, Muster in Daten zu erkennen, um die Daten zu klassifizieren. Ähnlich wie beim $k$-Nearest-Neighbour Klassifizierer, erhalten wir auch hier keine Wahrscheinlichkeiten zu bestimmten Klassenzugehörigkeiten, sondern nur die Vorhersage, ob ein bestimmter Datensatz zu einer Klasse gehört oder nicht.\n",
    "\n",
    "Dabei versuchen SVM's Trennlinien (im mehrdimensionalen Hyperebenen) zwischen den Bereichen so zu legen, dass der leere Bereich (=Streifen, genannt **Margin**) möglichst groß wird. Die folgende Abbildung zeigt dies für 2 klar getrennte Bereiche. In der Praxis ist die Ausgangslage meistens nicht so eindeutig, sprich hier ist es meistens *nicht* möglich, ohne Fehler eine Trennlinie/Hyperebene zu finden."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Margin_Visualization](../resources/SVM_Margin.png)\n",
    "\n",
    "(von https://www.iunera.com/kraken/fabric/support-vector-machines-svm/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welche Hyperplane wäre hier die beste?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Multiple_Hyperplanes](../resources/SVM_Multiple_Hyperplanes.jpg)\n",
    "\n",
    "![SVM_Wrong_Hyperplanes](../resources/SVM_Wrong_Hyperplane.jpg)\n",
    "\n",
    "![SVM_Right_Hyperplanes](../resources/SVM_Right_Hyperplane.jpg)\n",
    "\n",
    "(von VL: Supervised Machine Learning JKU WS 2022)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Da der Trennbereich so groß wie möglich sein soll, ist klarerweise die letzte Hyperplane die beste."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training vs. Inferenz von SVM's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wir wollen hier nochmal formal festhalten, wie eine Support Vector Machine an den Daten angepasst wird (Training) und wie sie dann verwendet wird.\n",
    "\n",
    "**Training:**\n",
    "* Die optimale Hyperebene wird berechnet.\n",
    "* Je nach Parameter $C$ werden mehr oder weniger Fehlklassifikationen erlaubt\n",
    "\n",
    "**Verwendung (=Inferenz):**\n",
    "* Ein neuer Datenpunkt wird in den Raum gesetzt\n",
    "* Je nachdem, auf welcher Seite der Punkt liegt, bekommt er das Label $1$ oder $0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiclass SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wie gehen wir vor, wenn wir mehrere Klassen haben?\n",
    "\n",
    "Hier gibt es *mehrere Möglichkeiten*.\n",
    "\n",
    "1. **One-vs-One (OvO)**\n",
    "    * Für $N$ Klassen wird für jede mögliche Paarung von Klassen eine eigene SVM trainiert\n",
    "    * Das bedeutet, es werden $N\\cdot (N-1)/2$​ Klassifikatoren trainiert\n",
    "    * Vorhersage: Jeder Klassifikator gibt eine Stimme für eine der beiden Klassen ab, und die Klasse mit den meisten Stimmen gewinnt (Mehrheitsentscheidung)\n",
    "    * Vorteil: Hohe Genauigkeit\n",
    "    * Nachteil: Teuer (Rechenleistung) bei vielen Klassen\n",
    "\n",
    "2. **One-vs-All (OvA)**\n",
    "    * Für $N$ Klassen werden $N$ SVMs trainiert\n",
    "    * Jede SVM unterscheidet eine Klasse von allen anderen Klassen (z. B. \"Ist es Klasse A oder nicht?\")\n",
    "    * Vorhersage: Die SVM mit der höchsten Entscheidungsfunktion bestimmt die Klasse\n",
    "    * Vorteil: Weniger Modelle als *OvO*\n",
    "    * Nachteil: Schlechtere Performance bei unausgeglichenen Daten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Die Support Vektoren"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nachdem dieses Machine Learning Modell *Support Vector Machine* heißt, ist diese Namensgebung zu hinterfragen sehr legitim. Der Name kommt von den sogenannten **Support Vektoren**.\n",
    "\n",
    "**Support Vektoren** (deutsch: Stützvektoren) sind die Datenpunkte, die am nächsten zur anderen Klasse sind. Diese werden verwendet, um die optimale Hyperebene zu finden. Die Berechnung der Hyperebene werden wir nicht händisch durchmachen, es kann bequem in Python gemacht werden. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Der Hyperparameter $C$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Wird **Regularisierungsparameter** genannt\n",
    "* Kontrolliert den Trade-off zwischen **Maximierung der Margin** und **Minimierung von Fehlklassifikationen**\n",
    "* Höheres $C$: Strengere Klassifikation, weniger Fehlklassifikationen erlaubt und somit kleinere Margin\n",
    "* Niedrigeres $C$: Größere Margin, erlaubt mehr Fehlklassifikationen für bessere Generalisierung"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 248436 characters]",
      "text/plain": [
       "<Figure size 1500x1000 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Dummy Dataset erstellen\n",
    "X, y = make_classification(\n",
    "    n_samples=200, \n",
    "    n_features=2,    # Gesamtzahl der Features\n",
    "    n_informative=2, \n",
    "    n_redundant=0,   # Keine redundanten Features\n",
    "    n_repeated=0,    # Keine wiederholten Features\n",
    "    n_classes=2, \n",
    "    n_clusters_per_class=1, \n",
    "    random_state=42\n",
    ")\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "# Verschiedene C-Werte testen\n",
    "C_values = [0.01, 0.1, 1, 10, 100]\n",
    "plt.figure(figsize=(15, 10))\n",
    "\n",
    "for i, C in enumerate(C_values, 1):\n",
    "    model = SVC(C=C, kernel='linear')\n",
    "    model.fit(X_train, y_train)\n",
    "    \n",
    "    # Plot der Entscheidungsgrenze\n",
    "    plt.subplot(2, 3, i)\n",
    "    plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap='coolwarm', edgecolors='k')\n",
    "    \n",
    "    # Grenzen für die Darstellung\n",
    "    ax = plt.gca()\n",
    "    xlim = ax.get_xlim()\n",
    "    ylim = ax.get_ylim()\n",
    "    \n",
    "    xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 100),\n",
    "                         np.linspace(ylim[0], ylim[1], 100))\n",
    "    Z = model.decision_function(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    \n",
    "    plt.contour(xx, yy, Z, levels=[-1, 0, 1], linestyles=['dashed', 'solid', 'dashed'], colors='black')\n",
    "    plt.title(f'C = {C}')\n",
    "    \n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welche Werte von $C$ begünstigen *Overfitting* und welche *Underfitting*?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kernel SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Als Motivation wird nun folgendes Szenario präsentiert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Annahme wir wollen folgende 2 Klassen mit Hilfe von einer Linie separieren."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Motivation_1d](../resources/Kernel_SVM_Motivation.jpg)\n",
    "\n",
    "(von https://medium.com/towards-data-science/svm-and-kernel-svm-fed02bef1200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dies ist nicht möglich. Jedoch wäre es einfach, wenn wir auch nicht-lineare Separierungen zulassen würden. Dies ist equivalent zu folgendem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Motivation_1d](../resources/Kernel_SVM_Motivation_Solution.jpg)\n",
    "\n",
    "(von https://medium.com/towards-data-science/svm-and-kernel-svm-fed02bef1200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dies bringt uns zu den sogenannten **Kernel SVM's**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Da für komplizierte Datasets das finden einer *Ebene* nahezu unmöglich ist, gibt es eine Erweiterung zu den klassischen Support Vector Machines. Diese werden **kernel SVM's** genannt. Dabei werden die Datenpunkte mit Hilfe einer *kernel-Funktion* in einen höherdimensionalen Raum gebracht. Anschließend wird eine Hyperebene gesucht, welche die Daten separiert. Dieser Vorgang ist in den folgenden Bildern visualisiert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Kernel_SVM_Preview](../resources/Kernel_SVM_Preview.jpg)\n",
    "\n",
    "![Kernel_SVM_Preview_2](../resources/Kernel_SVM_Preview2.jpg)\n",
    "\n",
    "(beide von https://medium.com/@abhishekjainindore24/svm-kernels-and-its-type-dfc3d5f2dcd8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Wichtig:** Mit *Kernel SVM* schaffen wir eine nichtlineare Separierung der Daten im ursprünglichen Raum. Der Umweg über einen höher-dimensionalen Raum, wo wir linear separieren können, beschreibt nur den Vorgang, der im Hintergrund passiert. Insgesamt sind wir trotzdem noch in eine (in diesem Fall *nichtlineare*) Separierung der Daten interessiert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Info:** Die Funktionsweise der *Kernel-SVM's* passiert mit dem sogenannten **Kernel-Trick**. Dieser erlaubt uns die effiziente Separierung auch in hohen Dimensionen am Computer. Dies wird hier nicht genauer erläutert."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Die verschiedenen Kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linearer Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "    \\phi(x,y)= x\\cdot y\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Linear_Kernel](../resources/SVM_linear_kernel.jpg)\n",
    "\n",
    "(von *Mathematical Pictures at a Data Science Exhibition*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Polynomial Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "\t\t\\phi(x, y) := \\left(x\\cdot y +\\beta\\right)^\\alpha,\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Polynomial_Kernel](../resources/SVM_Polynomial_Kernel3.png)\n",
    "\n",
    "(von *Mathematical Pictures at a Data Science Exhibition*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RBF Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "    \\phi(x,y) = \\exp\\left(-\\frac{1}{2\\sigma^2}\\vert x-y\\vert^2\\right)\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_RBF_Kernel](../resources/SVM_Gaussian_Kernel.png)\n",
    "\n",
    "(von *Mathematical Pictures at a Data Science Exhibition*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "    \\phi(x,y) = \\tanh(\\alpha (x\\cdot y) + \\beta)\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SVM_Sigmoid_Kernel](../resources/SVM_sigmoid_kernel.jpg)\n",
    "\n",
    "(von [Link](https://www.linkedin.com/pulse/support-vector-machine-explained-theory-visualization-zixuan-zhang/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vorteile und Nachteile von (Kernel) SVM's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Vorteile:**\n",
    "* Gute theoretische Fundierung (Eingeschränkte Optimierung)\n",
    "* Schnelles Training und Testing\n",
    "* Gute Performance bei hoher Anzahl an Features und nur begrenzter Anzahl an Datenpunkten\n",
    "\n",
    "**Nachteile:**\n",
    "* Schlecht, wenn Daten mit viel Rauschen überlagert sind\n",
    "* Wenig geeignet für große Datasets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tipps und Tricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Normalisierung (zBsp. $z$-score Normalisierung) der Daten auch hier erforderlich\n",
    "* Wähle den richtigen Kernel!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Kernel) SVM's in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = datasets.load_iris()\n",
    "X = iris.data[:, :2]  # Wir verwenden nur die ersten zwei Merkmale für bessere Visualisierung\n",
    "y = iris.target\n",
    "\n",
    "# Aufteilen der Daten in Trainings- und Testset (80% Training, 20% Test)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5.1, 3.5],\n",
       "       [4.9, 3. ],\n",
       "       [4.7, 3.2],\n",
       "       [4.6, 3.1],\n",
       "       [5. , 3.6],\n",
       "       [5.4, 3.9],\n",
       "       [4.6, 3.4],\n",
       "       [5. , 3.4],\n",
       "       [4.4, 2.9],\n",
       "       [4.9, 3.1],\n",
       "       [5.4, 3.7],\n",
       "       [4.8, 3.4],\n",
       "       [4.8, 3. ],\n",
       "       [4.3, 3. ],\n",
       "       [5.8, 4. ],\n",
       "       [5.7, 4.4],\n",
       "       [5.4, 3.9],\n",
       "       [5.1, 3.5],\n",
       "       [5.7, 3.8],\n",
       "       [5.1, 3.8],\n",
       "       [5.4, 3.4],\n",
       "       [5.1, 3.7],\n",
       "       [4.6, 3.6],\n",
       "       [5.1, 3.3],\n",
       "       [4.8, 3.4],\n",
       "       [5. , 3. ],\n",
       "       [5. , 3.4],\n",
       "       [5.2, 3.5],\n",
       "       [5.2, 3.4],\n",
       "       [4.7, 3.2],\n",
       "       [4.8, 3.1],\n",
       "       [5.4, 3.4],\n",
       "       [5.2, 4.1],\n",
       "       [5.5, 4.2],\n",
       "       [4.9, 3.1],\n",
       "       [5. , 3.2],\n",
       "       [5.5, 3.5],\n",
       "       [4.9, 3.6],\n",
       "       [4.4, 3. ],\n",
       "       [5.1, 3.4],\n",
       "       [5. , 3.5],\n",
       "       [4.5, 2.3],\n",
       "       [4.4, 3.2],\n",
       "       [5. , 3.5],\n",
       "       [5.1, 3.8],\n",
       "       [4.8, 3. ],\n",
       "       [5.1, 3.8],\n",
       "       [4.6, 3.2],\n",
       "       [5.3, 3.7],\n",
       "       [5. , 3.3],\n",
       "       [7. , 3.2],\n",
       "       [6.4, 3.2],\n",
       "       [6.9, 3.1],\n",
       "       [5.5, 2.3],\n",
       "       [6.5, 2.8],\n",
       "       [5.7, 2.8],\n",
       "       [6.3, 3.3],\n",
       "       [4.9, 2.4],\n",
       "       [6.6, 2.9],\n",
       "       [5.2, 2.7],\n",
       "       [5. , 2. ],\n",
       "       [5.9, 3. ],\n",
       "       [6. , 2.2],\n",
       "       [6.1, 2.9],\n",
       "       [5.6, 2.9],\n",
       "       [6.7, 3.1],\n",
       "       [5.6, 3. ],\n",
       "       [5.8, 2.7],\n",
       "       [6.2, 2.2],\n",
       "       [5.6, 2.5],\n",
       "       [5.9, 3.2],\n",
       "       [6.1, 2.8],\n",
       "       [6.3, 2.5],\n",
       "       [6.1, 2.8],\n",
       "       [6.4, 2.9],\n",
       "       [6.6, 3. ],\n",
       "       [6.8, 2.8],\n",
       "       [6.7, 3. ],\n",
       "       [6. , 2.9],\n",
       "       [5.7, 2.6],\n",
       "       [5.5, 2.4],\n",
       "       [5.5, 2.4],\n",
       "       [5.8, 2.7],\n",
       "       [6. , 2.7],\n",
       "       [5.4, 3. ],\n",
       "       [6. , 3.4],\n",
       "       [6.7, 3.1],\n",
       "       [6.3, 2.3],\n",
       "       [5.6, 3. ],\n",
       "       [5.5, 2.5],\n",
       "       [5.5, 2.6],\n",
       "       [6.1, 3. ],\n",
       "       [5.8, 2.6],\n",
       "       [5. , 2.3],\n",
       "       [5.6, 2.7],\n",
       "       [5.7, 3. ],\n",
       "       [5.7, 2.9],\n",
       "       [6.2, 2.9],\n",
       "       [5.1, 2.5],\n",
       "       [5.7, 2.8],\n",
       "       [6.3, 3.3],\n",
       "       [5.8, 2.7],\n",
       "       [7.1, 3. ],\n",
       "       [6.3, 2.9],\n",
       "       [6.5, 3. ],\n",
       "       [7.6, 3. ],\n",
       "       [4.9, 2.5],\n",
       "       [7.3, 2.9],\n",
       "       [6.7, 2.5],\n",
       "       [7.2, 3.6],\n",
       "       [6.5, 3.2],\n",
       "       [6.4, 2.7],\n",
       "       [6.8, 3. ],\n",
       "       [5.7, 2.5],\n",
       "       [5.8, 2.8],\n",
       "       [6.4, 3.2],\n",
       "       [6.5, 3. ],\n",
       "       [7.7, 3.8],\n",
       "       [7.7, 2.6],\n",
       "       [6. , 2.2],\n",
       "       [6.9, 3.2],\n",
       "       [5.6, 2.8],\n",
       "       [7.7, 2.8],\n",
       "       [6.3, 2.7],\n",
       "       [6.7, 3.3],\n",
       "       [7.2, 3.2],\n",
       "       [6.2, 2.8],\n",
       "       [6.1, 3. ],\n",
       "       [6.4, 2.8],\n",
       "       [7.2, 3. ],\n",
       "       [7.4, 2.8],\n",
       "       [7.9, 3.8],\n",
       "       [6.4, 2.8],\n",
       "       [6.3, 2.8],\n",
       "       [6.1, 2.6],\n",
       "       [7.7, 3. ],\n",
       "       [6.3, 3.4],\n",
       "       [6.4, 3.1],\n",
       "       [6. , 3. ],\n",
       "       [6.9, 3.1],\n",
       "       [6.7, 3.1],\n",
       "       [6.9, 3.1],\n",
       "       [5.8, 2.7],\n",
       "       [6.8, 3.2],\n",
       "       [6.7, 3.3],\n",
       "       [6.7, 3. ],\n",
       "       [6.3, 2.5],\n",
       "       [6.5, 3. ],\n",
       "       [6.2, 3.4],\n",
       "       [5.9, 3. ]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SVM-Modell mit RBF-Kernel\n",
    "svm_model = SVC(kernel='rbf', gamma='scale', C=1.0)\n",
    "svm_model = svm_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9\n"
     ]
    }
   ],
   "source": [
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_decision_boundary(X, y, model):\n",
    "    h = .02  # Schrittweite für das Meshgrid\n",
    "    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "\n",
    "    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    plt.contourf(xx, yy, Z, alpha=0.3)\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', marker='o')\n",
    "    plt.xlabel('Feature 1')\n",
    "    plt.ylabel('Feature 2')\n",
    "    plt.title('SVM Decision Boundary')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 62848 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_boundary(X_train, y_train, svm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Was passiert bei einem linearen Kernel?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(kernel=&#x27;linear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;SVC<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.svm.SVC.html\">?<span>Documentation for SVC</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>SVC(kernel=&#x27;linear&#x27;)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "SVC(kernel='linear')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_model = SVC(kernel='linear', C=1.0)\n",
    "svm_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9\n"
     ]
    }
   ],
   "source": [
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 62940 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_boundary(X_train, y_train, svm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aufgabe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Verwende nun die folgenden Datasets und versuche die bestmögliche Performance zu erreichen.\n",
    "* Lade dazu das Dataset mit den bekannten Methoden (Laden mit Hilfe von `pd.read_csv`)\n",
    "* Überlege, wie du bei schlechter Performance diese verbessern kannst. Zum Beipsiel: Normalisieren, Ausreißer entfernen etc.\n",
    "* Müssen wir ggf. Features entfernen?\n",
    "* Gehören ggf. Features mit einem Ordinal-Encoder oder mit einem Onehot-Encoder encodiert?\n",
    "* Verwende für jedes Dataset eigene Code-Zellen und dokumentiere für die verschiedenen Durchläufe die Ergebnisse (zBsp. Accuracy, Confusion Matrix oder den MSE)\n",
    "\n",
    "**Datasets:**\n",
    "* Breast Cancer `breast_cancer.csv` (verwendet von https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic)\n",
    "* Diabetes `diabetes.csv` (verwendet von https://www.kaggle.com/uciml/pima-indians-diabetes-database)\n",
    "* Stroke Prediction `stroke.csv` (verwendet von https://www.kaggle.com/datasets/fedesoriano/stroke-prediction-dataset)\n",
    "\n",
    "*Hinweis:* Überlege dir stets, welchen Problemtyp du verwendest und verwende dementsprechend das richtige Model! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9824561403508771\n",
      "Accuracy: 0.956140350877193\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "ds = pd.read_csv(\"../../_data/breast_cancer.csv\") # ggf. etwas anders als unser bisheriges Dataset\n",
    "ds = ds.dropna()\n",
    "y = ds.Diagnosis\n",
    "X = ds.drop('Diagnosis', axis=1)\n",
    "X = X.select_dtypes(include=[np.number])\n",
    "\n",
    "# Aufteilen der Daten in Trainings- und Testset (80% Training, 20% Test)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "# SVM-Modell mit RBF-Kernel\n",
    "svm_model = SVC(kernel='rbf', gamma='scale', C=100)\n",
    "svm_model = svm_model.fit(X_train, y_train)\n",
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
    "# plot_decision_boundary(X_train, y_train, svm_model)\n",
    "\n",
    "svm_model = SVC(kernel='linear', C=1.0)\n",
    "svm_model.fit(X_train, y_train)\n",
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
    "# plot_decision_boundary(X_train, y_train, svm_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.7662337662337663\n",
      "Accuracy: 0.7532467532467533\n"
     ]
    }
   ],
   "source": [
    "ds = pd.read_csv(\"../../_data/diabetes.csv\") # ggf. etwas anders als unser bisheriges Dataset\n",
    "ds = ds.dropna()\n",
    "y = ds.Outcome\n",
    "X = ds.drop('Outcome', axis=1)\n",
    "X = X.select_dtypes(include=[np.number])\n",
    "\n",
    "# Aufteilen der Daten in Trainings- und Testset (80% Training, 20% Test)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "# SVM-Modell mit RBF-Kernel\n",
    "svm_model = SVC(kernel='rbf', gamma='scale', C=1.0)\n",
    "svm_model = svm_model.fit(X_train, y_train)\n",
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
    "# plot_decision_boundary(X_train, y_train, svm_model)\n",
    "\n",
    "svm_model = SVC(kernel='linear', C=1.0)\n",
    "svm_model.fit(X_train, y_train)\n",
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
    "# plot_decision_boundary(X_train, y_train, svm_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9460285132382892\n",
      "Accuracy: 0.945010183299389\n"
     ]
    }
   ],
   "source": [
    "ds = pd.read_csv(\"../../_data/stroke.csv\") # ggf. etwas anders als unser bisheriges Dataset\n",
    "ds = ds.dropna()\n",
    "y = ds.stroke\n",
    "X = ds.drop('stroke', axis=1)\n",
    "X = X.select_dtypes(include=[np.number])\n",
    "\n",
    "# Aufteilen der Daten in Trainings- und Testset (80% Training, 20% Test)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "# SVM-Modell mit RBF-Kernel\n",
    "svm_model = SVC(kernel='rbf', gamma='scale', C=1.0)\n",
    "svm_model = svm_model.fit(X_train, y_train)\n",
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
    "# plot_decision_boundary(X_train, y_train, svm_model)\n",
    "\n",
    "svm_model = SVC(kernel='linear', C=1.0)\n",
    "svm_model.fit(X_train, y_train)\n",
    "# Vorhersagen auf dem Testset\n",
    "y_pred = svm_model.predict(X_test)\n",
    "\n",
    "# Evaluierung der Genauigkeit\n",
    "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
    "# plot_decision_boundary(X_train, y_train, svm_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_decision_boundary(X, y, model):\n",
    "    h = .02  # Schrittweite für das Meshgrid\n",
    "    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "\n",
    "    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    plt.contourf(xx, yy, Z, alpha=0.3)\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', marker='o')\n",
    "    plt.xlabel('Feature 1')\n",
    "    plt.ylabel('Feature 2')\n",
    "    plt.title('SVM Decision Boundary')\n",
    "    plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dsai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
