Stop tracking .venv

Decision_tree.ipynb

@@ -4,11 +4,13 @@
    "cell_type": "markdown",
    "id": "946e852d",
    "metadata": {},
-   "source": []
+   "source": [
+    "### Toymodel as an initial test"
+   ]
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": null,
    "id": "0952f099",
    "metadata": {},
    "outputs": [
@@ -46,26 +48,10 @@
       "   macro avg       0.74      0.75      0.74      6033\n",
       "weighted avg       0.81      0.81      0.81      6033\n",
       "\n",
-      "<class 'pandas.core.frame.DataFrame'>\n",
-      "Index: 30162 entries, 1 to 32560\n",
-      "Data columns (total 13 columns):\n",
-      " #   Column          Non-Null Count  Dtype\n",
-      "---  ------          --------------  -----\n",
-      " 0   age             30162 non-null  int64\n",
-      " 1   workclass       30162 non-null  int64\n",
-      " 2   education.num   30162 non-null  int64\n",
-      " 3   marital.status  30162 non-null  int64\n",
-      " 4   occupation      30162 non-null  int64\n",
-      " 5   relationship    30162 non-null  int64\n",
-      " 6   race            30162 non-null  int64\n",
-      " 7   sex             30162 non-null  int64\n",
-      " 8   capital.gain    30162 non-null  int64\n",
-      " 9   capital.loss    30162 non-null  int64\n",
-      " 10  hours.per.week  30162 non-null  int64\n",
-      " 11  native.country  30162 non-null  int64\n",
-      " 12  income          30162 non-null  int64\n",
-      "dtypes: int64(13)\n",
-      "memory usage: 3.2 MB\n"
+      "Index(['age', 'workclass', 'education.num', 'marital.status', 'occupation',\n",
+      "       'relationship', 'race', 'sex', 'capital.gain', 'capital.loss',\n",
+      "       'hours.per.week', 'native.country'],\n",
+      "      dtype='object')\n"
      ]
     }
    ],
@@ -80,6 +66,8 @@
     "from sklearn.metrics import mean_squared_error, mean_absolute_error , r2_score\n",
     "from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay\n",
     "from scipy.stats import randint\n",
+    "from sklearn.naive_bayes import NB\n",
+    "\n",
     "\n",
     "\n",
     "\n",
@@ -107,8 +95,8 @@
     "y = df_encoded['income']\n",
     "\n",
     "# Split the data\n",
-    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
-    "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=42)\n",
+    "X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
+    "X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.25, random_state=42)\n",
     "\n",
     "\n",
     "# Build pipeline\n",
@@ -148,7 +136,7 @@
     "print(\"Classification Report:\")\n",
     "print(classification_report(y_val, y_pred, target_names=[\"Poor\", \"Rich\"]))\n",
     "\n",
-    "df_encoded.info()"
+    "print(X_train_val.columns)"
    ]
   },
   {
@@ -156,12 +144,13 @@
    "id": "bfaae28c",
    "metadata": {},
    "source": [
-    "### Hyperparameter tuning"
+    "### Hyperparameter tuning\n",
+    "I think this section is pretty unecessary. It can probably be removed, but I'm going to keep it for now, just in case. "
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 12,
    "id": "e567e4e9",
    "metadata": {},
    "outputs": [
@@ -172,47 +161,32 @@
       "Classification Report:\n",
       "              precision    recall  f1-score   support\n",
       "\n",
-      "        Poor       0.88      0.93      0.90      4524\n",
-      "        Rich       0.75      0.62      0.68      1509\n",
+      "        Poor       0.89      0.93      0.91      4524\n",
+      "        Rich       0.74      0.64      0.69      1509\n",
       "\n",
-      "    accuracy                           0.85      6033\n",
-      "   macro avg       0.81      0.78      0.79      6033\n",
-      "weighted avg       0.85      0.85      0.85      6033\n",
+      "    accuracy                           0.86      6033\n",
+      "   macro avg       0.81      0.78      0.80      6033\n",
+      "weighted avg       0.85      0.86      0.85      6033\n",
       "\n",
-      "Best max_depth: 12\n",
-      "Best min_samples_split: 18\n"
+      "Best max_depth: 11\n",
+      "Best min_samples_split: 3\n",
+      "Best min_samples_leaf: 4\n"
      ]
     }
    ],
    "source": [
-    "# Hyperparameters search space\n",
-    "param_dist = {\n",
-    "    'full_dt_classifier__max_depth': randint(3, 20),\n",
-    "    'full_dt_classifier__min_samples_split': randint(2, 20),\n",
-    "    'full_dt_classifier__min_samples_leaf': randint(1, 15),\n",
-    "}\n",
+    "param_grid = {\n",
+    "                'full_dt_classifier__max_depth': [None, 10, 11, 12, 13, 14, 15, 16],\n",
+    "                'full_dt_classifier__min_samples_split': [3, 4, 5, 6, 7, 8, 9],\n",
+    "                'full_dt_classifier__min_samples_leaf': [1, 2, 3, 4, 5]\n",
+    "            }\n",
+    "skf = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)\n",
+    "grid_search = GridSearchCV(model, param_grid, scoring='accuracy', cv=skf, n_jobs=-1)\n",
+    "grid_search.fit(X_train, y_train)\n",
     "\n",
-    "# Ranodmized search for hyperparameter tuning\n",
-    "random_search = RandomizedSearchCV(\n",
-    "    estimator=model,\n",
-    "    param_distributions=param_dist,\n",
-    "    n_iter = 300,\n",
-    "    cv = 10,\n",
-    "    scoring = 'r2',\n",
-    "    n_jobs = -1,\n",
-    "    random_state = 42\n",
-    ")\n",
-    "\n",
-    "#K_fold = KFold(n_splits=10, shuffle=True, random_state=42)\n",
-    "#cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)\n",
-    "#grid_search = GridSearchCV(model, param_grid, scoring='accuracy', cv=cv, n_jobs=-1)\n",
-    "\n",
-    "# Fit search\n",
-    "random_search.fit(X_train, y_train)\n",
-    "random_search.fit(X_train, y_train)\n",
     "\n",
     "# Best model training\n",
-    "best_model = random_search.best_estimator_\n",
+    "best_model = grid_search.best_estimator_\n",
     "y_pred_best = best_model.predict(X_val)\n",
     "\n",
     "print(\"Classification Report:\")\n",
@@ -223,7 +197,8 @@
     "best_min_samples_leaf = best_model.named_steps['full_dt_classifier'].min_samples_leaf\n",
     "\n",
     "print(f'Best max_depth: {best_max_depth}')\n",
-    "print(f'Best min_samples_split: {best_min_samples_split}')\n"
+    "print(f'Best min_samples_split: {best_min_samples_split}')\n",
+    "print(f'Best min_samples_leaf: {best_min_samples_leaf}')"
    ]
   },
   {
@@ -232,7 +207,7 @@
    "metadata": {},
    "source": [
     "### Classifier comparison\n",
-    "(This is taken from the ensemble lab)"
+    "This is taken from the ensemble lab. A lot of models are excluded, but they are easy to implement into the class, for now It's kept short to minimize the execution time. We can also tune more parameters for each model, but of course that would also increase the execution time."
    ]
   },
   {
@@ -255,7 +230,7 @@
     "from catboost import CatBoostClassifier  # Requires installation of the package; Not a native function in sklearn\n",
     "\n",
     "class ClassifierComparisonOpt:\n",
-    "    def __init__(self, X, y, test_size=0.3, use_bootstrap=True, random_state=42, cv_folds=10):\n",
+    "    def __init__(self, X, y, test_size=0.25, use_bootstrap=True, random_state=42, cv_folds=10):\n",
     "        # Split data stratified by labels\n",
     "        X_train, X_test, y_train, y_test = train_test_split(\n",
     "            X, y, test_size=test_size, stratify=y, random_state=random_state)\n",
@@ -265,6 +240,10 @@
     "        self.X_train = scaler.fit_transform(X_train)\n",
     "        self.X_test  = scaler.transform(X_test)\n",
     "\n",
+    "        # Scaling not necessary for decision trees\n",
+    "        #self.X_train = X_train\n",
+    "        #self.X_test = X_test\n",
+    "\n",
     "        self.y_train       = y_train\n",
     "        self.y_test        = y_test\n",
     "        self.use_bootstrap = use_bootstrap\n",
@@ -272,55 +251,31 @@
     "        self.models        = {}\n",
     "        self.results       = {}\n",
     "        self.results_df    = None\n",
+    "        self.feature_names = list(X.columns)\n",
     "\n",
-    "    # Define the classifiers and their hyperparameters\n",
     "    def get_models_with_params(self):\n",
     "        return {\n",
     "            'Decision Tree': (DecisionTreeClassifier(random_state=42), {\n",
-    "                'max_depth': [None, 10, 11, 12, 13, 14, 15, 16],\n",
-    "                'min_samples_split': [3, 4, 5, 6, 7, 8, 9]\n",
+    "                'max_depth': [None, 5, 10, 20],\n",
+    "                'min_samples_split': [2, 5, 10],\n",
+    "                #'min_samples_leaf': [2, 4, 6]\n",
     "            }),\n",
     "            'Random Forest': (RandomForestClassifier(bootstrap=self.use_bootstrap, random_state=42), {\n",
-    "                'n_estimators': [50, 100, 200], # I currently don't know how to tune this\n",
-    "                'max_depth': [None, 10, 11, 12, 13, 14, 15, 16],\n",
-    "                'min_samples_split': [3, 4, 5, 6, 7, 8, 9]\n",
-    "            }),\n",
-    "            #'Bagging': (BaggingClassifier(bootstrap=self.use_bootstrap, random_state=42), {\n",
-    "            #    'n_estimators': [50, 100, 200],\n",
-    "            #    'max_samples': [0.5, 1.0],\n",
-    "            #    'oob_score': [True, False],\n",
-    "            #}),\n",
-    "            # SAMME: Stagewise Additive Modeling using a Multi-class Exponential loss\n",
-    "            #'AdaBoost': (AdaBoostClassifier(algorithm='SAMME', random_state=42), {\n",
-    "            #    'n_estimators': [50, 100, 200],\n",
-    "            #    'learning_rate': [0.01, 0.1, 0.2, 0.5, 1.0]\n",
-    "            #}),\n",
-    "            #'Gradient Boosting': (GradientBoostingClassifier(random_state=42), {\n",
-    "            #    'n_estimators': [50, 100, 200],\n",
-    "            #    'learning_rate': [0.01, 0.1, 0.2, 0.5, 1.0],\n",
-    "            #    'max_depth': [3, 5, 10]\n",
-    "            #}),\n",
-    "            #'XGBoost': (XGBClassifier(eval_metric='mlogloss', random_state=42), {\n",
-    "            #    'n_estimators': [50, 100, 200],\n",
-    "            #    'learning_rate': [0.01, 0.1, 0.2, 0.5, 1.0],\n",
-    "            #    'max_depth': [3, 5, 10]\n",
-    "            #}),\n",
-    "            #'LightGBM': (LGBMClassifier(random_state=42), {\n",
-    "            #    'n_estimators': [50, 100, 200],\n",
-    "            #    'learning_rate': [0.01, 0.1, 0.2, 0.5, 1.0],\n",
-    "            #    'max_depth': [-1, 5, 10]\n",
-    "            #}),\n",
-    "            #'CatBoost': (CatBoostClassifier(verbose=0, random_state=42), {\n",
-    "            #    'iterations': [50, 100, 200],\n",
-    "            #    'learning_rate': [0.01, 0.1, 0.2, 0.5, 1.0],\n",
-    "            #    'depth': [4, 6, 10]\n",
-    "            #})\n",
+    "                'n_estimators': [50, 100, 200],\n",
+    "                'max_depth': [None, 5, 10, 20],\n",
+    "                'min_samples_split': [2, 5, 10],\n",
+    "                #'min_samples_leaf': [2, 4, 6]\n",
+    "            })\n",
     "        }\n",
     "\n",
-    "    def fit_models(self):\n",
+    "    def fit_models(self, models_with_params=None):\n",
+    "        # Set self.get_models_with_params() as a default parameter distribution\n",
+    "        if models_with_params == None:\n",
+    "            models_with_params = self.get_models_with_params()\n",
+    "\n",
     "        results_list = []\n",
     "        cv = StratifiedKFold(n_splits=self.cv_folds, shuffle=True, random_state=42)\n",
-    "        models_with_params = self.get_models_with_params()\n",
+    "        #models_with_params = self.get_models_with_params()\n",
     "\n",
     "        for name, (model, param_grid) in models_with_params.items():\n",
     "            print(f\"Tuning {name} ...\")\n",
@@ -366,6 +321,7 @@
     "    # Show feature importance for models that support it          \n",
     "    def show_feature_importance(self):\n",
     "        importance = {}\n",
+    "        features = self.feature_names\n",
     "\n",
     "        for name, result in self.models.items():\n",
     "            model = result['model']\n",
@@ -382,9 +338,10 @@
     "\n",
     "        for name, imp in importance.items():\n",
     "            sorted_idx = np.argsort(imp)[::-1]\n",
+    "            sorted_features = [features[i] for i in sorted_idx]\n",
     "            plt.figure()\n",
     "            plt.bar(range(len(imp)), imp[sorted_idx], align='center')\n",
-    "            plt.xticks(range(len(imp)), sorted_idx)\n",
+    "            plt.xticks(range(len(imp)), sorted_features, rotation=70)\n",
     "            plt.title(f\"Feature importance for {name}\")\n",
     "            plt.xlabel(\"Feature index\")\n",
     "            plt.ylabel(\"Importance score\")\n",
@@ -392,9 +349,17 @@
     "            plt.show()\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "da377fc1",
+   "metadata": {},
+   "source": [
+    "### Compare models with coarse hyperparameter tuning"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 58,
    "id": "d6fd1fee",
    "metadata": {},
    "outputs": [
@@ -406,19 +371,60 @@
       "Tuning Random Forest ...\n",
       "\n",
       "------ Results Sorted by Accuracy ------\n",
-      "        Model  Accuracy  Precision   Recall  F1 Score                                                    Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
-      "Random Forest  0.857617   0.852147 0.857617  0.852588 {'max_depth': 16, 'min_samples_split': 5, 'n_estimators': 200}         303.932920             0.216372      304.149293\n",
-      "Decision Tree  0.846381   0.840787 0.846381  0.842219                      {'max_depth': 10, 'min_samples_split': 9}           3.946769             0.001706        3.948475\n",
+      "        Model  Accuracy  Precision   Recall  F1 Score                                                     Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
+      "Random Forest  0.858777   0.853301 0.858777  0.853292 {'max_depth': 20, 'min_samples_split': 10, 'n_estimators': 100}         186.568059             0.070525      186.638585\n",
+      "Decision Tree  0.848168   0.844754 0.848168  0.846079                       {'max_depth': 10, 'min_samples_split': 5}          11.661536             0.001839       11.663375\n",
       "\n",
       "------ Results Sorted by Total Time ------\n",
-      "        Model  Accuracy  Precision   Recall  F1 Score                                                    Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
-      "Decision Tree  0.846381   0.840787 0.846381  0.842219                      {'max_depth': 10, 'min_samples_split': 9}           3.946769             0.001706        3.948475\n",
-      "Random Forest  0.857617   0.852147 0.857617  0.852588 {'max_depth': 16, 'min_samples_split': 5, 'n_estimators': 200}         303.932920             0.216372      304.149293\n"
+      "        Model  Accuracy  Precision   Recall  F1 Score                                                     Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
+      "Decision Tree  0.848168   0.844754 0.848168  0.846079                       {'max_depth': 10, 'min_samples_split': 5}          11.661536             0.001839       11.663375\n",
+      "Random Forest  0.858777   0.853301 0.858777  0.853292 {'max_depth': 20, 'min_samples_split': 10, 'n_estimators': 100}         186.568059             0.070525      186.638585\n"
+     ]
+    }
+   ],
+   "source": [
+    "# optimized classifiers and find their hyperparameters\n",
+    "clf_opt = ClassifierComparisonOpt(X_train_val, y_train_val)\n",
+    "clf_opt.fit_models()\n",
+    "clf_opt.print_summary()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dc529395",
+   "metadata": {},
+   "source": [
+    "### Finetune and compare again\n",
+    "After tuning with a coarse grid we can use the result to select a finer grid and find better hyperparameters. This will only slightly improve the models, you will generally get diminishing returns the finer you tune."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 57,
+   "id": "4185a428",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tuning Decision Tree ...\n",
+      "Tuning Random Forest ...\n",
+      "\n",
+      "------ Results Sorted by Accuracy ------\n",
+      "        Model  Accuracy  Precision   Recall  F1 Score                                                     Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
+      "Random Forest  0.860766   0.855462 0.860766  0.854443 {'max_depth': 15, 'min_samples_split': 10, 'n_estimators': 100}         145.080616             0.069650      145.150266\n",
+      "Decision Tree  0.849163   0.843062 0.849163  0.839388                        {'max_depth': 8, 'min_samples_split': 6}           1.626699             0.001421        1.628121\n",
+      "\n",
+      "------ Results Sorted by Total Time ------\n",
+      "        Model  Accuracy  Precision   Recall  F1 Score                                                     Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
+      "Decision Tree  0.849163   0.843062 0.849163  0.839388                        {'max_depth': 8, 'min_samples_split': 6}           1.626699             0.001421        1.628121\n",
+      "Random Forest  0.860766   0.855462 0.860766  0.854443 {'max_depth': 15, 'min_samples_split': 10, 'n_estimators': 100}         145.080616             0.069650      145.150266\n"
      ]
     },
     {
      "data": {
-      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAhzhJREFUeJztnXtcVGX+xz/nDMKAAgJyD+SiiRpqabreu5hobmluqV1WpHR3LbeMss0s70ZamVu52mW72c1dM9v9VZhR2pbm3SK8hFwEFUQF5BYgc57fHzRHhpmBeZBhvsT3/XrNS/nOM2fe33Mehu8853nOUYQQAgzDMAzDMB0I1dUCDMMwDMMwbQ0XQAzDMAzDdDi4AGIYhmEYpsPBBRDDMAzDMB0OLoAYhmEYhulwcAHEMAzDMEyHgwsghmEYhmE6HFwAMQzDMAzT4eACiGEYhmGYDgcXQAxDnLfeeguKoiA3N9fVKsyvZGZmYuzYsfD19YWiKNiyZYurlZxKVFQUZsyYIfWaxYsXQ1EU5wgxTCvABRBDDvMffFuPxx9/3CnvuXPnTixevBilpaVO2X5HpqqqCosXL8b27dtdrdJqJCYmIj09HStWrMCGDRswaNAgp71Xbm6uxe9Ap06d0K1bNwwbNgxPPPEE8vLynPbe7Y0ZM2bY/exo+JAt5pjfJgrfC4yhxltvvYWkpCQsXboU0dHRFs9dddVVGDBgQKu/53PPPYd58+YhJycHUVFRrb79y8FkMuHixYvw8PBol9+oz507h8DAQCxatAiLFy92tc5l88svv8DLywsLFizA8uXLnf5+ubm5iI6Oxp133ombb74ZmqahpKQEe/fuxebNm6EoCv75z39i2rRpTnOoqamBqqro1KmTw6+pq6tDXV0djEaj07was2vXLmRlZek/5+TkYOHChfjTn/6EkSNH6vHY2FgMHTq0zbwYmri5WoBh7DF+/HinfrNuCyorK9G5c+fL2obBYIDBYGglo7ZD0zTU1ta6WqPVOXv2LACga9eurbZNR/rJNddcg3vuucciduLECYwdOxaJiYno3bs3+vfv32pODfHw8JB+jZubG9zc2vZPzNChQy0Km3379mHhwoUYOnSo1b5rSGv8njLtDz4FxrRbPv/8c4wcORKdO3eGt7c3JkyYgIyMDIs2P/74I2bMmIGYmBgYjUaEhITg3nvvxfnz5/U2ixcvxrx58wAA0dHR+jB5bm6ufvrhrbfesnp/RVEsRjTMcx4OHz6Mu+66C35+fhgxYoT+/LvvvouBAwfC09MT/v7+mDZtGvLz85vN09YcoKioKPz+97/H9u3bMWjQIHh6eiI+Pl4/zbR582bEx8fDaDRi4MCBOHjwoMU2Z8yYgS5duiA7OxsJCQno3LkzwsLCsHTpUjQeFK6srMQjjzyCiIgIeHh4oFevXnjuuees2imKgjlz5uC9995D37594eHhgfXr1yMwMBAAsGTJEn3fmvebI8en4b49fvw4ZsyYga5du8LX1xdJSUmoqqqy2mfvvvsuBg8eDC8vL/j5+WHUqFH44osvLNo40n8as3jxYnTv3h0AMG/ePCiKYjFiePDgQYwfPx4+Pj7o0qULbrzxRnz//fcW2zAfzx07duD+++9HUFAQrrjiiibf1x7du3fHW2+9hdraWqxatcriudLSUsydO1c/bj169MDKlSuhaZpFO03T8Pe//13vL4GBgRg3bhz27dunt2k8B+jixYtYsmQJevbsCaPRiICAAIwYMQLbtm2z2FeNRyzr6uqwbNkyxMbGwsPDA1FRUXjiiSdQU1Nj0c7cv7/99lsMHjwYRqMRMTExeOedd1q0nxrS3P53tF8cPXoUt99+O/z9/WE0GjFo0CD85z//uWw/pu3gESCGLBcuXMC5c+csYt26dQMAbNiwAYmJiUhISMDKlStRVVWFdevWYcSIETh48KD+R2nbtm3Izs5GUlISQkJCkJGRgVdffRUZGRn4/vvvoSgKJk+ejJ9//hkffPABXnjhBf09AgMD9W/7Mtxxxx3o2bMnnn76ab1IWLFiBZ566ilMmTIFM2fOxNmzZ/HSSy9h1KhROHjwYItGE44fP4677roLf/7zn3HPPffgueeewy233IL169fjiSeewP333w8ASElJwZQpU3Ds2DGo6qXvPCaTCePGjcPvfvc7rFq1CqmpqVi0aBHq6uqwdOlSAIAQArfeeiu+/vpr3HfffRgwYAC2bt2KefPm4dSpU3jhhRcsnL766iv861//wpw5c9CtWzf0798f69atw+zZs3Hbbbdh8uTJAIB+/foBcOz4NGTKlCmIjo5GSkoKDhw4gNdffx1BQUFYuXKl3mbJkiVYvHgxhg0bhqVLl8Ld3R27d+/GV199hbFjxwJwvP80ZvLkyejatSsefvhh/ZRUly5dAAAZGRkYOXIkfHx88Nhjj6FTp0545ZVXcN1112HHjh0YMmSIxbbuv/9+BAYGYuHChaisrJQ69g0ZOnQoYmNjLYqPqqoqjB49GqdOncKf//xnREZGYufOnZg/fz4KCgqwZs0ave19992Ht956C+PHj8fMmTNRV1eH//3vf/j+++/tjsAuXrwYKSkpmDlzJgYPHoyysjLs27cPBw4cwE033WTXdebMmXj77bdx++2345FHHsHu3buRkpKCI0eO4OOPP7Zoe/z4cdx+++247777kJiYiDfeeAMzZszAwIED0bdv3xbvLzO29r+j/SIjIwPDhw9HeHg4Hn/8cXTu3Bn/+te/MGnSJHz00Ue47bbbLtuPaQMEwxDjzTffFABsPoQQory8XHTt2lXMmjXL4nWFhYXC19fXIl5VVWW1/Q8++EAAEN98840ee/bZZwUAkZOTY9E2JydHABBvvvmm1XYAiEWLFuk/L1q0SAAQd955p0W73NxcYTAYxIoVKyzi6enpws3NzSpub380dOvevbsAIHbu3KnHtm7dKgAIT09PceLECT3+yiuvCADi66+/1mOJiYkCgPjrX/+qxzRNExMmTBDu7u7i7NmzQgghtmzZIgCI5cuXWzjdfvvtQlEUcfz4cYv9oaqqyMjIsGh79uxZq31lxtHjY9639957r0Xb2267TQQEBOg/Z2ZmClVVxW233SZMJpNFW03ThBBy/ccW5j7x7LPPWsQnTZok3N3dRVZWlh47ffq08Pb2FqNGjdJj5uM5YsQIUVdX1+R7NfV+DZk4caIAIC5cuCCEEGLZsmWic+fO4ueff7Zo9/jjjwuDwSDy8vKEEEJ89dVXAoB48MEHrbZp3l9C1Pe3xMRE/ef+/fuLCRMmNOltPmZmDh06JACImTNnWrR79NFHBQDx1VdfWbxf4z5QVFQkPDw8xCOPPNLk+zZk7969Vr+/9va/TL+48cYbRXx8vKiurtZjmqaJYcOGiZ49ezrsx7gWPgXGkGXt2rXYtm2bxQOoHzUoLS3FnXfeiXPnzukPg8GAIUOG4Ouvv9a34enpqf+/uroa586dw+9+9zsAwIEDB5zi/Ze//MXi582bN0PTNEyZMsXCNyQkBD179rTwlaFPnz4W8x3MIww33HADIiMjreLZ2dlW25gzZ47+f/MprNraWnz55ZcAgM8++wwGgwEPPvigxeseeeQRCCHw+eefW8RHjx6NPn36OJyD7PFpvG9HjhyJ8+fPo6ysDACwZcsWaJqGhQsXWox2mfMD5PqPo5hMJnzxxReYNGkSYmJi9HhoaCjuuusufPvtt7qjmVmzZrXa3C7zKFR5eTkA4N///jdGjhwJPz8/ixzHjBkDk8mEb775BgDw0UcfQVEULFq0yGqbTU2479q1KzIyMpCZmemw42effQYASE5Otog/8sgjAIBPP/3UIt6nTx+LicuBgYHo1auXzX7cEhrvf0f7RXFxMb766itMmTIF5eXlervz588jISEBmZmZOHXqVKs4Ms6FT4ExZBk8eLDNIXjzh+4NN9xg83U+Pj76/4uLi7FkyRJ8+OGHKCoqsmh34cKFVrS9ROOVa5mZmRBCoGfPnjbby6ysaUjDIgcAfH19AQARERE24yUlJRZxVVUt/lgDwJVXXgkA+nyjEydOICwsDN7e3hbtevfurT/fkMa5N4fs8Wmcs5+fH4D63Hx8fJCVlQVVVZsswmT6j6OcPXsWVVVV6NWrl9VzvXv3hqZpyM/Ptzh1I7uvmqKiogIA9OOUmZmJH3/8UZ9/1Rjzvs7KykJYWBj8/f2l3m/p0qWYOHEirrzySlx11VUYN24c/vjHP+qnNm1x4sQJqKqKHj16WMRDQkLQtWtXq77U+FgD9ce7cT9uKbZ+T4Hm+8Xx48chhMBTTz2Fp556ymbboqIihIeHt4on4zy4AGLaHeZJnBs2bEBISIjV8w1XnkyZMgU7d+7EvHnzMGDAAHTp0gWapmHcuHFWk0FtYe9bsMlksvuahqMaZl9FUfD555/b/MZv/vYui73RA3tx0QZXvGice3PIHp/WyE2m/zgT2X3VFD/99BOCgoL0P9KapuGmm27CY489ZrO9udBtKaNGjUJWVhY++eQTfPHFF3j99dfxwgsvYP369Zg5c2aTr3X0Ug7O7se2fk+B5vuFud2jjz6KhIQEm9tuXOQxNOECiGl3xMbGAgCCgoIwZswYu+1KSkqQlpaGJUuWYOHChXrc1rC9vQ9l8whD4wskNv622pyvEALR0dGX/YenNdE0DdnZ2RZOP//8MwDokz27d++OL7/8EuXl5RajQEePHtWfbw57+1bm+DhKbGwsNE3D4cOH7V4vytH+I0NgYCC8vLxw7Ngxq+eOHj0KVVWtRuZaC/O1bxou846NjUVFRUWz+cXGxmLr1q0oLi6WHgXy9/dHUlISkpKSUFFRgVGjRmHx4sV2C6Du3btD0zRkZmbqI4gAcObMGZSWljrUl5yJo/3CPGraqVOnVus/jGvgOUBMuyMhIQE+Pj54+umncfHiRavnzSu3zN8gG39jbLgCxoz5GiCNCx0fHx9Line truncated
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAIhCAYAAACmMgXJAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAA6hpJREFUeJzsnXtYVNX6x797DzdRUC4CiigXDRRRTNO8Z2lalrfy1ul4OWkdy0op/WUXL2lhVuapLLtZZqfynDLrZJFlaRfNu0cENQQRlJsKCIKAzH5/f3Bmy8AMzMIZZhHv53l4lHfW7P1Zey32vLP2WnsrRERgGIZhGIZpRqjOFmAYhmEYhmlsOAFiGIZhGKbZwQkQwzAMwzDNDk6AGIZhGIZpdnACxDAMwzBMs4MTIIZhGIZhmh2cADEMwzAM0+zgBIhhGIZhmGYHJ0AMwzAMwzQ7OAFiGMn54IMPoCgK0tPTna3C/I+UlBTceuutaN26NRRFwZYtW5yt5FBCQ0MxY8YMofcsXboUiqI4Rohh7AAnQIx0mD7wLf088cQTDtnnrl27sHTpUhQWFjpk+82Z0tJSLF26FDt27HC2it2YPn06EhMT8dxzz2Hjxo3o06ePw/aVnp5u9jfg6uoKf39/DBgwAE8++SQyMjIctu+mxowZM6yeO6r/iCZzzJ8ThZ8FxsjGBx98gJkzZ+LZZ59FWFiY2Wvdu3dHbGys3ff50ksvYcGCBTh16hRCQ0Ptvv1rwWg04sqVK3B3d2+S36jPnz+Ptm3bYsmSJVi6dKmzda6Zy5cvw9PTE0899RRWrFjh8P2lp6cjLCwMU6dOxe233w5N01BQUIB9+/Zh8+bNUBQF7733HqZMmeIwh/LycqiqCldXV5vfU1lZicrKSnh4eDjMqya7d+9Gamqq/vupU6ewePFi3H///Rg8eLAej4iIQP/+/RvNi5ETF2cLMIw1brvtNod+s24MSkpK0LJly2vahsFggMFgsJNR46FpGioqKpytYXfOnTsHAGjTpo3dtmlLP7n++utx7733msVOnz6NW2+9FdOnT0fXrl3Rs2dPuzlVx93dXfg9Li4ucHFp3I+Y/v37myU2+/fvx+LFi9G/f/9ax6469vg7ZZoefAmMabJ8++23GDx4MFq2bAkvLy+MHj0aSUlJZmWOHDmCGTNmIDw8HB4eHggKCsLf/vY3XLhwQS+zdOlSLFiwAAAQFhamD5Onp6frlx8++OCDWvtXFMVsRMM05yE5ORn33HMPfHx8MGjQIP31jz76CL1790aLFi3g6+uLKVOmIDMzs956WpoDFBoaijvuuAM7duxAnz590KJFC8TExOiXmTZv3oyYmBh4eHigd+/eOHTokNk2Z8yYgVatWiEtLQ0jR45Ey5Yt0b59ezz77LOoOShcUlKCxx57DCEhIXB3d0dkZCReeumlWuUURcHcuXPxz3/+E9HR0XB3d8e6devQtm1bAMCyZcv0Y2s6bra0T/Vje/LkScyYMQNt2rRB69atMXPmTJSWltY6Zh999BH69u0LT09P+Pj4YMiQIdi2bZtZGVv6T02WLl2KTp06AQAWLFgARVHMRgwPHTqE2267Dd7e3mjVqhVuueUW/P7772bbMLXnzp078eCDDyIgIAAdOnSoc7/W6NSpEz744ANUVFRg1apVZq8VFhZi3rx5ert17twZL7zwAjRNMyunaRr+8Y9/6P2lbdu2GDVqFPbv36+XqTkH6MqVK1i2bBm6dOkCDw8P+Pn5YdCgQfj+++/NjlXNEcvKykosX74cERERcHd3R2hoKJ588kmUl5eblTP1719//RV9+/aFh4cHwsPD8eGHHzboOFWnvuNva784fvw47r77bvj6+sLDwwN9+vTBV199dc1+TOPBI0CMtFy8eBHnz583i/n7+wMANm7ciOnTp2PkyJF44YUXUFpaijfffBODBg3CoUOH9A+l77//HmlpaZg5cyaCgoKQlJSEt99+G0lJSfj999+hKAomTJiAP/74A5988gleeeUVfR9t27bVv+2LMHHiRHTp0gXPP/+8niQ899xzeOaZZzBp0iTMmjUL586dw2uvvYYhQ4bg0KFDDRpNOHnyJO655x488MADuPfee/HSSy/hzjvvxLp16/Dkk0/iwQcfBADEx8dj0qRJOHHiBFT16nceo9GIUaNG4cYbb8SqVauQkJCAJUuWoLKyEs8++ywAgIgwZswY/PTTT7jvvvsQGxuL7777DgsWLMDZs2fxyiuvmDn9+OOP+Ne//oW5c+fC398fPXv2xJtvvok5c+Zg/PjxmDBhAgCgR48eAGxrn+pMmjQJYWFhiI+Px8GDB/Huu+8iICAAL7zwgl5m2bJlWLp0KQYMGIBnn30Wbm5u2LNnD3788UfceuutAGzvPzWZMGEC2rRpg/nz5+uXpFq1agUASEpKwuDBg+Ht7Y2FCxfC1dUVb731Fm666Sbs3LkT/fr1M9vWgw8+iLZt22Lx4sUoKSkRavvq9O/fHxEREWbJR2lpKYYOHYqzZ8/igQceQMeOHbFr1y4sWrQI2dnZWLNmjV72vvvuwwcffIDbbrsNs2bNQmVlJX755Rf8/vvvVkdgly5divj4eMyaNQt9+/ZFUVER9u/fj4MHD2LEiBFWXWfNmoUNGzbg7rvvxmOPPYY9e/YgPj4ex44dwxdffGFW9uTJk7j77rtx3333Yfr06Vi/fj1mzJiB3r17Izo6usHHy4Sl429rv0hKSsLAgQMRHByMJ554Ai1btsS//vUvjBs3Dp9//jnGjx9/zX5MI0AMIxnvv/8+AbD4Q0RUXFxMbdq0odmzZ5u9Lycnh1q3bm0WLy0trbX9Tz75hADQzz//rMdefPFFAkCnTp0yK3vq1CkCQO+//36t7QCgJUuW6L8vWbKEANDUqVPNyqWnp5PBYKDnnnvOLJ6YmEguLi614taOR3W3Tp06EQDatWuXHvvuu+8IALVo0YJOnz6tx9966y0CQD/99JMemz59OgGghx9+WI9pmkajR48mNzc3OnfuHBERbdmyhQDQihUrzJzuvvtuUhSFTp48aXY8VFWlpKQks7Lnzp2rdaxM2No+pmP7t7/9zazs+PHjyc/PT/89JSWFVFWl8ePHk9FoNCuraRoRifUfS5j6xIsvvmgWHzduHLm5uVFqaqoey8rKIi8vLxoyZIgeM7XnoEGDqLKyss591bW/6owdO5YA0MWLF4mIaPny5dSyZUv6448/zMo98cQTZDAYKCMjg4iIfvzxRwJAjzzySK1tmo4XUVV/mz59uv57z549afTo0XV6m9rMxOHDhwkAzZo1y6zc448/TgDoxx9/NNtfzT6Ql5dH7u7u9Nhjj9W53+rs27ev1t+vteMv0i9uueUWiomJobKyMj2maRoNGDCAunTpYrMf41z4EhgjLWvXrsX3339v9gNUjRoUFhZi6tSpOH/+vP5jMBjQr18//PTTT/o2WrRoof+/rKwM58+fx4033ggAOHjwoEO8//73v5v9vnnzZmiahkmTJpn5BgUFoUuXLma+InTr1s1svoNphOHmm29Gx44da8XT0tJqbWPu3Ln6/02XsCoqKvDDDz8AAL755hsYDAY88sgjZu977LHHQET49ttvzeJDhw5Ft27dbK6DaPvUPLaDBw/GhQsXUFRUBADYsmULNE3D4sWLzUa7TPUDxPqPrRiNRmzbtg3jxo1DeHi4Hm/Xrh3uuece/Prrr7qjidmzZ9ttbpdpFKq4uBgA8O9//xuDBw+Gj4+PWR2HDx8Oo9GIn3/+GQDw+eefQ1EULFmypNY265pw36ZNGyQlJSElJcVmx2+++QYAEBcXZxZ/7LHHAABbt241i3fr1s1s4nLbtm0RGRlpsR83hJrH39Z+kZ+fjx9//BGTJk1CcXGxXu7ChQsYOXIkUlJScPbsWbs4Mo6FL4Ex0tK3b1+LQ/Cmk+7NN99s8X3e3t76//Pz87Fs2TJ8+umnyMvLMyt38eJFO9pepebKtZSUFBARunTpYrG8yMqa6lRPcgCgdevWAICQkBCL8YKCArO4qqpmH9YAcN111wGAPt/o9OnTaN++Pby8vMzKde3aVX+9OjXrXh+i7VOzzj4+PgCq6ubt7Y3U1FSoqlpnEibSf2zl3LlzKC0tRWRkZK3XunbtCk3TkJmZaXbpRvRY1cWlS5cAQG+nlJQUHDlyRJ9/VRPTsU5NTUX79u3h6+srtL9nn30WY8eOxXXXXYfu3btj1KhR+Otf/6pf2rTE6dOnoaoqOnfubBYPCgpCmzZtavWlmm0NVLV3zX7cUCz9nQL194uTJ0+CiPDMM8/gmWeesVg2Ly8PwcHBdvFkHAcnQEyTwzSJc+PGjQgKCqr1evWVJ5MmTcKuXbuwYMECxMbGolWrVtA0DaNGjao1GdQS1r4FG41Gq++pPqph8lUUBd9++63Fb/ymb++iWBs9sBanRrjjRc2614do+9ijbiL9x5GIHqu6OHr0KAICAvQPaU3TMGLECCxcuNBieVOi21CGDBmC1NRUfPnll9i2bRveffddvPLKK1i3bh1mzZpV53ttvZWDo/uxpb9ToP5+YSr3+OOPY+TIkRa3XTPJY+SEEyCmyREREQEACAgIwPDhw62WKygowPbt27Fs2TIsXrxYj1satrd2UjaNMNS8QWLNb6v1+RIRwsLCrvmDx55omoa0tDQzpz/++AMA9MmenTp1wg8//IDi4mKzUaDjx4/rr9eHtWMr0j62EhERAU3TkJycbPV+Ubb2HxHatm0LT09PnDhxotZrx48fh6qqtUbm7IXp3jfVl3lHRETg0qVL9dYvIiIC3333HfLz84VHgXx9fTFz5kzMnDkTly5dwpAhQ7B06VKrCVCnTp2gaRpSUlL0EUQAyM3NRWFhoU19yZHY2i9Mo6aurq526z+Mc+A5QEyTY+TIkfD29sbzzz+PK1eu1HrdtHLL9A2y5jfG6itgTJjuAVIz0fH29oa/v78+Z8LLine truncated
       "text/plain": [
        "<Figure size 640x480 with 1 Axes>"
       ]
@@ -428,7 +434,7 @@
     },
     {
      "data": {
-      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAHHCAYAAABEEKc/AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAkwRJREFUeJztnXlcVFX/xz/3Dg6LLAoKqLFr4IIbKlmmmTyiWWaaptnjktlTaaX8HnsySyUrTNN8StM2bTV9KrOnRcowtXLfElEIWQSVRQUFIUDmnt8f01yYDebgDHPk+b5fr3kpZ87ceX/PPXPnO+eec6/EGGMgCIIgCIIgVGRnCxAEQRAEQYgGJUgEQRAEQRAmUIJEEARBEARhAiVIBEEQBEEQJlCCRBAEQRAEYQIlSARBEARBECZQgkQQBEEQBGECJUgEQRAEQRAmUIJEEARBEARhAiVIBHGD8MEHH0CSJOTm5jpbhfiLzMxMDB8+HD4+PpAkCVu3bnW2UrND/ZJoqVCCRAiL4cBr6fHss8865D337NmDxYsX4/Llyw7Z/v8ylZWVWLx4MXbu3OlsFbsxdepUpKam4uWXX8bHH3+Mfv36Oey9cnNzjT4DsizD19cXI0eOxN69ex32vjcapu1U/3HLLbc4W88iGzduxKpVq5ytQZjg4mwBgmiMF198EWFhYUZlPXr0cMh77dmzB4mJiZg2bRratGnjkPdoKn//+98xceJEuLq6OlulSVRWViIxMREAcMcddzhXxg78+eef2Lt3LxYsWIDZs2c32/tOmjQJd911F3Q6Hf744w+89dZbGDp0KA4ePIjo6Ohm8xAdQzvVp3379k6yaZiNGzfixIkTmDNnjrNViHpQgkQIz8iRIx36y7w5qKioQOvWra9rGxqNBhqNxk5GzYeiKKipqXG2ht25cOECANg1kbaln/Tt2xcPPfSQ+vftt9+OkSNHYu3atXjrrbfs5nKjY9pO9qKqqgparRayTCdgWjq0h4kbnm3btuH2229H69at4eXlhVGjRiEtLc2ozvHjxzFt2jSEh4fDzc0NgYGBePjhh3Hp0iW1zuLFizFv3jwAQFhYmDosn5ubqw7bf/DBB2bvL0kSFi9ebLQdSZJw8uRJPPjgg2jbti0GDRqkPv/JJ58gJiYG7u7u8PX1xcSJE5Gfn99onJbmeoSGhuLuu+/Gzp070a9fP7i7uyM6Olo9jbVlyxZER0fDzc0NMTExOHr0qNE2p02bBk9PT2RnZyM+Ph6tW7dGx44d8eKLL4IxZlS3oqIC//d//4egoCC4uroiMjISr732mlk9SZIwe/ZsfPrpp+jevTtcXV2xbt069dd7YmKi2raGdrNl/9Rv29OnT6ujfD4+Ppg+fToqKyvN2uyTTz7BgAED4OHhgbZt22Lw4MH48ccfjerY0n9MWbx4MUJCQgAA8+bNgyRJCA0NVZ8/evQoRo4cCW9vb3h6emLYsGHYt2+f0TYM+3PXrl144okn4O/vj5tuuqnB97XE7bffDgDIysoyKt+wYQPuvPNO+Pv7w9XVFd26dcPatWvNXm/oQ7/++isGDBgANzc3hIeH46OPPjKrm5aWhjvvvBPu7u646aab8NJLL0FRFIteb731lrr/O3bsiFmzZpmdur7jjjvQo0cPHD9+HEOGDIGHhwc6d+6ML774AgCwa9cuxMbGwt3dHZGRkfjpp5+428ca2dnZGD9+PHx9feHh4YFbbrkF3333nVGdnTt3QpIkbNq0Cc8//zw6deoEDw8PlJWVAQD279+PESNGwMfHBx4eHhgyZAh+++03o22Ul5djzpw5CA0NhaurK/z9/fG3v/0NR44cUdvgu+++w5kzZ9TPRf2+RDgPGkEihOfKlSu4ePGiUVm7du0AAB9//DGmTp2K+Ph4vPrqq6isrMTatWsxaNAgHD16VD3QbN++HdnZ2Zg+fToCAwORlpaGd955B2lpadi3bx8kScLYsWPxxx9/4LPPPsPrr7+uvkf79u3V0QIexo8fjy5duuCVV15Rk4iXX34ZL7zwAiZMmIBHHnkEFy5cwJtvvonBgwfj6NGjTRqNOH36NB588EH84x//wEMPPYTXXnsN99xzD9atW4fnnnsOTzzxBAAgKSkJEyZMQEZGhtGvX51OhxEjRuCWW27BsmXLkJycjEWLFqG2thYvvvgiAIAxhtGjR+Pnn3/GjBkz0Lt3b/zwww+YN28ezp07h9dff93IaceOHfjPf/6D2bNno127dujVqxfWrl2Lxx9/HPfddx/Gjh0LAOjZsycA2/ZPfSZMmICwsDAkJSXhyJEjeO+99+Dv749XX31VrZOYmIjFixfj1ltvxYsvvgitVov9+/djx44dGD58OADb+48pY8eORZs2bTB37lz1VI6npycAfRJx++23w9vbG8888wxatWqFt99+G3fccYf6hV+fJ554Au3bt8fChQtRUVHBte8BqAlz27ZtjcrXrl2L7t27Y/To0XBxccE333yDJ554AoqiYNasWUZ1T58+jfvvvx8zZszA1KlTsX79ekybNg0xMTHo3r07AKCwsBBDhw5FbW0tnn32WbRu3RrvvPMO3N3dzZwWL16MxMRExMXF4fHHH0dGRgbWrl2LgwcP4rfffkOrVq3UuqWlpbj77rsxceJEjB8/HmvXrsXEiRPx6aefYs6cOXjsscfw4IMPYvny5bj//vuRn58PLy+vRtulsrLS7Ljh4+ODVq1aoaioCLfeeisqKyvx1FNPwc/PDx9++CFGjx6NL774Avfdd5/R65YsWQKtVot//vOfqK6uhlarxY4dOzBy5EjExMRg0aJFkGVZTUp/+eUXDBgwAADw2GOP4YsvvsDs2bPRrVs3XLp0Cb/++itOnTqFvn37YsGCBbhy5QrOnj2rfo4MfYlwMowgBGXDhg0MgMUHY4yVl5ezNm3asJkzZxq9rrCwkPn4+BiVV1ZWmm3/s88+YwDY7t271bLly5czACwnJ8eobk5ODgPANmzYYLYdAGzRokXq34sWLWIA2KRJk4zq5ebmMo1Gw15++WWj8tTUVObi4mJWbq096ruFhIQwAGzPnj1q2Q8//MAAMHd3d3bmzBm1/O2332YA2M8//6yWTZ06lQFgTz75pFqmKAobNWoU02q17MKFC4wxxrZu3coAsJdeesnI6f7772eSJLHTp08btYcsyywtLc2o7oULF8zayoCt+8fQtg8//LBR3fvuu4/5+fmpf2dmZjJZltl9993HdDqdUV1FURhjfP3HEoY+sXz5cqPyMWPGMK1Wy7KystSy8+fPMy8vLzZ48GC1zLA/Bw0axGpraxt8r/rvl5iYyC5cuMAKCwvZL7/8wvr3788AsM8//9yovqU2jY+PZ+Hh4UZlhj5Uv52Li4uZq6sr+7//+z+1bM6cOQwA279/v1E9Hx8fo35ZXFzMtFotGz58uFHbr169mgFg69evV8uGDBnCALCNGzeqZenp6Wof2rdvn1pu6NeWPoOW2snSw9D3DbH88ssv6uvKy8tZWFgYCw0NVb1//vlnBoCFh4cbtaeiKKxLly4sPj5e7U+M6ds8LCyM/e1vf1PLfHx82KxZsxp0HjVqFAsJCWmwDtH80Ck2QnjWrFmD7du3Gz0A/ajD5cuXMWnSJFy8eFF9aDQaxMbG4ueff1a3Uf9XblVVFS5evKiuaDEMddubxx57zOjvLVu2QFEUTJgwwcg3MDAQXbp0MfLloVu3bhg4cKD6t2GE4s4770RwcLBZeXZ2ttk26k8yNpwiq6mpUU9pfP/999BoNHjqqaeMXvd///d/YIxh27ZtRuVDhgxBt27dbI6Bd/+Ytu3tt9+OS5cuqac+tm7dCkVRsHDhQrO5IobRKJ7+Yys6nQ4//vgjxowZg/DwcLW8Q4cOePDBB/Hrr7+qjgZmzpzJNbds0aJFaN++PQIDA3H77bfj1KlTWLFiBe6//36jevXb1DAKO2TIEGRnZ+PKlStGdbt166aeqgP0o6aRkZFGfeX777/HLbfcoo6MGOpNnjzZaFs//fQTampqMGfOHKO2nzlzJry9vc1OY3l6emLixInq35GRkWjTpg26du1qNNrWUP+1xKOPPmp23OjVq5cay4ABA4xOfXt6euLRRx9Fbm4uTp48abStqVOnGrXnsWPHkJmZiQcffBCXLl1S+05FRQWGDRuG3bt3q6ce27Rpg/379+P8+fM2eRPiQKfYCOEZMGCAxUnamZmZAPSJgCW8vb3V/5eUlCAxMRGbNm1CcXGxUT3TLwt7YbryLjMzE4wxdOnSxWL9+qcdeKifBAH60wgAEBQUZLG8tLTUqFyWZaMvcwC4+eabAdSdvjlz5gw6duxodmqja9eu6vP1MY29MXj3j2nMhtNLpaWl8Pb2RlZWFmRZbjBJ4+k/tnLhwgVUVlYiMjLS7LmuXbtCURTk5+erp60A/rZ69NFHMX78eFRVVWHHjh144403oNPpzOr99ttvWLRoEfbu3Ws2P+vKlStqfwDM2xPQt2n9vnLmzBmz04MAzGI19AXTcq1Wi/DwcLO+ctNNN5mdQvXx8bG5/1qjS5cuiIuLs/ictVjq9+f6K2UtfZYBfeJkjStXrqBt27ZYtmwZpk6diqCgIMTExOCuu+7ClClTzD5zhHhQgkTcsBh+oX388ccIDAw0e97Fpa57T5gwAXv27MG8efPQu3dveHp6QlEUjBgxwuok0/qYHsANWPpiMmA6N0NRFEiShG3btlkcMWjqvANrow/WypnJpGpHYGleSkPw7h97xMbTfxwJb1vV/+K/++67odFo8Oyzz2Lo0KHqD4msrCwMGzYMUVFRWLlyJYKCgqDVavH999/j9ddfN2tTZ/YVEfuvKZY+ywCwfPly9O7d2+JrDJ/nCRMm4Pbbb8dXX32FH3/8EcuXL8err76KLVu2YOTIkQ71Jq4PSpCIG5aIiAgAgL+/v9VfioD+F2dKSgoSExOxcOFCtdzwK7A+1hIhwwiLine truncated
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAIhCAYAAAC15UU9AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAA91JJREFUeJzsnX9cVFX+/1/3DgKigIACSiiihoqkhmmWaaal5mb+SNNq/bGubZaZ0tZqlj+y0srMLS23H7Zum+mncu2HxWaaWok/8sdXGpU1EEERUEFBEEbmvr9/THNhfsEcnGGO9H4+HjzK95y59/k+58ydM+fec69CRASGYRiGYRhGR/W1AMMwDMMwjGzwAIlhGIZhGMYOHiAxDMMwDMPYwQMkhmEYhmEYO3iAxDAMwzAMYwcPkBiGYRiGYezgARLDMAzDMIwdPEBiGIZhGIaxgwdIDMMwDMMwdvAAiWGuEf75z39CURRkZ2f7WoX5jePHj+Ouu+5CaGgoFEXBpk2bfK3U4HC/ZBorPEBipMV64HX2N2fOHK/sc9euXVi4cCEuXLjgle3/nikvL8fChQuxfft2X6t4jEmTJiE9PR0vvvgiPvzwQ/Tq1ctr+8rOzrb5DKiqivDwcAwbNgxpaWle2++1hn091fy7+eabfa3nlHXr1mHFihW+1mDs8PO1AMPUxfPPP4/27dvbxLp16+aVfe3atQuLFi3C5MmT0aJFC6/so7788Y9/xPjx4xEQEOBrlXpRXl6ORYsWAQBuv/1238p4gMuXLyMtLQ3z5s3DjBkzGmy/EyZMwN133w2z2Yz//e9/eOuttzBw4EDs27cPSUlJDeYhO9Z6qkmrVq18ZFM769atwy+//IJZs2b5WoWpAQ+QGOkZNmyYV3+ZNwRlZWVo1qzZVW3DYDDAYDB4yKjh0DQNJpPJ1xoe5+zZswDg0YG0O/3kxhtvxEMPPaT/+7bbbsOwYcPw9ttv46233vKYy7WOfT15ioqKCvj7+0NV+QRMY4dbmLnm+eabb3DbbbehWbNmCA4OxvDhw2E0Gm3KHD58GJMnT0Z8fDwCAwMRHR2NP/3pTzh//rxeZuHChXjqqacAAO3bt9en5bOzs/Vp+3/+858O+1cUBQsXLrTZjqIoOHLkCB544AGEhYWhX79++uv//ve/kZycjKZNmyI8PBzjx49Hbm5unXk6u9YjLi4Of/jDH7B9+3b06tULTZs2RVJSkn4aa+PGjUhKSkJgYCCSk5Nx8OBBm21OnjwZzZs3R1ZWFoYMGYJmzZqhTZs2eP7550FENmXLysrw5JNPIjY2FgEBAUhISMCyZcscyimKghkzZuCjjz5CYmIiAgICsHr1av3X+6JFi/S6tdabO+1Ts25//fVXfZYvNDQUU6ZMQXl5uUOd/fvf/0bv3r0RFBSEsLAw9O/fH99++61NGXf6jz0LFy5Eu3btAABPPfUUFEVBXFyc/vrBgwcxbNgwhISEoHnz5hg0aBB2795tsw1re+7YsQOPPvooIiMjcd1119W6X2fcdtttAIDMzEyb+AcffIA77rgDkZGRCAgIQNeuXfH22287vN/ah3788Uf07t0bgYGBiI+Px7/+9S+HskajEXfccQeaNm2K6667Di+88AI0TXPq9dZbb+nt36ZNGzz22GMOp65vv/12dOvWDYcPH8aAAQMQFBSEjh074tNPPwUA7NixA3369EHTpk2RkJCA7777Trh+XJGVlYWxY8ciPDwcQUFBuPnmm7F582abMtu3b4eiKFi/fj2effZZxMTEICgoCCUlJQCAPXv2YOjQoQgNDUVQUBAGDBiAn376yWYbpaWlmDVrFuLi4hAQEIDIyEjceeedOHDggF4HmzdvxsmTJ/XPRc2+xPgOnkFipOfixYs4d+6cTaxly5YAgA8//BCTJk3CkCFD8PLLL6O8vBxvv/02+vXrh4MHD+oHmi1btiArKwtTpkxBdHQ0jEYj3nnnHRiNRuzevRuKomD06NH43//+h48//hivv/66vo9WrVrpswUijB07Fp06dcJLL72kDyJefPFFPPfccxg3bhz+/Oc/4+zZs3jzzTfRv39/HDx4sF6zEb/++iseeOAB/OUvf8FDDz2EZcuW4Z577sHq1avxzDPP4NFHHwUALFmyBOPGjUNGRobNr1+z2YyhQ4fi5ptvxiuvvILU1FQsWLAAVVVVeP755wEARIQRI0bg+++/x9SpU9GjRw/897//xVNPPYXTp0/j9ddft3Hatm0b/u///g8zZsxAy5Yt0b17d7z99tuYPn06Ro0ahdGjRwMAbrjhBgDutU9Nxo0bh/bt22PJkiU4cOAA3nvvPURGRuLll1/WyyxatAgLFy7ELbfcgueffx7+/v7Ys2cPtm3bhrvuuguA+/3HntGjR6NFixaYPXu2fiqnefPmACyDiNtuuw0hISF4+umn0aRJE/zjH//A7bffrn/h1+TRRx9Fq1atMH/+fJSVlQm1PQB9wBwWFmYTf/vtt5GYmIgRI0bAz88PX375JR599FFomobHHnvMpuyvv/6K++67D1OnTsWkSZOwZs0aTJ48GcnJyUhMTAQA5OfnY+DAgaiqqsKcOXPQrFkzvPPOO2jatKmD08KFC7Fo0SIMHjwY06dPR0ZGBt5++23s27cPP/30E5o0aaKXLS4uxh/+8AeMHz8eY8eOxdtvv43x48fjo48+wqxZs/DII4/ggQcewKuvvor77rsPubm5CA4OrrNeysvLHY4boaGhaNKkCQoKCnDLLbegvLwcM2fOREREBNauXYsRI0bg008/xahRo2zet3jxYvj7++Ovf/0rKisr4e/vj23btmHYsGFITk7GggULoKqqPij94Ycf0Lt3bwDAI488gk8//RQzZsxA165dcf78efz44484evQobrzxRsybNw8XL17EqVOn9M+RtS8xPoYYRlI++OADAuD0j4iotLSUWrRoQdOmTbN5X35+PoWGhtrEy8vLHbb/8ccfEwDauXOnHnv11VcJAJ04ccKm7IkTJwgAffDBBw7bAUALFizQ/71gwQICQBMmTLApl52dTQaDgV588UWbeHp6Ovn5+TnEXdVHTbd27doRANq1a5ce++9//0sAqGnTpnTy5Ek9/o9//IMA0Pfff6/HJk2aRADo8ccf12OaptHw4cPJ39+fzp49S0REmzZtIgD0wgsv2Djdd999pCgK/frrrzb1oaoqGY1Gm7Jnz551qCsr7raPtW7/9Kc/2ZQdNWoURURE6P8+fvw4qapKo0aNIrPZbFNW0zQiEus/zrD2iVdffdUmPnLkSPL396fMzEw9lpeXR8HBwdS/f389Zm3Pfv36UVVVVa37qrm/RYsW0dmzZyk/P59++OEHuummmwgAffLJJzblndXpkCFDKD4+3iZm7UM167mwsJACAgLoySef1GOzZs0iALRnzx6bcqGhoTb9srCwkPz9/emuu+6yqfuVK1cSAFqzZo0eGzBgAAGgdevW6bFjx47pfWj37t163NqvnX0GndWTsz9r37fm8sMPP+jvKy0tpfbt21NcXJzu/f333xMAio+Pt6lPTdOoU6dONGTIEL0/EVnqvH379nTnnXfqsdDQUHrsscdqdR4+fDi1a9eu1jJMw8On2BjpWbVqFbZs2WLzB1hmHS5cuIAJEybg3Llz+p/BYECfPn3w/fff69uo+Su3oqIC586d01e0WKe6Pc0jjzxi8++NGzdC0zSMGzfOxjc6OhqdOnWy8RWha9eu6Nu3r/5v6wzFHXfcgbZt2zrEs7KyHLZR8yJj6ykyk8mkn9L4+uuvYTAYMHPmTJv3PfnkkyAifPPNNzbxAQMGoGvXrm7nINo+9nV722234fz58/qpj02bNkHTNMyfP9/hWhHrbJRI/3EXs9mMb7/9FiNHjkR8fLweb926NR544AH8+OOPuqOVadOmCV1btmDBArRq1QrR0dG47bbbcPToUbz22mu47777bMrVrFPrLOyAAQOQlZWFixcv2pTt2rWrfqoOsMyaJiQk2PSVr7/+GjfffLM+M2It9+CDD9ps67vvvoPJZMKsWbNs6n7atGkICQlxOI3VvHlzjB8/Xv93QkICWrRogS5dutjMttXWf53x8MMPOxw3unfvrufSu3dvm1PfzZs3x8MPP4zs7GwcOXLEZluTJk2yqc9Dhw7h+PHjeOCBB3D+/Hm975SVlWHQoEHYuXOnfuqxRYsW2LNnD/Ly8tzyZuSBT7Ex0tO7d2+nF2kfP34cgGUg4IyQkBD9/4uKirBo0SKsX78ehYWFNuXsvyw8hf3Ku+PHj4OI0KlTJ6fla552EKHmIAiwnEYAgNjYWKfx4uJim7iqqjZf5gBw/fXXA6g+fXPy5Em0adPG4dRGly5d9NdrYp97XYi2j33O1tNLxcXFCAkJQWZmJlRVrXWQJtJ/3OXs2bMoLy9HQkKCw2tdunSBpmnIzc3VT1sB4nX18MMPY+zYsaioqMC2bdvwxhtvwGw2O5T76aefsGDBAqSlpTlcn3Xx4kW9PwCO9QlY6rRmXzl58qTD6UEADrla+4J93N/fH/Hx8Q595brrrnM4hRoaGup2/3VFp06dMHjwYKevucqlZn+uuVLW2WcZsAycXHHx4kWEhYXhlVdewaRJkxAbG4vk5GTcfffdmDhxosNnjpEPHiAx1yzWX2gffvghoqOjHV7386vu3uPGjcOuXbvw1FNPoUePHmjevDk0TcPQoUNdXmRaE/sDuBVnX0xW7K/N0DQNiqLgm2++cTpjUN/rDlzNPriKk91F1d7A2XUptSHaPp7ITaT/eBPRuqr5xf+HP/wBBoMBc+bMwcCBA/UfEpmZmRg0aBA6d+6M5cuXIzY2Fv7+/vj666/x+uuvO9SpL/uKjP3XHmefZQB49dVX0aNHD6fvsX6ex40bh9tuuw3/+c9/8O233+LVV1/Fyy+/jI0bN2LYsGFe9WauDh4gMdcsHTp0AABERka6/KUIWH5xbt2Line truncated
       "text/plain": [
        "<Figure size 640x480 with 1 Axes>"
       ]
@@ -438,20 +444,189 @@
     }
    ],
    "source": [
-    "# ===== MAIN =====\n",
+    "# Define the classifiers and their hyperparameters in a finer grid\n",
+    "def get_fine_grids(use_bootstrap=True):\n",
+    "    return {\n",
+    "        'Decision Tree': (DecisionTreeClassifier(random_state=42), {\n",
+    "            'max_depth': [8, 10, 12],\n",
+    "            'min_samples_split': [4, 5, 6],\n",
+    "        }),\n",
+    "        'Random Forest': (RandomForestClassifier(bootstrap=use_bootstrap, random_state=42), {\n",
+    "            'n_estimators': [75, 100, 150], \n",
+    "            'max_depth': [15, 20, 25],\n",
+    "            'min_samples_split': [8, 10, 12],\n",
+    "        })\n",
+    "    }\n",
+    "\n",
     "# optimized classifiers and find their hyperparameters\n",
-    "clf_opt = ClassifierComparisonOpt(X_train, y_train)\n",
-    "clf_opt.fit_models()\n",
+    "clf_opt = ClassifierComparisonOpt(X_train_val, y_train_val)\n",
+    "fine_grid = get_fine_grids()\n",
+    "clf_opt.fit_models(fine_grid)\n",
     "clf_opt.print_summary()\n",
     "\n",
     "# Show feature importance for each model\n",
     "clf_opt.show_feature_importance()"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "01f5866d",
+   "metadata": {},
+   "source": [
+    "### Remove redundant features\n",
+    "This section is pretty bad :(. If we want to remove redunant features we need to select different features for different models, which I have not implemented."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "id": "776cdb17",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tuning Decision Tree ...\n",
+      "\n",
+      "------ Results Sorted by Accuracy ------\n",
+      "        Model  Accuracy  Precision   Recall  F1 Score                                                      Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
+      "Decision Tree  0.846842   0.840202 0.846842  0.840706 {'max_depth': 10, 'min_samples_leaf': 1, 'min_samples_split': 5}          10.301274             0.007761       10.309036\n",
+      "\n",
+      "------ Results Sorted by Total Time ------\n",
+      "        Model  Accuracy  Precision   Recall  F1 Score                                                      Best Params  Training Time (s)  Prediction Time (s)  Total Time (s)\n",
+      "Decision Tree  0.846842   0.840202 0.846842  0.840706 {'max_depth': 10, 'min_samples_leaf': 1, 'min_samples_split': 5}          10.301274             0.007761       10.309036\n"
+     ]
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAIhCAYAAACmMgXJAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAtw1JREFUeJzs3XlcVFX/B/DPvcOugCzKAIEsIijiklsqahmGZi6ZudQvl9TK8inD9FEztzRNfcwWy7J8UttsMevJQg3XEndNwg0RBEQ22UG2uef3B82VYWZ07nVgGO73/Xr5Kg53Zs753MvMmXPPuZdjjDEQQgghhCgIb+kKEEIIIYQ0NuoAEUIIIURxqANECCGEEMWhDhAhhBBCFIc6QIQQQghRHOoAEUIIIURxqANECCGEEMWhDhAhhBBCFIc6QIQQQghRHOoAEdLEff755+A4DqmpqZauCvlHUlISHnnkEbi6uoLjOOzcudPSVWpQAQEBmDx5sqTHLFmyBBzHNUyFCDED6gCRJkf7gW/o37x58xrkNY8cOYIlS5agsLCwQZ5fycrLy7FkyRIcOHDA0lUxm0mTJiEhIQErVqzAtm3b0KNHjwZ7rdTUVJ2/AVtbW3h6eqJv375YsGAB0tLSGuy1rc3kyZONvnfU/Se1M0eaJ47uBUaams8//xxTpkzBsmXLEBgYqPO7Tp06oWvXrmZ/zbVr12LOnDlISUlBQECA2Z//Xmg0GlRXV8Pe3t4qv1Hn5eWhdevWWLx4MZYsWWLp6tyzW7duwcnJCa+//jqWL1/e4K+XmpqKwMBATJgwAY8++igEQUBBQQFOnDiBHTt2gOM4fPbZZxg/fnyD1aGyshI8z8PW1tbkx9TU1KCmpgYODg4NVq/64uPjkZycLP6ckpKCRYsW4bnnnkP//v3F8uDgYPTp06fR6kWaJhtLV4AQY4YOHdqg36wbQ1lZGVq0aHFPz6FSqaBSqcxUo8YjCAKqqqosXQ2zy83NBQC0atXKbM9pynFy//334//+7/90yq5du4ZHHnkEkyZNQocOHdClSxez1akue3t7yY+xsbGBjU3jfsT06dNHp2Nz8uRJLFq0CH369NHLri5z/J0S60OnwIjV+u2339C/f3+0aNECzs7OGDZsGBITE3W2OXfuHCZPnoygoCA4ODhArVbj2Wefxc2bN8VtlixZgjlz5gAAAgMDxWHy1NRU8fTD559/rvf6HMfpjGho5zycP38eTz31FNzc3BAZGSn+/osvvkD37t3h6OgId3d3jB8/Hunp6Xdtp6E5QAEBAXjsscdw4MAB9OjRA46OjoiIiBBPM+3YsQMRERFwcHBA9+7dcebMGZ3nnDx5Mlq2bImrV68iOjoaLVq0gI+PD5YtW4b6g8JlZWWYPXs2/Pz8YG9vj9DQUKxdu1ZvO47jMHPmTHz55ZcIDw+Hvb09Nm7ciNatWwMAli5dKmarzc2U/VM32ytXrmDy5Mlo1aoVXF1dMWXKFJSXl+tl9sUXX6BXr15wcnKCm5sbBgwYgD179uhsY8rxU9+SJUvQtm1bAMCcOXPAcZzOiOGZM2cwdOhQuLi4oGXLlnj44Ydx9OhRnefQ7s+DBw/ixRdfRJs2bXDffffd8XWNadu2LT7//HNUVVVh9erVOr8rLCzErFmzxP3Wrl07vP322xAEQWc7QRDw7rvvisdL69atMWTIEJw8eVLcpv4coOrqaixduhQhISFwcHCAh4cHIiMjsXfvXp2s6o9Y1tTU4M0330RwcDDs7e0REBCABQsWoLKyUmc77fH9xx9/oFevXnBwcEBQUBC2bt0qK6e67pa/qcfFxYsXMWbMGLi7u8PBwQE9evTAzz//fM/1I42HRoBIk1VUVIS8vDydMk9PTwDAtm3bMGnSJERHR+Ptt99GeXk5PvroI0RGRuLMmTPih9LevXtx9epVTJkyBWq1GomJifjkk0+QmJiIo0ePguM4jB49GpcvX8bXX3+Nd955R3yN1q1bi9/2pXjyyScREhKCt956S+wkrFixAm+88QbGjh2LadOmITc3F++//z4GDBiAM2fOyBpNuHLlCp566ik8//zz+L//+z+sXbsWw4cPx8aNG7FgwQK8+OKLAICVK1di7NixuHTpEnj+9ncejUaDIUOG4IEHHsDq1asRGxuLxYsXo6amBsuWLQMAMMYwYsQI7N+/H1OnTkXXrl2xe/duzJkzB9evX8c777yjU6d9+/bh22+/xcyZM+Hp6YkuXbrgo48+wowZM/D4449j9OjRAIDOnTsDMG3/1DV27FgEBgZi5cqVOH36ND799FO0adMGb7/9trjN0qVLsWTJEvTt2xfLli2DnZ0djh07hn379uGRRx4BYPrxU9/o0aPRqlUrvPrqq+IpqZYtWwIAEhMT0b9/f7i4uGDu3LmwtbXFxx9/jAcffBAHDx5E7969dZ7rxRdfROvWrbFo0SKUlZVJ2vd19enTB8HBwTqdj/LycgwcOBDXr1/H888/D39/fxw5cgTz58/HjRs3sH79enHbqVOn4vPPP8fQoUMxbdo01NTU4PDhwzh69KjREdglS5Zg5cqVmDZtGnr16oXi4mKcPHkSp0+fxuDBg43Wddq0adiyZQvGjBmD2bNn49ixY1i5ciUuXLiAH3/8UWfbK1euYMyYMZg6dSomTZqEzZs3Y/LkyejevTvCw8Nl56VlKH9Tj4vExET069cPvr6+mDdvHlq0aIFvv/0Wo0aNwg8//IDHH3/8nutHGgEjpIn573//ywAY/McYYyUlJaxVq1Zs+vTpOo/Lyspirq6uOuXl5eV6z//1118zAOzQoUNi2Zo1axgAlpKSorNtSkoKA8D++9//6j0PALZ48WLx58WLFzMAbMKECTrbpaamMpVKxVasWKFTnpCQwGxsbPTKjeVRt25t27ZlANiRI0fEst27dzMAzNHRkV27dk0s//jjjxkAtn//frFs0qRJDAD717/+JZYJgsCGDRvG7OzsWG5uLmOMsZ07dzIAbPny5Tp1GjNmDOM4jl25ckUnD57nWWJios62ubm5ellpmbp/tNk+++yzOts+/vjjzMPDQ/w5KSmJ8TzPHn/8cabRaHS2FQSBMSbt+DFEe0ysWbNGp3zUqFHMzs6OJScni2WZmZnM2dmZDRgwQCzT7s/IyEhWU1Nzx9e60+vVNXLkSAaAFRUVMcYYe/PNN1mLFi3Y5cuXdbabN28eU6lULC0tjTHG2L59+xgA9vLLL+s9pzYvxmqPt0mTJok/d+nShQ0bNuyO9dbuM62zZ88yAGzatGk627322msMANu3b5/O69U/BnJycpi9vT2bPXv2HV+3rhMnTuj9/RrLX8px8fDDD7OIiAhWUVEhlgmCwPr27ctCQkJMrh+xLDoFRpqsDRs2YO/evTr/gNpRg8LCQkyYMAF5eXniP5VKhd69e2P//v3iczg6Oor/X1FRgby8PDzwwAMAgNOnTzdIvV944QWdn3fs2AFBEDB27Fid+qrVaoSEhOjUV4qOHTvqzHfQjjAMGjQI/v7+euVXr17Ve46ZM2eK/689hVVVVYXff/8dAPDrr79CpVLh5Zdf1nnc7NmzwRjDb7/9plM+cOBAdOzY0eQ2SN0/9bPt378/bt68ieLiYgDAzp07IQgCFi1apDPapW0fIO34MZVGo8GePXswatQoBAUFieXe3t546qmn8Mcff4h11Jo+fbrZ5nZpR6FKSkoAAN999x369+8PNzc3nTZGRUVBo9Hg0KFDAIAffvgBHMdh8eLFes95pwn3rVq1QmJiIpKSkkyu46+//goAiImJ0SmfPXs2AGDXrl065R07dtSZuNy6dWuEhoYaPI7lqJ+/qcdFfn4+9u3bh7Fjx6KkpETc7ubNm4iOjkZSUhKuX79uljqShkWnwEiT1atXL4ND8No33UGDBhl8nIuLi/j/+fn5WLp0Kb755hvk5OTobFdUVGTG2t5Wf+VaUlISGGMICQkxuL2UlTV11e3kAICrqysAwM/Pz2B5QUGBTjnP8zof1gDQvn17ABDnG127dg0+Pj5wdnbW2a5Dhw7i7+uq3/a7kbp/6rfZzc0NQG3bXFxckJycDJ7n79gJk3L8mCo3Nxfl5eUIDQ3V+12HDh0gCALS09N1Tt1IzepOSktLAUDcT0lJSTh37pw4/6o+bdbJycnw8fGBu7u7pNdbtmwZRo4cifbt26NTp04YMmQInnnmGfHUpiHXrl0Dz/No166dTrlarUarVq30jqX6+xqo3d/1j2O5DP2dAnc/Lq5cuQLGGN544w288cYbBrfNycmBr6+vWepJGg51gIjV0U7i3LZtG9Rqtd7v6648GTt2LI4cOYI5c+aga9euaNmyJQRBwJAhQ/Qmgxpi7FuwRqMx+pi6oxra+nIch99++83gN37tt3epjI0eGCtnjXDFi/ptvxup+8ccbZNy/DQkqVndyd9//402bdqIH9KCIGDw4MGYO3euwe21HV25BgwYgOTkZPz000/Ys2cPPv30U7zzzjvYuHEjpk2bdsfHmnoph4Y+jg39nQJ3Py6027322muIjo42+Nz1O3mkaaIOELE6wcHBAIA2bdogKirK6HYFBQWIi4vD0qVLsWjRIrHc0LC9sTdl7QhD/Qsk1v+2erf6MsYQGBh4zx885iQIAq5evapTp8uXLwOAONmzbdu2+P3331FSUqIzCnTx4kXx93djLFsp+8dUwcHBEAQB58+fN3q9KFOPHylat24NJycnXLp0Se93Fy9eBM/zeiNz5qK99k3dZd7BwcEoLS29a/uCg4Oxe/du5OfnSx4Fcnd3x5QpUzBlyhSUlpZiwIABWLJkidEOUNu2bSEIApKSksQRRADIzs5GYWGhScdSQzL1uNCOmtra2prt+CGWQXOAiNWJjo6Gi4sL3nrrLVRXV+v9XrtyS/sNsv43xrorYLS01wCp39FxcXGBp6enOGdC68MPPzS5vqNHj4ZKpcLSpUv16sIY01vLine truncated
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>age</th>\n",
+       "      <th>education.num</th>\n",
+       "      <th>relationship</th>\n",
+       "      <th>capital.gain</th>\n",
+       "      <th>capital.loss</th>\n",
+       "      <th>hours.per.week</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>19841</th>\n",
+       "      <td>35</td>\n",
+       "      <td>10</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>45</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>24329</th>\n",
+       "      <td>50</td>\n",
+       "      <td>9</td>\n",
+       "      <td>4</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>12</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>10002</th>\n",
+       "      <td>43</td>\n",
+       "      <td>9</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>45</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>25682</th>\n",
+       "      <td>23</td>\n",
+       "      <td>10</td>\n",
+       "      <td>3</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>24</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>13800</th>\n",
+       "      <td>30</td>\n",
+       "      <td>10</td>\n",
+       "      <td>1</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>40</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "       age  education.num  relationship  capital.gain  capital.loss  \\\n",
+       "19841   35             10             0             0             0   \n",
+       "24329   50              9             4             0             0   \n",
+       "10002   43              9             0             0             0   \n",
+       "25682   23             10             3             0             0   \n",
+       "13800   30             10             1             0             0   \n",
+       "\n",
+       "       hours.per.week  \n",
+       "19841              45  \n",
+       "24329              12  \n",
+       "10002              45  \n",
+       "25682              24  \n",
+       "13800              40  "
+      ]
+     },
+     "execution_count": 39,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Remove redundant features\n",
+    "X_filtered = X_train_val.drop(columns=['sex', 'native.country', 'marital.status', 'race', 'occupation', 'workclass'])\n",
+    "# ===== MAIN =====\n",
+    "\n",
+    "# optimized classifiers and find their hyperparameters\n",
+    "clf_opt = ClassifierComparisonOpt(X_filtered, y_train_val)\n",
+    "clf_opt.fit_models()\n",
+    "clf_opt.print_summary()\n",
+    "\n",
+    "# Show feature importance for each model\n",
+    "clf_opt.show_feature_importance()\n",
+    "X_filtered.head()"
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": ".venv",
    "language": "python",
    "name": "python3"
   },
@@ -465,7 +640,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.13.7"
+   "version": "3.12.12"
   }
  },
  "nbformat": 4,

Report/MLPproject.tex

@@ -81,7 +81,7 @@ With our numerical version of the dataset we found with the info function in pan
 \subsection{Training, validation and test sets}
 Before doing any sort of training or analysis on the data, se split it into training, test and validation data. We did this by first splitting a random 20\% of the data into test data. This data is reserved for the final testing of the model and will not be touched until the model is finished. Then we did a further split of the rest of the data were 25\% was designated as validation data. This data will be used for calibration of the model and hyperparameter tuning. The rest of the data which is 60\% of the total data or around 18000 data points will be used to train the model.
 \section{Model selection}
-When selecting the model to use for this project we have to limit us to using models that are appropriate to the type of problem that we are trying to solve. The problem is a classification task so all models that are used for regression are immediately invalid. There are plenty of different types of classification models left to choose from. Many of them however, are good for data that has non-discrete features. This includes models such as logistic regression, KNN and other similar types of classification models. Also since we have so many features that are non-numerical and converted into arbitrary numbers these types of models would not be optimal. What is left is the Gaussian Naïve Baye's and the different tree based models. Since our data again is made up of arbitrary numbers it is not possible to assume thata we have normally distributed data. Therefore we are left with the tree based models such as the decision tree and random forests. We decided to implement two different types of models. We first do a decision tree and see how good we can get that model to work. We then do a random forest which may not be the absolute best model but since it is a continuation on the decision tree it might be interesting to see if it performs better. We then do analysis on both methods and see if these models are good enough and if there is any meaningful difference between the two.
+When selecting the model to use for this project we have to limit us to using models that are appropriate to the type of problem that we are trying to solve. The problem is a classification task so all models that are used for regression are immediately invalid. There are plenty of different types of classification models left to choose from. Many of them however, are good for data that has non-discrete features. This includes models such as logistic regression, KNN and other similar types of classification models. Also since we have so many features that are non-numerical and converted into arbitrary numbers these types of models would not be optimal. What is left is the Gaussian Naïve Baye's and the different tree based models. Naïve Baye's can be a bit troublesome for this dataset since we have found that some parameters are slightly correlated. However, this does not necessarliy make in an inappropriate method as it has been found to perform well despite this strict assumption. Therefore we are left with the tree based models such as the decision tree and random forests. We decided to implement two different types of models. We first do a decision tree and see how good we can get that model to work. We then do a random forest which may not be the absolute best model but since it is a continuation on the decision tree it might be interesting to see if it performs better. We then do analysis on both methods and see if these models are good enough and if there is any meaningful difference between the two.
 
 \section{Model Training and Hyperparameter Tuning}
 During the model training there are some important changes we can make to improve the accuracy of our model. One thing we implement is cross validation. Since there is a great spread in our data we choose to use randomized search. %Add more here and change type of x-val if needed. How many folds?