{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Problem Set 5: Credit Card Fraud Detection\n", "\n", "**Release Date:** 12 March 2024\n", "\n", "**Due Date:** 23:59, 30 March 2024" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview\n", "\n", "In class, we discussed about logistic regression, and how it can be useful as a classification algorithm. In this problem set, we get some hands-on practice by implementing logistic regression on a Credit Card Fraud Detection dataset. Note that for this problem set, you should only be using the scikit-learn (sklearn) library for the last part (Tasks 5.x) on SVM.\n", "\n", "**Required Files**:\n", "\n", "* ps5.ipynb\n", "* credit_card.csv\n", "* restaurant_data.csv\n", "\n", "**Honour Code**: Note that plagiarism will not be condoned! You may discuss with your classmates and check the internet for references, but you MUST NOT submit code/report that is copied directly from other sources!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Important\n", "\n", "Similar to PS0, your implementation in the following tasks **should NOT involve any iteration, including `map` and `filter`, or recursion**. Instead, you should work with the operations available in NumPy. Solutions that violate this will be penalised.\n", "\n", "- You are allowed to use any mathematical functions, but this does not mean that you are allowed to use any NumPy function (there are NumPy functions that aren’t mathematical functions). For example, `np.vectorize` is not allowed since it is iterative. If you are in doubt about which functions are allowed, you should ask in the forum.\n", "\n", "There is, however, an exception for **Tasks 3.4 and 3.5**. In the pseudo-code for the algorithm required, there is an explicit for loop. Hence, only for these task you may use **a single for/while loop** to iterate for the number of epochs required." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "Fraudulent credit card transaction is a common phenomenon in many parts of the world and can lead to potentially large amounts of losses for both companies and customers. Therefore, we hope to help credit card companies recognize those fraudulent transactions so that customers are not charged for items that they did not purchase.\n", "\n", "We are given a dataset that contains transactions made by credit cards holders in `credit_card.csv`. If we think about what type of data might be included in the input variables under the given context, we might realize that those input variables are likely to include word descriptions, such as shop name or locations. In this problem set, we don't need to worry about language processing as the data are pre-processed to contain only numeric values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting Started\n", "\n", "Take a look at the columns in the dataset `credit_card.csv`. We have V1-V20, 'Amount', and 'Time' as input features, and 'Class' as output which takes the value 1 if it's fraud and 0 otherwise. This dataset presents 492 frauds out of 284,807 transactions. That means, there are 284,808 rows (including the header) in the csv file.\n", "\n", "We will use this dataset to implement logistic regression using batch and stochastic gradient descent for binary classification." ] }, { "cell_type": "code", "execution_count": 118, "metadata": {}, "outputs": [], "source": [ "# Initial imports and setup\n", "\n", "import numpy as np\n", "import os\n", "import pandas as pd\n", "\n", "from sklearn import svm\n", "from sklearn import model_selection\n", "\n", "# Read credit card data into a Pandas dataframe for large tests\n", "\n", "dirname = os.getcwd()\n", "credit_card_data_filepath = os.path.join(dirname, 'credit_card.csv')\n", "restaurant_data_filepath = os.path.join(dirname, 'restaurant_data.csv')\n", "\n", "credit_df = pd.read_csv(credit_card_data_filepath)\n", "X = credit_df.values[:, :-1]\n", "y = credit_df.values[:, -1:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basics of Pandas\n", "\n", "[Pandas](https://pandas.pydata.org/) is an open source data analysis and manipulation tool in Python. In this problem set, we read the CSV into a [Pandas dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) to provide some nifty methods that makes it easier for us to handle large amounts of data. You can think of a dataframe as a large table that stores our dataset in a neat and optimized manner, making it fast for retrieval and manipulation and data. Using Pandas, we can quickly gain an overview of the type and values of the data stored, distributions of values within the dataset, and even ways to perform sampling.\n", "\n", "In the new few sections, we will explore some basic functions of Pandas to help us get started. You do not need to submit any codes for this section, and can simply run the cells to follow along." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exploring the dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the [`head`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.head.html) method on the dataframe, we can get an overview of the data. By default, the method returns the first 5 entries in the dataframe." ] }, { "cell_type": "code", "execution_count": 119, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeV1V2V3V4V5V6V7V8V9...V13V14V15V16V17V18V19V20AmountClass
00.0-1.359807-0.0727812.5363471.378155-0.3383210.4623880.2395990.0986980.363787...-0.991390-0.3111691.468177-0.4704010.2079710.0257910.4039930.251412149.620
10.01.1918570.2661510.1664800.4481540.060018-0.082361-0.0788030.085102-0.255425...0.489095-0.1437720.6355580.463917-0.114805-0.183361-0.145783-0.0690832.690
21.0-1.358354-1.3401631.7732090.379780-0.5031981.8004990.7914610.247676-1.514654...0.717293-0.1659462.345865-2.8900831.109969-0.121359-2.2618570.524980378.660
31.0-0.966272-0.1852261.792993-0.863291-0.0103091.2472030.2376090.377436-1.387024...0.507757-0.287924-0.631418-1.059647-0.6840931.965775-1.232622-0.208038123.500
42.0-1.1582330.8777371.5487180.403034-0.4071930.0959210.592941-0.2705330.817739...1.345852-1.1196700.175121-0.451449-0.237033-0.0381950.8034870.40854269.990
\n", "

5 rows × 23 columns

\n", "
" ], "text/plain": [ " Time V1 V2 V3 V4 V5 V6 V7 \\\n", "0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599 \n", "1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803 \n", "2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 1.800499 0.791461 \n", "3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 1.247203 0.237609 \n", "4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 0.095921 0.592941 \n", "\n", " V8 V9 ... V13 V14 V15 V16 V17 \\\n", "0 0.098698 0.363787 ... -0.991390 -0.311169 1.468177 -0.470401 0.207971 \n", "1 0.085102 -0.255425 ... 0.489095 -0.143772 0.635558 0.463917 -0.114805 \n", "2 0.247676 -1.514654 ... 0.717293 -0.165946 2.345865 -2.890083 1.109969 \n", "3 0.377436 -1.387024 ... 0.507757 -0.287924 -0.631418 -1.059647 -0.684093 \n", "4 -0.270533 0.817739 ... 1.345852 -1.119670 0.175121 -0.451449 -0.237033 \n", "\n", " V18 V19 V20 Amount Class \n", "0 0.025791 0.403993 0.251412 149.62 0 \n", "1 -0.183361 -0.145783 -0.069083 2.69 0 \n", "2 -0.121359 -2.261857 0.524980 378.66 0 \n", "3 1.965775 -1.232622 -0.208038 123.50 0 \n", "4 -0.038195 0.803487 0.408542 69.99 0 \n", "\n", "[5 rows x 23 columns]" ] }, "execution_count": 119, "metadata": {}, "output_type": "execute_result" } ], "source": [ "credit_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we can inspect the [value counts](https://pandas.pydata.org/docs/reference/api/pandas.Series.value_counts.html) of the 'Class' property in the dataframe to know the number of fraudulent and non-fraudulent transactions." ] }, { "cell_type": "code", "execution_count": 120, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 284315\n", "1 492\n", "Name: Class, dtype: int64" ] }, "execution_count": 120, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Inspect the number of fraudulent and non-fraudulent transactions.\n", "credit_df['Class'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Indexing and selecting data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similar to NumPy, we can also index and select data on Pandas dataframes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For example, we can [select columns in the dataframe by their labels](https://pandas.pydata.org/docs/user_guide/indexing.html#basics). In the following example, we use `'Class'` to index the 'Class' column in the credit dataframe." ] }, { "cell_type": "code", "execution_count": 121, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 0\n", "1 0\n", "2 0\n", "3 0\n", "4 0\n", " ..\n", "284802 0\n", "284803 0\n", "284804 0\n", "284805 0\n", "284806 0\n", "Name: Class, Length: 284807, dtype: int64" ] }, "execution_count": 121, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Select the 'Class' column in the credit dataframe\n", "credit_df['Class']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or, we can use integer indexing." ] }, { "cell_type": "code", "execution_count": 122, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeV1V2V3V4V5V6V7V8V9...V13V14V15V16V17V18V19V20AmountClass
00.0-1.359807-0.0727812.5363471.378155-0.3383210.4623880.2395990.0986980.363787...-0.991390-0.3111691.468177-0.4704010.2079710.0257910.4039930.251412149.620
10.01.1918570.2661510.1664800.4481540.060018-0.082361-0.0788030.085102-0.255425...0.489095-0.1437720.6355580.463917-0.114805-0.183361-0.145783-0.0690832.690
\n", "

2 rows × 23 columns

\n", "
" ], "text/plain": [ " Time V1 V2 V3 V4 V5 V6 V7 \\\n", "0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599 \n", "1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803 \n", "\n", " V8 V9 ... V13 V14 V15 V16 V17 \\\n", "0 0.098698 0.363787 ... -0.991390 -0.311169 1.468177 -0.470401 0.207971 \n", "1 0.085102 -0.255425 ... 0.489095 -0.143772 0.635558 0.463917 -0.114805 \n", "\n", " V18 V19 V20 Amount Class \n", "0 0.025791 0.403993 0.251412 149.62 0 \n", "1 -0.183361 -0.145783 -0.069083 2.69 0 \n", "\n", "[2 rows x 23 columns]" ] }, "execution_count": 122, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Obtain the first 2 rows\n", "credit_df[0:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also [select columns in the dataframe that fulfils some condition](https://pandas.pydata.org/docs/user_guide/indexing.html#boolean-indexing). In the example below, `credit_df['Class'] == 0` returns a [Pandas Series](https://pandas.pydata.org/docs/reference/api/pandas.Series.html) of length 284807, which is the size of our dataset. It contains the value `True` if and only if the `Class` value for the particular entry is of value 0, and `False` otherwise. We can use this Boolean series to index the credit dataframe to return the rows where the `Class` field is 0. Does this remind you of how NumPy arrays operate?" ] }, { "cell_type": "code", "execution_count": 123, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 True\n", "1 True\n", "2 True\n", "3 True\n", "4 True\n", " ... \n", "284802 True\n", "284803 True\n", "284804 True\n", "284805 True\n", "284806 True\n", "Name: Class, Length: 284807, dtype: bool" ] }, "execution_count": 123, "metadata": {}, "output_type": "execute_result" } ], "source": [ "credit_df['Class'] == 0" ] }, { "cell_type": "code", "execution_count": 124, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeV1V2V3V4V5V6V7V8V9...V13V14V15V16V17V18V19V20AmountClass
00.0-1.359807-0.0727812.5363471.378155-0.3383210.4623880.2395990.0986980.363787...-0.991390-0.3111691.468177-0.4704010.2079710.0257910.4039930.251412149.620
10.01.1918570.2661510.1664800.4481540.060018-0.082361-0.0788030.085102-0.255425...0.489095-0.1437720.6355580.463917-0.114805-0.183361-0.145783-0.0690832.690
21.0-1.358354-1.3401631.7732090.379780-0.5031981.8004990.7914610.247676-1.514654...0.717293-0.1659462.345865-2.8900831.109969-0.121359-2.2618570.524980378.660
31.0-0.966272-0.1852261.792993-0.863291-0.0103091.2472030.2376090.377436-1.387024...0.507757-0.287924-0.631418-1.059647-0.6840931.965775-1.232622-0.208038123.500
42.0-1.1582330.8777371.5487180.403034-0.4071930.0959210.592941-0.2705330.817739...1.345852-1.1196700.175121-0.451449-0.237033-0.0381950.8034870.40854269.990
..................................................................
284802172786.0-11.88111810.071785-9.834783-2.066656-5.364473-2.606837-4.9182157.3053341.914428...-0.6892564.626942-0.9244591.1076411.9916910.510632-0.6829201.4758290.770
284803172787.0-0.732789-0.0550802.035030-0.7385890.8682291.0584150.0243300.2948690.584800...1.214756-0.6751431.164931-0.711757-0.025693-1.221179-1.5455560.05961624.790
284804172788.01.919565-0.301254-3.249640-0.5578282.6305153.031260-0.2968270.7084170.432454...-0.183699-0.5106021.3292840.1407160.3135020.395652-0.5772520.00139667.880
284805172788.0-0.2404400.5304830.7025100.689799-0.3779610.623708-0.6861800.6791450.392087...-1.0420820.4496241.962563-0.6085770.5099281.1139812.8978490.12743410.000
284806172792.0-0.533413-0.1897330.703337-0.506271-0.012546-0.6496171.577006-0.4146500.486180...-0.188093-0.0843160.041333-0.302620-0.6603770.167430-0.2561170.382948217.000
\n", "

284315 rows × 23 columns

\n", "
" ], "text/plain": [ " Time V1 V2 V3 V4 V5 \\\n", "0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 \n", "1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 \n", "2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 \n", "3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 \n", "4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 \n", "... ... ... ... ... ... ... \n", "284802 172786.0 -11.881118 10.071785 -9.834783 -2.066656 -5.364473 \n", "284803 172787.0 -0.732789 -0.055080 2.035030 -0.738589 0.868229 \n", "284804 172788.0 1.919565 -0.301254 -3.249640 -0.557828 2.630515 \n", "284805 172788.0 -0.240440 0.530483 0.702510 0.689799 -0.377961 \n", "284806 172792.0 -0.533413 -0.189733 0.703337 -0.506271 -0.012546 \n", "\n", " V6 V7 V8 V9 ... V13 V14 \\\n", "0 0.462388 0.239599 0.098698 0.363787 ... -0.991390 -0.311169 \n", "1 -0.082361 -0.078803 0.085102 -0.255425 ... 0.489095 -0.143772 \n", "2 1.800499 0.791461 0.247676 -1.514654 ... 0.717293 -0.165946 \n", "3 1.247203 0.237609 0.377436 -1.387024 ... 0.507757 -0.287924 \n", "4 0.095921 0.592941 -0.270533 0.817739 ... 1.345852 -1.119670 \n", "... ... ... ... ... ... ... ... \n", "284802 -2.606837 -4.918215 7.305334 1.914428 ... -0.689256 4.626942 \n", "284803 1.058415 0.024330 0.294869 0.584800 ... 1.214756 -0.675143 \n", "284804 3.031260 -0.296827 0.708417 0.432454 ... -0.183699 -0.510602 \n", "284805 0.623708 -0.686180 0.679145 0.392087 ... -1.042082 0.449624 \n", "284806 -0.649617 1.577006 -0.414650 0.486180 ... -0.188093 -0.084316 \n", "\n", " V15 V16 V17 V18 V19 V20 Amount \\\n", "0 1.468177 -0.470401 0.207971 0.025791 0.403993 0.251412 149.62 \n", "1 0.635558 0.463917 -0.114805 -0.183361 -0.145783 -0.069083 2.69 \n", "2 2.345865 -2.890083 1.109969 -0.121359 -2.261857 0.524980 378.66 \n", "3 -0.631418 -1.059647 -0.684093 1.965775 -1.232622 -0.208038 123.50 \n", "4 0.175121 -0.451449 -0.237033 -0.038195 0.803487 0.408542 69.99 \n", "... ... ... ... ... ... ... ... \n", "284802 -0.924459 1.107641 1.991691 0.510632 -0.682920 1.475829 0.77 \n", "284803 1.164931 -0.711757 -0.025693 -1.221179 -1.545556 0.059616 24.79 \n", "284804 1.329284 0.140716 0.313502 0.395652 -0.577252 0.001396 67.88 \n", "284805 1.962563 -0.608577 0.509928 1.113981 2.897849 0.127434 10.00 \n", "284806 0.041333 -0.302620 -0.660377 0.167430 -0.256117 0.382948 217.00 \n", "\n", " Class \n", "0 0 \n", "1 0 \n", "2 0 \n", "3 0 \n", "4 0 \n", "... ... \n", "284802 0 \n", "284803 0 \n", "284804 0 \n", "284805 0 \n", "284806 0 \n", "\n", "[284315 rows x 23 columns]" ] }, "execution_count": 124, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Obtain the credit dataframe where the 'Class' field is 0\n", "credit_df[credit_df['Class'] == 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also [concatenate](https://pandas.pydata.org/docs/reference/api/pandas.concat.html) Pandas series or dataframes together! In this example, we explored how we can concat the first 2 rows and last 2 rows of the dataframe. Similar to NumPy, you would also need to specify the axis for concatenation." ] }, { "cell_type": "code", "execution_count": 125, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeV1V2V3V4V5V6V7V8V9...V13V14V15V16V17V18V19V20AmountClass
00.0-1.359807-0.0727812.5363471.378155-0.3383210.4623880.2395990.0986980.363787...-0.991390-0.3111691.468177-0.4704010.2079710.0257910.4039930.251412149.620
10.01.1918570.2661510.1664800.4481540.060018-0.082361-0.0788030.085102-0.255425...0.489095-0.1437720.6355580.463917-0.114805-0.183361-0.145783-0.0690832.690
284805172788.0-0.2404400.5304830.7025100.689799-0.3779610.623708-0.6861800.6791450.392087...-1.0420820.4496241.962563-0.6085770.5099281.1139812.8978490.12743410.000
284806172792.0-0.533413-0.1897330.703337-0.506271-0.012546-0.6496171.577006-0.4146500.486180...-0.188093-0.0843160.041333-0.302620-0.6603770.167430-0.2561170.382948217.000
\n", "

4 rows × 23 columns

\n", "
" ], "text/plain": [ " Time V1 V2 V3 V4 V5 V6 \\\n", "0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 \n", "1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 \n", "284805 172788.0 -0.240440 0.530483 0.702510 0.689799 -0.377961 0.623708 \n", "284806 172792.0 -0.533413 -0.189733 0.703337 -0.506271 -0.012546 -0.649617 \n", "\n", " V7 V8 V9 ... V13 V14 V15 \\\n", "0 0.239599 0.098698 0.363787 ... -0.991390 -0.311169 1.468177 \n", "1 -0.078803 0.085102 -0.255425 ... 0.489095 -0.143772 0.635558 \n", "284805 -0.686180 0.679145 0.392087 ... -1.042082 0.449624 1.962563 \n", "284806 1.577006 -0.414650 0.486180 ... -0.188093 -0.084316 0.041333 \n", "\n", " V16 V17 V18 V19 V20 Amount Class \n", "0 -0.470401 0.207971 0.025791 0.403993 0.251412 149.62 0 \n", "1 0.463917 -0.114805 -0.183361 -0.145783 -0.069083 2.69 0 \n", "284805 -0.608577 0.509928 1.113981 2.897849 0.127434 10.00 0 \n", "284806 -0.302620 -0.660377 0.167430 -0.256117 0.382948 217.00 0 \n", "\n", "[4 rows x 23 columns]" ] }, "execution_count": 125, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.concat([credit_df[:2], credit_df[-2:]], axis=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Processing\n", "\n", "As you might have realised, managing Pandas is not so much different from how you would operate on NumPy or basic Python structures! We have provided some teaser on basic Pandas. Now, we will see how we can use Pandas to help us in resampling.\n", "\n", "In particular, you might find the [`sample` method](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html) helpful in this section." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 1.1: Problem with imbalanced data\n", "\n", "Our first task is to describe in one or two sentences what problem we might encounter if we directly use the given dataset without processing it.\n", "\n", "Hint: consider the 'Class' column, and think about how a high accuracy prediction can be achieved in a wrong way" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 1.2: Resampling methods\n", "\n", "When we are faced with the issue of imbalanced data, there are several ways to deal with it. A more direct way might be just to collect more data instances. We realized that in our case this doesn't work well because the events unevenly occur. We then look at how to resample the existing instances.\n", "\n", "In this problem set, you are introduced with three resampling methods: undersampling, oversampling, and SMOTE." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Concept 1.2.1: Undersampling\n", "\n", "
\n", "\"visualisation\n", "
Figure 1: Visualisation of undersampling.
\n", "
\n", "\n", "The figure above illustrates undersampling. In undersampling, we remove samples from the majority class. More specifically, we randomly take subsamples of the majority class such that the size of two classes is the same." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Task 1.2.1: Random undersampling in practice\n", "\n", "Your task is to observe the data in `credit_card.csv` and randomly remove some observations of the majority class until the two classes balance out." ] }, { "cell_type": "code", "execution_count": 126, "metadata": {}, "outputs": [], "source": [ "def random_undersampling(df: pd.DataFrame) -> pd.DataFrame:\n", " '''\n", " Given credit card dataset with 0 as the majority for 'Class' column. Returns credit card data with two classes\n", " having the same shape, given raw data read from csv.\n", "\n", " Parameters\n", " ----------\n", " df: pd.DataFrame\n", " The potentially imbalanced dataset.\n", "\n", " Returns\n", " -------\n", " Undersampled dataset with equal number of fraudulent and legitimate data.\n", " '''\n", "\n", " # TODO: add your solution here and remove `raise NotImplementedError`\n", " ones = df[df['Class'] == 1]\n", " return pd.concat([ones,df[df['Class'] == 0].sample(len(ones))], axis=0)" ] }, { "cell_type": "code", "execution_count": 127, "metadata": {}, "outputs": [], "source": [ "# small data\n", "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [111.6, 10, 1], [111.4, 10, 0], [111.5, 10, 1], [111.6, 10, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "# checks that the randomly undersampled dataset has 3 fraudulent and 3 legitimate data points\n", "assert np.all(np.equal(random_undersampling(df1)['Class'].value_counts(), (3, 3)))\n", "\n", "# credit card data\n", "assert np.all(np.equal(random_undersampling(credit_df)['Class'].value_counts(), (492, 492)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now you might realize the drawback of undersampling - by removing data randomly, you might have removed some valuable information from the dataset. Is there any way to do better without losing the valuable information?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Concept 1.2.2: Oversampling\n", "\n", "
\n", "\"visualisation\n", "
Figure 2: Visualisation of oversampling.
\n", "
\n", "\n", "The figure above illustrates oversampling. In oversampling, we duplicate records of the minority class such that the size of two classes balance out." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Task 1.2.2: Oversampling in practice\n", "\n", "Your task is to observe the data in `credit_card.csv` and apply the oversampling technique to the dataset." ] }, { "cell_type": "code", "execution_count": 128, "metadata": {}, "outputs": [], "source": [ "def random_oversampling(df: pd.DataFrame) -> pd.DataFrame:\n", " '''\n", " Given credit card dataset with 0 as the majority for 'Class' column. Returns credit card data with two classes\n", " having the same shape, given raw data read from csv.\n", "\n", " Parameters\n", " ----------\n", " df: pd.DataFrame\n", " The potentially imbalanced dataset.\n", "\n", " Returns\n", " -------\n", " Oversampled dataset with equal number of fraudulent and legitimate data.\n", " '''\n", "\n", " # TODO: add your solution here and remove `raise NotImplementedError`\n", " zeros = df[df['Class'] == 0]\n", " ones = df[df['Class'] == 1]\n", " return pd.concat([zeros, ones.sample(len(zeros), replace=True)], axis=0)" ] }, { "cell_type": "code", "execution_count": 129, "metadata": {}, "outputs": [], "source": [ "# small data\n", "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], \n", " [111.4, 10, 0], [111.5, 10, 0], [111.4, 10, 0], \n", " [111.5, 10, 1], [111.6, 10, 1], [111.6, 10, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "assert np.all(np.equal(random_oversampling(df1)['Class'].value_counts(), (6, 6)))\n", "\n", "# credit card data\n", "assert np.all(np.equal(random_oversampling(credit_df)['Class'].value_counts(), (284315, 284315)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You might realize the bright side of oversampling - you don't lose certain valuable information. At the same time, you also realize the drawback of oversampling - it can cause overfitting and a poor generalization of the test set. Now the question is, instead of simply duplicating records, is there any other way to increase the number of records in the minority set?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Concept 1.2.3: SMOTE (for further reading only)\n", "\n", "
\n", "\"visualisation\n", "
Figure 3: Visualisation of SMOTE.
\n", "
\n", "\n", "The figure above illustrates Synthetic Minority Oversampling Technique (SMOTE).\n", "\n", "The SMOTE algorithm works in these four steps:\n", "\n", "1. Consider minority and majority instances in vector space.\n", "1. For each minority-class instance pair, interpolate their feature values.\n", "1. Randomly synthesize instances and label with minority class\n", "1. More instances added to minority class\n", "\n", "In this case, you increase the number of minority instances without simply duplicating the values. Now your newly inserted minority record is not an exact copy of an existing data point, but it is also not too different from the known observations in your minority class. Outside this problem set, when we want to do data resampling, we can use the [Python imbalanced-learn library](https://pypi.org/project/imbalanced-learn/). We will not be implementing SMOTE algorithm for this problem set." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Word of caution when resampling\n", "\n", "In the section above, we introduced three resampling techniques. One very important thing to note is, in practice you should first split dataset to train–test sets, then resample the train dataset before training the model. This is done to avoid data leakage (snooping).\n", "\n", "Data leakage can cause you to create overly optimistic if not completely invalid predictive models. It is when information from outside the training dataset is used to train the model. The additional information might allow the model to learn something that it otherwise would not know and in turn invalidate the estimated performance of the mode being constructed." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Splitting data\n", "\n", "Before we can start training our models, we need to **randomly** partition our dataset into training data and testing data. Remember that we are trying to make a model that can predict fraudulence of data points that the model has never seen. It would be unwise to measure the accuracy of the model using the same data it trained on." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 2.1: Splitting data\n", "\n", "In this task, you need to implement `train_test_split`. This function takes `X`, `y`, and `test_size` as arguments, and output `X_train`, `X_test`, `y_train`, and `y_test` you obtained." ] }, { "cell_type": "code", "execution_count": 130, "metadata": {}, "outputs": [], "source": [ "def train_test_split(X: np.ndarray, y: np.ndarray, test_size: float=0.25):\n", " '''\n", " Randomly split the data into two sets.\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) dataset (features).\n", " y: np.ndarray\n", " (m,) dataset (corresponding targets).\n", " test_size: np.float64\n", " fraction of the dataset to be put in the test dataset.\n", "\n", " Returns\n", " -------\n", " A tuple of four elements (X_train, X_test, y_train, y_test):\n", " X_train: np.ndarray\n", " (m - k, n) training dataset (features).\n", " X_test: np.ndarray\n", " (k, n) testing dataset (features).\n", " y_train: np.ndarray\n", " (m - k,) training dataset (corresponding targets).\n", " y_test: np.ndarray\n", " (k,) testing dataset (corresponding targets).\n", " '''\n", "\n", " m = X.shape[0]\n", " k = int(m * test_size)\n", " mask = np.full(m, False)\n", " indices = np.random.choice(m, k, replace=False)\n", " mask[indices] = True\n", " inverse_mask = ~mask\n", "\n", " return X[inverse_mask], X[mask], y[inverse_mask], y[mask]\n" ] }, { "cell_type": "code", "execution_count": 131, "metadata": {}, "outputs": [], "source": [ "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 30, 0], \n", " [111.4, 40, 0], [111.5, 50, 0], [111.6, 60, 1],\n", " [111.7, 70, 0], [111.8, 80, 1], [111.9, 90, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "expected1 = [7, 2, 7, 2]\n", "\n", "X1_train, X1_test, y1_train, y1_test = train_test_split(X1, y1, test_size=0.25)\n", "\n", "assert len(X1_train) == expected1[0] and\\\n", " len(X1_test) == expected1[1] and\\\n", " len(y1_train) == expected1[2] and\\\n", " len(y1_test) == expected1[3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Logistic regression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Gradient descent\n", "\n", "Recall the gradient descent method you learned and applied in the previous problem set. Gradient descent is an iterative optimization method which finds the minimum of a differentiable function. It's often used to find the coefficients that minimize the cost function. Here is a brief recap of how that's done:\n", "\n", "
\n", "\"visualisation\n", "
Figure 4: Visualisation of gradient descent.
\n", "
\n", "\n", "We start at an initial weight vector $\\mathbf{w} = (w_0, w_1, ..., w_n)$. Then we take incremental steps along the steepest slope to get to the bottom where the minimum cost lies. Along the way, we keep updating the weight coefficients by computing the gradients using the training samples from the training dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Concept: Batch gradient descent\n", "\n", "In batch gradient descent, **all** the training data (entire batch) is taken into consideration in a single step. For one step of gradient descent, we calculate $w_j \\leftarrow w_j - \\alpha \\frac{\\partial J(\\mathbf{w})}{\\partial w_j}$ (where $\\alpha$ is the learning rate) simultaneously for all $j$ based on all the training examples and then use that mean gradient to update our parameters.\n", "\n", "By continuing this for iterations until convergence, or until it's time to stop, we reach the optimal or near optimal parameters $\\mathbf{w}$ leading to minimum cost." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Concept: Stochastic gradient descent\n", "\n", "Think about what's a limitation of the (entire) batch gradient descent introduced earlier. What if the dataset is very large? In Stochastic Gradient Descent (SGD), we consider only the error on a single record $(x^{(*)}, y^{(*)})$ at a time. How does this work?\n", "\n", "During the training loop, we will randomly sample an instance in the dataset. This instance will be used to update the weights of the model $w_j \\leftarrow w_j - \\alpha \\frac{\\partial J(\\mathbf{w})}{\\partial w_j}$ (where $\\alpha$ is the learning rate) simultaneously for all $j$.\n", "\n", "Note that SGD does not necessarily decrease the batch loss in each iteration! However, on average, the loss will still decrease. Since we work on only a single record at any time, SGD provides the advantage of a much smaller memory requirement, and faster computation time. SGD also comes with its own set of disadvantages, for example, we lose benefits of any vectorization." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Concept: Gradient descent termination condition\n", "\n", "The next question is when to stop? The ideal case is to run until convergence, but convergence might be hard to obtain. Here are some criterias you can use:\n", "\n", "* Stop when error change is small and/or\n", "* Stop when error is small\n", "* Stop when maximum number of iterations is reached" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's try to apply the batch gradient descent and stochastic gradient descent.\n", "Task 3.1 to 3.4 will gradually guide you to complete the logistic regression using batch gradient descent, while task 3.5 expects you to implement the stochastic one. For the task in this problem set, you can assume that the bias column has been added for all input `X`. That means given `X` as an argument, you don't need to manually add the bias column again in your code." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.1: Cost function\n", "\n", "Recall that for logistic regression, we want an error function for an individual value to be:\n", "\n", "
\n", "\"error\n", "
Figure 5: Cost function.
\n", "
\n", "\n", "We can simplify the condition and transform it into the equivalent cost function:\n", "\n", "
\n", "\"error\n", "
Figure 6: Cost without conditions.
\n", "
\n", "\n", "In this task, you need to implement `cost_function` $E$ as mentioned above. This function takes `X`, `y`, and `weight_vector` $\\mathbf{w}$ as arguments, and returns the error $E$. Note that for this task, the $E$ should account for **all** the training data.\n", "\n", "Here, we are using the $\\log$ function and we need to handle the case of computing $\\log(0)$. There are many ways to handle this. In this task, we will handle $\\log(0)$ by using the machine epsilon for numpy `float64` type, and use the trick $\\log(x + eps)$ which allows $x$ to be $0$. If $x$ was any other value, $\\log(x + eps)$ would be very close to $\\log(x)$. This helps to improve numerical stability in computation." ] }, { "cell_type": "code", "execution_count": 132, "metadata": {}, "outputs": [], "source": [ "def cost_function(X: np.ndarray, y: np.ndarray, weight_vector: np.ndarray):\n", " '''\n", " Cross entropy error for logistic regression\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) training dataset (features).\n", " y: np.ndarray\n", " (m,) training dataset (corresponding targets).\n", " weight_vector: np.ndarray\n", " (n,) weight parameters.\n", "\n", " Returns\n", " -------\n", " Cost\n", " '''\n", " \n", " # Machine epsilon for numpy `float64` type\n", " eps = np.finfo(np.float64).eps\n", " y_predicted = 1/(1+np.exp(-X @ weight_vector)) + eps\n", " first = -y * np.log(y_predicted)\n", " second = (1-y) * np.log(1-y_predicted)\n", "\n", " return np.mean(first-second)" ] }, { "cell_type": "code", "execution_count": 133, "metadata": {}, "outputs": [], "source": [ "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [111.6, 10, 1], [111.4, 10, 0], [111.5, 10, 1], [111.6, 10, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "w1 = np.transpose([0.002, 0.1220])\n", "\n", "assert np.round(cost_function(X1, y1, w1), 5) == np.round(1.29333, 5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.2: Weight update\n", "\n", "In this task, you need to implement `weight_update`. This function takes `X`, `y`, `alpha`, and a `weight_vector` as arguments, and output the new weight vector. Each call to the function should make one update on the weight vector." ] }, { "cell_type": "code", "execution_count": 134, "metadata": {}, "outputs": [], "source": [ "def weight_update(X: np.ndarray, y: np.ndarray, alpha: np.float64, weight_vector: np.ndarray) -> np.ndarray:\n", " '''\n", " Do the weight update for one step in gradient descent\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) training dataset (features).\n", " y: np.ndarray\n", " (m,) training dataset (corresponding targets).\n", " alpha: np.float64\n", " logistic regression learning rate.1\n", " weight_vector: np.ndarray\n", " (n,) weight parameters.\n", "\n", " Returns\n", " -------\n", " New weight vector after one round of update.\n", " '''\n", " y_predicted = 1/(1+np.exp(-X @ weight_vector))\n", " \n", " return weight_vector - alpha * X.T @ (y_predicted - y) / len(y)" ] }, { "cell_type": "code", "execution_count": 135, "metadata": {}, "outputs": [], "source": [ "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [111.6, 10, 1],[111.4, 10, 0], [111.5, 10, 1], [111.6, 10, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "w1 = np.transpose([2.2000, 12.20000])\n", "a1 = 1e-5\n", "nw1 = np.array([2.199,12.2])\n", "\n", "assert np.array_equal(np.round(weight_update(X1, y1, a1, w1), 3), nw1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.3: Logistic regression classification\n", "\n", "Remember that logistic regression is used for classification even though the function gives a probability output. In this task, you classify each element in `X`, given `weight_vector` using `prob_threshold` as the threshold, and output the classification result as an `np.ndarray`.\n", "\n", "If the probability predicted by the `weight_vector` exceeds the `prob_threshold`, we should classify it as fraud (`y = 1`). Otherwise, we should classify it as legitimate (`y = 0`)." ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [], "source": [ "def logistic_regression_classification(X: np.ndarray, weight_vector: np.ndarray, prob_threshold: np.float64=0.5):\n", " '''\n", " Do classification task using logistic regression.\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) training dataset (features).\n", " weight_vector: np.ndarray\n", " (n,) weight parameters.\n", " prob_threshold: np.float64\n", " the threshold for a prediction to be considered fraudulent.\n", "\n", " Returns\n", " -------\n", " Classification result as an (m,) np.ndarray\n", " '''\n", " y_predicted = 1/(1+np.exp(-X @ weight_vector))\n", " return (y_predicted > prob_threshold).astype(int)" ] }, { "cell_type": "code", "execution_count": 137, "metadata": {}, "outputs": [], "source": [ "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [211.6, 80, 1],[111.4, 10, 0], [111.5, 80, 1], [211.6, 80, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "w1 = np.transpose([-0.000002, 0.000003])\n", "expected1 = np.transpose([0, 0, 0, 0, 0, 0, 0, 1, 0])\n", "result1 = logistic_regression_classification(X1, w1)\n", "\n", "assert result1.shape == expected1.shape and (result1 == expected1).all()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.4: Logistic regression using batch gradient descent\n", "\n", "In this task, you need to implement a fixed learning rate algorithm for logistic regression. This function takes `X_train`, `y_train`, `max_num_epochs`, `threshold`, and `alpha` as arguments, and output the final `weight_vector` you obtained. You can make use of the previous functions you have implemented earlier.\n", "\n", "Specifically, an epoch refers to one complete pass through the dataset. Hence, one iteration of gradient descent, which simultaneously updates the weight vector based on the cost function computed on the entire dataset, corresponds to one epoch. Batch gradient descent is terminated when the number of update rounds exceeds the maximum number of epochs (`max_num_epochs`) given, or when the computed error falls below the threshold value (`threshold`)." ] }, { "cell_type": "code", "execution_count": 138, "metadata": {}, "outputs": [], "source": [ "def logistic_regression_batch_gradient_descent(X_train: np.ndarray, y_train: np.ndarray, max_num_epochs: int = 250, threshold: np.float64 = 0.05, alpha: np.float64 = 1e-5):\n", " '''\n", " Initialize your weight to zeros. Write your terminating condition, and run the weight update for some iterations.\n", " Get the resulting weight vector. Use that to do predictions, and output the final weight vector.\n", "\n", " Parameters\n", " ----------\n", " X_train: np.ndarray\n", " (m, n) training dataset (features).\n", " y_train: np.ndarray\n", " (m,) training dataset (corresponding targets).\n", " max_num_epochs: int\n", " this should be one of the terminating conditions. \n", " That means if you initialize num_update_rounds to 0, \n", " then you should stop updating the weights when num_update_rounds >= max_num_epochs.\n", " threshold: np.float64\n", " terminating when error <= threshold value, or if you reach the max number of update rounds first.\n", " alpha: np.float64\n", " logistic regression learning rate.\n", "\n", " Returns\n", " -------\n", " The final (n,) weight parameters\n", " '''\n", " weights = np.zeros(X_train.shape[1])\n", " print(weights)\n", " for _ in range(max_num_epochs):\n", " weights = weight_update(X_train, y_train, alpha, weights)\n", " if cost_function(X_train, y_train, weights) <= threshold:\n", " break\n", " return weights\n" ] }, { "cell_type": "code", "execution_count": 139, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0. 0.]\n" ] } ], "source": [ "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [211.6, 80, 1],[111.4, 10, 0], [111.5, 80, 1], [211.6, 80, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "max_num_epochs1 = 20\n", "expected1 = np.transpose([-0.0011538251272737235, 0.0019538526326272542])\n", "\n", "assert np.array_equal(np.round(logistic_regression_batch_gradient_descent(X1, y1, max_num_epochs1), 5), np.round(expected1, 5))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.5: Logistic regression using stochastic gradient descent\n", "\n", "In this task, you need to implement logistic regression using stochastic gradient descent. This function takes `X_train`, `y_train`, `max_num_iterations`, `threshold`, and `alpha` as arguments, and output the final `weight_vector` you obtained. You can make use of the previous functions you have implemented earlier.\n", "\n", "For this task, you should use `np.random.choice` to randomly choose the training data for each iteration. You should also use the `seed` argument to set the random seed for reproducibility." ] }, { "cell_type": "code", "execution_count": 182, "metadata": {}, "outputs": [], "source": [ "def weight_update_stochastic(X: np.ndarray, y: np.ndarray, alpha: np.float64, weight_vector: np.ndarray) -> np.ndarray:\n", " '''\n", " Do the weight update for one step in gradient descent.\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (1, n) training dataset (features).\n", " y: np.ndarray\n", " one y in training dataset (corresponding targets).\n", " alpha: np.float64\n", " logistic regression learning rate.\n", " weight_vector: np.ndarray\n", " (n, 1) vector of weight parameters.\n", "\n", " Returns\n", " -------\n", " New (n,) weight parameters after one round of update.\n", " '''\n", "\n", " index = np.random.choice(X.shape[0], 1)\n", " X_sel = X[index]\n", " y_sel = y[index]\n", " y_predicted = 1/(1+np.exp(-X_sel @ weight_vector))\n", " return weight_vector - alpha * X_sel.T @ (y_predicted - y_sel)" ] }, { "cell_type": "code", "execution_count": 183, "metadata": {}, "outputs": [], "source": [ "def logistic_regression_stochastic_gradient_descent(X_train: np.ndarray, y_train: np.ndarray, max_num_iterations: int=250, threshold: np.float64=0.05, alpha: np.float64=1e-5, seed :int=43) -> np.ndarray:\n", " '''\n", " Initialize your weight to zeros. Write a terminating condition, and run the weight update for some iterations.\n", " Get the resulting weight vector.\n", "\n", " Parameters\n", " ----------\n", " X_train: np.ndarray\n", " (m, n) training dataset (features).\n", " y_train: np.ndarray\n", " (m,) training dataset (corresponding targets).\n", " max_num_iterations: int\n", " this should be one of the terminating conditions. \n", " The gradient descent step should happen at most max_num_iterations times.\n", " threshold: np.float64\n", " terminating when error <= threshold value, or if you reach the max number of update rounds first.\n", " alpha: np.float64\n", " logistic regression learning rate.\n", " seed: int\n", " seed for random number generation.\n", "\n", " Returns\n", " -------\n", " The final (n,) weight parameters\n", " '''\n", " np.random.seed(seed)\n", "\n", " weights = np.zeros(X_train.shape[1])\n", " error = cost_function(X_train, y_train, weights)\n", " for i in range(max_num_iterations):\n", " if error <= threshold:\n", " break\n", " weights = weight_update_stochastic(X_train, y_train, alpha, weights)\n", " error = cost_function(X_train, y_train, weights)\n", " return weights" ] }, { "cell_type": "code", "execution_count": 184, "metadata": {}, "outputs": [], "source": [ "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [211.6, 80, 1],[111.4, 10, 0], [111.5, 80, 1], [211.6, 80, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "expected1 = cost_function(X1, y1, np.transpose(np.zeros(X1.shape[1])))\n", "\n", "assert cost_function(X1, y1, logistic_regression_stochastic_gradient_descent(X1, y1)) < expected1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.6.1: Stochastic gradient descent vs batch gradient descent\n", "\n", "Recall that it is ideal for gradient descent to run until convergence, but convergence might be hard to obtain.\n", "\n", "In this task, we will explore the relationship between the cross entropy loss vs the number of update rounds and/or the runtime it takes to run the update for both batch gradient descent (task 3.4) and stochastic gradient descent (task 3.5) using sample of `credit_card` dataset.\n", "\n", "To guide us in our explorations, let's start by collecting empirical results from running stochastic gradient descent and batch gradient descent. You will need to make the following 2 plots with the stated specifications:\n", "\n", "1. Plot of cross entropy loss against number of update rounds\n", " - $x$-axis label: \"Number of Update Rounds\"\n", " - $y$-axis label: \"Cross Entropy Loss\"\n", " - Title: \"Plot of cross entropy loss against number of update rounds\"\n", " - Legend to specify the data points corresponding to \"Batch Gradient Descent\" and \"Stochastic Gradient Descent\" respectively\n", "2. Plot of cross entropy loss against runtime (sec)\n", " - $x$-axis label: \"Runtime (sec)\"\n", " - $y$-axis label: \"Cross Entropy Loss\"\n", " - Title: \"Plot of cross entropy loss against runtime (sec)\"\n", " - Legend to specify the data points corresponding to \"Batch Gradient Descent\" and \"Stochastic Gradient Descent\" respectively\n", "\n", "For both plots, put both methods (batch gradient descent and stochastic gradient descent) in one diagram. Also ensure that the runtime is the cumulative runtime not the runtime per individual update round. \n", "\n", "You can refer to following script for creating these plots. Feel free to modify it if necessary.\n", "\n", "Once you are done, paste your plots in the corresponding answer box in Coursemology." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "from time import time\n", "\n", "# X_sample, y_sample = \n", "# num_interations = \n", "# batch_times = []\n", "# batch_costs = []\n", "\n", "# for i in range(50, num_interations + 1, 50):\n", "# start = time()\n", "# weight_vector = logistic_regression_batch_gradient_descent(X_sample, y_sample, i, 0, 1e-5)\n", "# stop = time()\n", "# batch_times.append(stop - start)\n", "# batch_costs.append(cost_function(X_sample, y_sample, weight_vector))\n", "# plt.plot(batch_times, batch_costs, 'g^', label=\"Batch Gradient Descent\")\n", "\n", "# stochastic_times = []\n", "# stochastic_costs = []\n", "# for i in range(50, num_interations + 1, 50):\n", "# start = time()\n", "# weight_vector = logistic_regression_stochastic_gradient_descent(X_sample, y_sample, i, 0, 1e-5)\n", "# stop = time()\n", "# stochastic_times.append(stop - start)\n", "# stochastic_costs.append(cost_function(X_sample, y_sample, weight_vector))\n", "# plt.plot(stochastic_times, stochastic_costs, 'bs', label=\"Stochastic Gradient Descent\")\n", "\n", "# plt.xlabel('Runtime (sec)')\n", "# plt.ylabel('Cross Entropy Loss')\n", "# plt.legend()\n", "# plt.title('Plot of cross entropy loss against runtime (sec)')\n", "\n", "# plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 3.6.2: Stochastic gradient descent vs batch gradient descent\n", "\n", "With reference to the 2 plots that you have made in task 3.6.1, what do you observe about the relationship between the cross entropy loss vs the number of update rounds and/or the runtime it takes to run the update for both batch gradient descent (task 3.4) and stochastic gradient descent (task 3.5)? Explain your observations in terms of the effect of size of data, number of update rounds, runtime and whether the algorithm will be stuck in a local minima and so on." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Multi-class logistic regression\n", "\n", "Now that you have helped the credit card companies recognize fraudulent transactions, customers are less likely to be charged for items that they did not purchase. They are very satisfied with how well your model performed. They want to offer you a meal, at a restaurant of your choice. You decide to make your choice based on the occupancy of the restaurant. Luckily for you, you have some starting data `restaurant_data.csv` to work with. The dataset provides three input features \"max_capacity\", \"feedback_score\", and \"average_expense\", and an output feature \"occupancy\" that takes the value of \"none\", \"some\", or \"full\"." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 4.1: Multi-class logistic regression using batch gradient descent\n", "\n", "In this task, you need to implement a fixed learning rate algorithm for multi-class logistic regression. This function takes `X_train`, `y_train`, `max_num_epochs`, `threshold`, `alpha`, and `class_i` as arguments, and output the final `weight_vector` you obtained. Here, argument `class_i` is the class that you want to train in this round. You can make use of the previous functions you have implemented earlier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def multi_class_logistic_regression_batch_gradient_descent(X_train: np.ndarray, y_train: np.ndarray, max_num_epochs: int, threshold: np.float64, alpha: np.float64, class_i: str) -> np.ndarray:\n", " '''\n", " Initialize your weight to zeros. Write your terminating condition, and run the weight update for some iterations.\n", " Get the resulting weight vector. Output the resulting weight vector.\n", "\n", " Parameters\n", " ----------\n", " X_train: np.ndarray\n", " (m, n) training dataset (features).\n", " y_train: np.ndarray\n", " (m,) training dataset (corresponding targets).\n", " alpha: np.float64\n", " logistic regression learning rate.\n", " max_num_epochs: int\n", " this should be one of the terminating conditions. \n", " That means if you initialize num_update_rounds to 0, \n", " then you should stop updating the weights when num_update_rounds >= max_num_epochs.\n", " threshold: float\n", " terminating when error <= threshold value, or if you reach the max number of update rounds first.\n", " class_i: string\n", " one of 'none', 'full', 'some'.\n", "\n", " Returns\n", " -------\n", " The final (n,) weight parameters\n", " '''\n", "\n", " # TODO: add your solution here and remove `raise NotImplementedError`\n", " raise NotImplementedError" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data1 = [[26, 9, 69, 'full'],\n", " [54, 3, 16, 'some'],\n", " [59, 7, 50, 'some' ],\n", " [33, 0, 45, 'full']]\n", "df1 = pd.DataFrame(data1, columns = ['max_capcity', 'feedback_score', 'average_expense', 'occupancy'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "max_num_epochs1 = 20\n", "expected1 = np.transpose([6.75, 0.125, -6.0])\n", "assert np.array_equal(multi_class_logistic_regression_batch_gradient_descent(X1, y1, max_num_epochs1, 0.05, 1, 'some'), expected1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 4.2: Do classification on multi-class problem\n", "\n", "Finally, you classify each element in `X`, given three weight vectors `weight_vector_none`, `weight_vector_some`, and `weight_vector_full`, and output a list of classification results in the form of an array (`['some', 'none', 'full', 'none', ...]`). Note that in the case of a tie, you can choose the class in the order of priority: `'none'`, `'some'`, `'full'`. You can make use of the previous functions you have implemented earlier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def multi_class_logistic_regression_classification(X: np.ndarray, weight_vector_none: np.ndarray, weight_vector_some: np.ndarray, weight_vector_full: np.ndarray):\n", " '''\n", " Do classification task using logistic regression.\n", " In the case of a tie, break the tie in the priority 'none' > 'some' > 'full'.\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) training dataset (features)\n", " weight_vector_none: np.ndarray\n", " (n,) weight parameters for the 'none' class.\n", " weight_vector_some: np.ndarray\n", " (n,) weight parameters for the 'some' class.\n", " weight_vector_full: np.ndarray\n", " (n,) weight parameters for the 'full' class.\n", "\n", " Returns\n", " -------\n", " Classification result as an (m,) np.ndarray. Eg ['some', 'none', 'full', ... ,'none'].\n", " '''\n", "\n", " # TODO: add your solution here and remove `raise NotImplementedError`\n", " raise NotImplementedError" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data1 = [[26, 9, 69, 'full'],\n", " [54, 3, 16, 'some'],\n", " [59, 7, 50, 'some' ],\n", " [33, 0, 45, 'full']]\n", "df1 = pd.DataFrame(data1, columns = ['max_capcity', 'feedback_score', 'average_expense', 'occupancy'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "\n", "w11 = np.transpose([0.0013351567670329624, 2.5757816929896605e-05, -0.001189020140476165])\n", "w12 = np.transpose([2.5757816929896605e-05, -0.001189020140476165, 0.0013351567670329624])\n", "w13 = np.transpose([2.5757816929896605e-05, 0.0013351567670329624, -0.001189020140476165])\n", "expected1 = np.transpose(['some', 'none', 'some', 'some'])\n", "\n", "result1 = multi_class_logistic_regression_classification(X1, w11, w12, w13)\n", "\n", "assert result1.shape == expected1.shape and (result1 == expected1).all()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Support Vector Machine\n", "\n", "Now let's apply the support vector machine we learnt in lecture to the credit card dataset. Here is a quick recap of what SVM is about: We are given a training dataset of $n$ points of the form $(x^{(i)}, y^{(i)})$, where the $y^{(i)}$ are either 1 or 0, each indicating the class to which the point $x^{(i)}$ belongs.\n", "\n", "Each $x^{(i)}$ $n$-dimensional real vector. We want to find the \"maximum-margin hyperplane\" that divides the group of points $x^{(i)}$ for which $y_{i}=1$ from the group of points for which $y_{i}=0$, which is defined so that the distance between the hyperplane and the nearest point $x^{(i)}$ from either group is maximized.\n", "\n", "
\n", "\"visualisation\n", "
Figure 7: Visualisation of support vector machine.
\n", "
\n", "\n", "How can we do construct such a \"maximum-margin hyperplane\"? Recall that in the lecture, we formulated SVMs as the following minimization problem:\n", "\n", "$$\n", "\\min_{w} C \\left[\\sum_{i = 1}^{m} y^{(i)}\\ Cost_1(\\mathbf{w}^{T} \\mathbf{x}^{(i)}) + (1 - y^{(i)})\\ Cost_0(\\mathbf{w}^{T} \\mathbf{x}^{(i)})\\right] + \\frac{1}{2} \\sum_{i = 1}^{n} \\mathbf{w}_i^2 $$\n", "\n", "where $Cost_1(\\mathbf{w}^T \\mathbf{x}^{(i)}) = \\max(0, 1 - \\mathbf{w}^T \\mathbf{x}^{(i)})$ and $Cost_0(\\mathbf{w}^T \\mathbf{x}^{(i)}) = \\max(0, 1 + \\mathbf{w}^T \\mathbf{x}^{(i)})$, with the hypothesis function $h_\\mathbf{w}(\\mathbf{w}) = 1$ if $\\mathbf{w}^T \\mathbf{x} \\geq 0$, and 0 otherwise.\n", "\n", "In particular, the summand term is known as the *hinge loss*. Thus, when $y^{(i)} = 0$, the more negative $\\mathbf{w}^T \\mathbf{x}$ is, the more $Cost_1$ penalises the model. Similarly, $Cost_0$ penalises the model the further $\\mathbf{w}^T \\mathbf{x}$ strays from the negative region. Notice how the cost function aligns with the goal of the hypothesis function?\n", "\n", "Furthermore, we also add a regularization parameter $C$ to the cost function to balance the trade-off between margin maximization and loss.\n", "\n", "Thus, solving this minimization problem allows us to find the weight vector $\\mathbf{w}$ that provides the maximum-margin hyperplane while balancing the trade-off of overfitting.\n", "\n", "With this set up in mind, how can we implement this in code? Thankfully, with modern machine learning libraries, much of the tedious work of solving this minimization problem has been done. As machine learning practitioners, one popular library that we commonly use (aside from [NumPy](https://numpy.org/doc/stable/index.html)) is [scikit-learn](https://scikit-learn.org/), which is built on top of NumPy and provides simple and efficient tools to implement common machine learning algorithms.\n", "\n", "In particular, we will look at how we can use the [scikit-learn](https://scikit-learn.org/) library to implement SVM." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 5.1: Linear SVM\n", "\n", "Take a look at the Python library [scikit-learn](https://scikit-learn.org/stable/modules/svm.html). It has been imported for your use. In this task, the function takes `X`, `y`. Note that here `X` and `y` are not training set but the entire dataset, so you are asked to do a train test data split with `test_size` of 0.3, and `random_state` of 42, **using what's provided in the library** (not what you implemented in task 2.1). Then, create an instance of a Linear SVM classifier using the default parameters, train the Linear SVM classifier, and output the predictions as well as the accuracy score. Note that the default kernel in sklearn is RBF, so you need to specify the kernel type you want to use.\n", "\n", "**For this task, you should make full use of the library functions, instead of doing computations on your own.**\n", "\n", "In particular, you can look into the scikit-learn methods to perform a [train-test data split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) and employ the [SVC model](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def linear_svm(X: np.ndarray, y: np.ndarray):\n", " '''\n", " Do classification using linear svm. Given X and y, note that here X and y are not training sets, but rather the\n", " entire dataset. Do a train test data split with test_size=0.3, and random_state=42.\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) whole dataset (features)\n", " y: np.ndarray\n", " (m,) whole dataset (corresponding targets)\n", "\n", " Returns\n", " -------\n", " pred: np.ndarray\n", " The predictions.\n", " acc: np.float64\n", " The accuracy on a scale up to 100.\n", " '''\n", "\n", " # TODO: add your solution here and remove `raise NotImplementedError`\n", " raise NotImplementedError" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# small data\n", "data1 = [[111.1, 10, 0], [111.2, 20, 0], [111.3, 10, 0], [111.4, 10, 0], [111.5, 10, 0], [211.6, 80, 1],\n", " [111.4, 10, 0], [111.5, 80, 1], [211.6, 80, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "expected1_y = np.transpose([0, 0, 1])\n", "expected1_accuracy = 66.66666666666666\n", "result1 = linear_svm(X1, y1)\n", "assert (result1[0] == expected1_y).all() and (result1[0]).shape == expected1_y.shape and round(result1[1], 5) == round(expected1_accuracy, 5)\n", "\n", "# subset of credit card data\n", "class_0 = credit_df[credit_df['Class'] == 0]\n", "class_1 = credit_df[credit_df['Class'] == 1]\n", "\n", "data_0 = class_0.sample(n=15, random_state=42)\n", "data_1 = class_1.sample(n=50, random_state=42)\n", "data_100 = pd.concat([data_1, data_0], axis=0)\n", "X = data_100.iloc[:, :-1].to_numpy()\n", "y = data_100.iloc[:, -1].to_numpy()\n", "\n", "expected_pred = np.transpose([1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])\n", "expected_accuracy = 80.0\n", "result = linear_svm(X, y.ravel())\n", "assert (result[0] == expected_pred).all() and (result[0]).shape == expected_pred.shape and round(result[1], 5) == round(expected_accuracy, 5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 5.2: Gaussian Kernel SVM\n", "\n", "Similarly, you will use scikit-learn to implement a Gaussian Kernel SVM. In this task, the function takes `X`, `y`. Note that here `X` and `y` are not training set but the entire dataset, so you are asked to do a train test data split with `test_size` of 0.3, and `random_state` of 42, using what's provided in the library. Then, create an instance of a Gaussian Kernel SVM classifier using the default parameters, train the Gaussian Kernel SVM classifier, and output the predictions as well as the accuracy score.\n", "\n", "**For this task, you should make full use of the library functions, instead of doing computations on your own.**\n", "\n", "As before, you can look into the scikit-learn methods to perform a [train-test data split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) and employ the [SVC model](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def gaussian_kernel_svm(X: np.ndarray, y: np.ndarray):\n", " '''\n", " Do classification using Gaussian Kernel svm. Given X and y, note that here X and y are not training sets, but\n", " rather the entire dataset. Do a train test data split with test_size=0.3, and random_state=42.\n", "\n", " Parameters\n", " ----------\n", " X: np.ndarray\n", " (m, n) whole dataset (features)\n", " y: np.ndarray\n", " (m,) whole dataset (corresponding targets)\n", "\n", " Returns\n", " -------\n", " pred: np.ndarray\n", " The predictions.\n", " acc: np.float64\n", " The accuracy on a scale up to 100.\n", " '''\n", "\n", " # TODO: add your solution here and remove `raise NotImplementedError`\n", " raise NotImplementedError" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# small data\n", "data1 = [[111.1, 10, -1], [111.2, 20, -1], [111.3, 10, -1], [111.4, 10, -1], [111.5, 10, -1], [211.6, 80, 1],\n", " [111.4, 10, -1], [111.5, 80, 1], [211.6, 80, 1]]\n", "df1 = pd.DataFrame(data1, columns = ['V1', 'V2', 'Class'])\n", "X1 = df1.iloc[:, :-1].to_numpy()\n", "y1 = df1.iloc[:, -1].to_numpy()\n", "expected1_y = np.transpose([-1, -1, 1])\n", "expected1_accuracy = 66.66666666666666\n", "result1 = gaussian_kernel_svm(X1, y1)\n", "assert (result1[0] == expected1_y).all() and (result1[0]).shape == expected1_y.shape and round(result1[1], 5) == round(expected1_accuracy, 5)\n", "\n", "\n", "# subset of credit card data\n", "class_0 = credit_df[credit_df['Class'] == 0]\n", "class_1 = credit_df[credit_df['Class'] == 1]\n", "\n", "data_0 = class_0.sample(n=15, random_state=42)\n", "data_1 = class_1.sample(n=50, random_state=42)\n", "data_100 = pd.concat([data_1, data_0], axis=0)\n", "X = data_100.iloc[:, :-1].to_numpy()\n", "y = data_100.iloc[:, -1].to_numpy()\n", "\n", "expected_pred = np.transpose([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])\n", "expected_accuracy = 80.0\n", "result = gaussian_kernel_svm(X, y.ravel())\n", "assert (result[0] == expected_pred).all() and (result[0]).shape == expected_pred.shape and round(result[1], 5) == round(expected_accuracy, 5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task 5.3: Linear SVM vs Gaussian Kernel SVM\n", "\n", "Based on your observations, when using support vector machines, how do you think we should choose between linear kernel vs Gaussian kernel? In other words, which kernel is better in what cases?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submission\n", "\n", "Once you are done, please submit your work to Coursemology, by copying the right snippets of code into the corresponding box that says \"Your answer,\"and click \"Save.\" After you save, you can still make changes to your submission.\n", "\n", "Once you are satisfied with what you have uploaded, click \"Finalize submission.\" Note that once your submission is finalized, it is considered to be submitted for grading and cannot be changed. If you need to undo this action, you will have to email your assigned tutor for help. Please do not finalize your submission until you are sure that you want to submit your solutions for grading.\n", "\n", "*Have fun and enjoy coding.*" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.9" }, "vscode": { "interpreter": { "hash": "a665b5d41d17b532ea9890333293a1b812fa0b73c9c25c950b3cedf1bebd0438" } } }, "nbformat": 4, "nbformat_minor": 2 }