{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "wa1XEEuWsrHE" }, "source": [ "# Tutorial 2: Random Forest Regression\n", "\n", "In this tutorial you will learn how to read and explore census data from the [IPUMS USA](https://usa.ipums.org/usa/) database. IPUMS USA collects, preserves and harmonizes U.S. census microdata and provides easy access to this data with enhanced documentation. Data includes decennial censuses from 1790 to 2010 and American Community Surveys (ACS) from 2000 to the present. In this tutorial, you will learn how to set up a [Random Forest model](https://elcok.github.io/BigData_AED/week2/lecture.html) using real-world data.\n", "\n", "### Important before we start\n", "---\n", "Make sure that you save this file before you continue, else you will lose everything. To do so, go to **Bestand/File** and click on **Een kopie opslaan in Drive/Save a Copy on Drive**!\n", "\n", "Now, rename the file into Week2_Tutorial2.ipynb. You can do so by clicking on the name in the top of this screen." ] }, { "cell_type": "markdown", "metadata": { "id": "vu0iEQ6nsrHL" }, "source": [ "

Tutorial Outline

\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "QyG3hfkEsrHM" }, "source": [ "## Learning Objectives\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "A1zOPoE6srHN" }, "source": [ "- Learn how to estimate a random forest model.\n", "- Deal with missing values in the model.\n", "- Evaluate performance of the random forest model.\n", "- Tuning the hyperparameters." ] }, { "cell_type": "markdown", "metadata": { "id": "4A-CxMMFsrHO" }, "source": [ "## 1. Introducing the packages\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "xHuaqP85srHO" }, "source": [ "Within this tutorial, we are going to make use of the following packages:\n", "\n", "[**sklearn**](https://scikit-learn.org/stable/index.html) is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection, model evaluation, and many other utilities.\n", "\n", "[**seaborn**](https://seaborn.pydata.org/index.html) is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.\n", "\n", "[**NumPy**](https://numpy.org/doc/stable/) is a Python library that provides a multidimensional array object, various derived objects, and an assortment of routines for fast operations on arrays.\n", "\n", "[**Pandas**](https://pandas.pydata.org/docs/) is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language." ] }, { "cell_type": "markdown", "metadata": { "id": "2_Ya4EmNsrHS" }, "source": [ "Now we will import these packages in the cell below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Yog-09VPsrHT" }, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestRegressor\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import r2_score\n", "from sklearn.metrics import mean_squared_error\n", "from sklearn.tree import plot_tree\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "\n", "pd.options.mode.chained_assignment = None\n", "sns.set_style(\"whitegrid\",{'axes.grid' : True})" ] }, { "cell_type": "markdown", "metadata": { "id": "sX0eAbSvsrHU" }, "source": [ "## 2. Importing the data\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "h-F-kn8DsrHV" }, "source": [ "Let's start again with reading the data. It is the dataset that we produced in the previous [Tutorial](https://elcok.github.io/BigData_AED/week2/tutorial1.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 338 }, "id": "Q4KRGG11srHV", "outputId": "062ed87c-9e4b-4976-9541-1bec0662267f" }, "outputs": [], "source": [ "data_in = pd.read_csv(r\"https://github.com/ElcoK/BigData_AED/raw/main/week2/usadataforweek2tut2.csv\")" ] }, { "cell_type": "markdown", "metadata": { "id": "m77b6Ha3srHW" }, "source": [ "And let's have a look at the data once more:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tJvtq16vsrHW" }, "outputs": [], "source": [ "data_in.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "rZwhvWRDsrHX" }, "source": [ "In the following line of code we drop duplicates based on the variable SERIAL. Remember that persons from the same household have the same SERIAL number." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "g5y5--2KsrHX", "scrolled": true }, "outputs": [], "source": [ "data = data_in.drop_duplicates(subset='SERIAL', keep=\"first\")" ] }, { "cell_type": "markdown", "metadata": { "id": "5-wJuSC2srHX" }, "source": [ "
\n", "Question 1: Why do we only want to have one person per household in our random forest model?\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "OLG02hVtsrHY" }, "source": [ "## 3. Dealing with missing values\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "kGHMJyEIsrHY" }, "source": [ "In the following lines of code we try to estimate a random forest model, but we run into an error because we have missing values in the dataset.\n", "\n", "We don't estimate the random forest model on the entire dataset, but we split the dataset in a training and testing sample, because we want to train the model on the training dataset and to test model performance on the test dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-McgXSi7srHZ" }, "outputs": [], "source": [ "training_data, testing_data = train_test_split(data, test_size=0.3, random_state=25)" ] }, { "cell_type": "markdown", "metadata": { "id": "R2hFhVTqsrHZ" }, "source": [ "Next we split the training and testing data in two datasets, where x_train contains the explanatory variables and y_train contains the dependent variable. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "I4D_JWtBsrHa" }, "outputs": [], "source": [ "x_train = training_data.drop(columns = ['COSTENERGY'])\n", "y_train = training_data[['COSTENERGY']]\n", "y_train = y_train.to_numpy() #here we convert y_train from a pandas dataframe with one column to a numpy array.\n", "\n", "x_test = testing_data.drop(columns = ['COSTENERGY'])\n", "y_test = testing_data[['COSTENERGY']]\n", "y_test = y_test.to_numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "fujmsmiVsrHa" }, "source": [ "Training and fitting a random forest model is done in one line of code:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qPHuv1o7srHb" }, "outputs": [], "source": [ "rf = RandomForestRegressor(n_estimators = 50, max_features = 'sqrt', max_depth = 5, random_state = 18).fit(x_train, y_train)" ] }, { "cell_type": "markdown", "metadata": { "id": "l3SJedjgsrHb" }, "source": [ "
\n", "Question 2: Explain in your own words the options in the RandomForestRegressor and how these options will affect the results. Also explain two other options in RandomForestRegressor that are not included in the line of code we have used to run the model.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "nllOrcLusrHc" }, "source": [ "The random forest model didn't run because there are missing values in the data. There are several options to deal with missing data. We could for example replace all the missing values by 0 or by the mean of that variable. First, let's check again which variables have missing values." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "a9NdIMrMsrHc" }, "outputs": [], "source": [ "data.isnull().sum().sort_values(ascending = False)" ] }, { "cell_type": "markdown", "metadata": { "id": "oQu21HtbsrHc" }, "source": [ "There are quite some missing values in the dependent variable **COSTENERGY**. And the number of NA's in **COSTENERGY** is suspiciously similar to the number of missing values in some other variables, such as **BUILTYR2**, **ROOMS**, **HHINCOME**, etc. Let's check what happens if we remove the rows from the dataset when there is no value for **COSTENERGY**. Probably we can remove a lot of missing values all at once. \n", "\n", "In the following line of code we remove all rows (`axis = 0` indicates rows, and `axis = 1` indicates columns) for which **COSTENERGY** is NA." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Cb8CwwCvsrHd" }, "outputs": [], "source": [ "datanan = data.dropna(axis=0, subset=['COSTENERGY'])" ] }, { "cell_type": "markdown", "metadata": { "id": "iIrYEz9wsrHd" }, "source": [ "Now let's check if this has removed a lot of NA's in other variables as well." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "wElwRTPesrHd" }, "outputs": [], "source": [ "datanan.isnull().sum().sort_values(ascending = False)" ] }, { "cell_type": "markdown", "metadata": { "id": "vVZHfPT7srHe" }, "source": [ "Indeed for 15145 households there was a lot of data missing and we can savely remove these households from the dataset, because 1) these instances are not very meaningful and 2) we could perfectly predict them if we keep them in the dataset using the missing values of **VEHICLES**, **ROOMS**, **HHINCOME**, etc." ] }, { "cell_type": "markdown", "metadata": { "id": "SQSp8mPssrHe" }, "source": [ "Now we have a handful variables left with missing values. Replace the missing values in all variables with code. You can check the code of the [previous tutorial](https://elcok.github.io/BigData_AED/week2/tutorial1.html) on how to do this." ] }, { "cell_type": "markdown", "metadata": { "id": "Bn-lL_OAsrHe" }, "source": [ "
\n", "Question 3: Explain what you would do with the remaining missing values in our data? There is not necessarily a right or wrong answer, but ofcourse your choices matter for the results.
\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "md4-3IYhsrHe" }, "outputs": [], "source": [ "datanan['RENT'] = datanan['RENT'] # fill na\n", "datanan['DEGFIELD'] = datanan['DEGFIELD'] # fill na\n", "datanan['MORTGAG2'] = datanan['MORTGAG2'] # fill na\n", "datanan['MORTGAGE'] = datanan['MORTGAGE'] # fill na\n", "datanan['PROPINSR'] = datanan['PROPINSR'] # fill na\n", "datanan['IND'] = datanan['IND'] # fill na\n", "datanan['OCC'] = datanan['OCC'] # fill na\n", "datanan['EMPSTAT'] = datanan['EMPSTAT'] # fill na" ] }, { "cell_type": "markdown", "metadata": { "id": "XSxk7_7JsrHe" }, "source": [ "## 4. Training the random forest model\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "kMdEDMYusrHf" }, "source": [ "Now we can actually estimate the random forest model. Note that we use the phrases *training a random forest model* or *estimating a the random forest model* interchangebly. In machine learning, we usually talk about training a model, while in statistics/econometrics, we would say estimating a model, but both terms mean essentially the same.\n", "\n", "Remember to split the data again in a training and testing set. Now we use the datanan dataset which has no missing values. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "TW-XHnQOsrHf" }, "outputs": [], "source": [ "training_data, testing_data = train_test_split(datanan, test_size=0.3, random_state=25)" ] }, { "cell_type": "markdown", "metadata": { "id": "HlHkvby2srHf" }, "source": [ "And now let's rerun the model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "n7TwRqqusrHf" }, "outputs": [], "source": [ "x_train = training_data.drop(['COSTENERGY'], axis = 1)\n", "y_train = training_data[['COSTENERGY']]\n", "y_train = y_train.to_numpy().ravel()\n", "\n", "x_test = testing_data.drop(['COSTENERGY'], axis = 1)\n", "y_test = testing_data[['COSTENERGY']]\n", "y_test = y_test.to_numpy().ravel()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nGoCjNqssrHg" }, "outputs": [], "source": [ "rf = RandomForestRegressor(n_estimators = 100, max_features = 'sqrt', max_depth = 3, random_state = 18).fit(x_train, y_train)" ] }, { "cell_type": "markdown", "metadata": { "id": "Plpna8dTsrHg" }, "source": [ "To get some insight in the random forest model, we can visualize the first tree of the model. [0] calls the first tree, but we can also plot any other tree of the model. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SkvnNsgqsrHg" }, "outputs": [], "source": [ "firsttree = rf.estimators_[0]\n", "plt.figure(figsize=(22,8))\n", "plot_tree(firsttree, feature_names=x_train.columns, filled=True, fontsize = 11)\n", "plt.title('title')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "KObpH9I7srHh" }, "source": [ "
\n", "Question 4: Interpret what you see in the decision tree. Hint: Explain the terms squared_error, samples and value in the tree.
" ] }, { "cell_type": "markdown", "metadata": { "id": "Mws06LWTsrHh" }, "source": [ "## 5. Prediction with the random forest model\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "7Ip4rq9wsrHh" }, "source": [ "In the next line of code, we use the trained random forest model to predict the dependent variable of the testing dataset. We use the function predict of the random forest regressor. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5E0QlyZgsrHi" }, "outputs": [], "source": [ "y_pred = rf.predict(x_test)" ] }, { "cell_type": "markdown", "metadata": { "id": "infAUvaAsrHi" }, "source": [ "## 6. Evaluate performance\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "k4Wxd2Q-srHi" }, "source": [ "Using the predicted dependent variable, we can evaluate the performance of the model. We first do this visually by plotting the dependent variable of the testing dataset together with the predicted values." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "iH3u1nR9srHj" }, "outputs": [], "source": [ "fst = 12\n", "plt.plot(y_test,'bo')\n", "plt.plot(y_pred,'ro')\n", "plt.legend(['Observed','Predicted'], fontsize = fst)\n", "plt.xticks(fontsize=fst)\n", "plt.yticks(fontsize=fst)\n", "plt.ylabel('ENERGYCOST',fontsize=fst)" ] }, { "cell_type": "markdown", "metadata": { "id": "JXe0KVeBsrHj" }, "source": [ "
\n", "Question 5: Interpret the plot you have created to evaluate the performance of your model.
" ] }, { "cell_type": "markdown", "metadata": { "id": "UC9ij4IXsrHj" }, "source": [ "Next we check performance in terms of a numeric criterion. We can for example check the mean squared error or the R2." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8qLB-9DAsrHj" }, "outputs": [], "source": [ "mse = mean_squared_error(y_test, y_pred)\n", "print(mse)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Dn-JB3hisrHk" }, "outputs": [], "source": [ "r2 = r2_score(y_test, y_pred)\n", "print(r2)" ] }, { "cell_type": "markdown", "metadata": { "id": "WwcU485nsrHk" }, "source": [ "
\n", "Question 6: What are the values you receive for the mean squared error and the r-squared? Also explain the two terms.
" ] }, { "cell_type": "markdown", "metadata": { "id": "27hUjpQCsrHk" }, "source": [ "## 7. Parameter tuning\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "Y-xji2g9srHk" }, "source": [ "The R2 score is pretty low, so we want to try other hyperparameters. We will change the depth and the number of trees in the forest. In the code below, we run a for loop that tries 3 different n_estimators and 3 different depths, so we estimate 9 different random forest models. We save the mse and r2 of each model in a list, i.e. list_mse and list_r2. It'll take a minute or two to run. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "VbNSA6U9srHl" }, "outputs": [], "source": [ "list_mse = []\n", "list_r2 = []\n", "\n", "for n_est in [75, 150]:\n", " for dep in [4, 7, 10]:\n", " rf = RandomForestRegressor(n_estimators = n_est, max_features = 'sqrt', max_depth = dep, random_state = 18).fit(x_train, y_train)\n", " y_pred = rf.predict(x_test)\n", " mse = mean_squared_error(y_test, y_pred)\n", " r2 = r2_score(y_test, y_pred)\n", " list_mse.append(mse)\n", " list_r2.append(r2)\n", "\n", "print(list_mse)\n", "print(list_r2)" ] }, { "cell_type": "markdown", "metadata": { "id": "W2CjF6uRsrHl" }, "source": [ "
\n", "Question 7: Interpret the results. Which model performs best. Think about how the for loop works. You could add print(n_est) and print(dep) in the for loop to make life easier.
" ] }, { "cell_type": "markdown", "metadata": { "id": "7k77zQLcsrHl" }, "source": [ "
\n", "Question 8: Try another parameter space but think about the time the model needs to run! The higher the number of trees and the depth, the longer it takes to train the model. You can copy the code from the previous cell and change the numbers over which the for loop iterates, i.e. the numbers in [75,150] and [5,7,9]. Because of time, I would advice not to try more than 4 models. Report the hypermeters you tried and the R2 score.
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "vagBxxQHsrHl" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "id": "_FGREwqbsrHm" }, "source": [ "## 8. Feature importance\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "dVUOoiOpsrHm" }, "source": [ "Next we want to see which variables have the most predictive value for **ENERGYCOSTS**. We can check the importance of each feature in the random forest model by calling rf.feature_importances_. We make a horizontal barplot to see the importance of each feature. You can fill in the hyperparameters for n_estimators and max_depth." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZnhE-P8HsrHm" }, "outputs": [], "source": [ "rf = RandomForestRegressor(n_estimators = ..., max_features = 'sqrt', max_depth = ..., random_state = 18).fit(x_train, y_train)\n", "sorted_idx = rf.feature_importances_.argsort() #we sort the importance of features from high to low.\n", "plt.figure(figsize=(7, 10.5))\n", "plt.barh(x_train.columns[sorted_idx], rf.feature_importances_[sorted_idx])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "06Am8RYtsrHn" }, "source": [ "
\n", "Question 9: According to the plot, which variables, also called features, are most important in the random forest? Should we leave out variables from the model. Hint: Not all variables in the model are meaningful, are they?
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nnq1lUzbsrHn" }, "outputs": [], "source": [] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" }, "vscode": { "interpreter": { "hash": "f323064ae63d54ed8d769390a968e914fbf7abacffc63e116cd2e04a08ed2d24" } } }, "nbformat": 4, "nbformat_minor": 4 }