{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "uWC4rptzkYkI" }, "source": [ "# Tutorial 1: Data Exploration and regression analysis\n", "\n", "In this tutorial you will learn how to read and explore census data from the [IPUMS USA](https://usa.ipums.org/usa/) database. IPUMS USA collects, preserves and harmonizes U.S. census microdata and provides easy access to this data with enhanced documentation. Data includes decennial censuses from 1790 to 2010 and American Community Surveys (ACS) from 2000 to the present.\n", "\n", "IPUMS provides census and survey data from around the world integrated across time and space. IPUMS integration and documentation makes it easy to study change, conduct comparative research, merge information across data types, and analyze individuals within family and community context.\n", "\n", "In this tutorial, you will learn how to read the IPUMPS USA data, explore and manipulate it (to prepare for analysis) and how to perform a simple correlation and regression analysis. We will use the IPUMS USA database throughout the next four tutorials.\n", "\n", "### Important before we start\n", "---\n", "Make sure that you save this file before you continue, else you will lose everything. To do so, go to **Bestand/File** and click on **Een kopie opslaan in Drive/Save a Copy on Drive**!\n", "\n", "Now, rename the file into Week2_Tutorial1.ipynb. You can do so by clicking on the name in the top of this screen." ] }, { "cell_type": "markdown", "metadata": { "id": "si8XPsJVkYkN" }, "source": [ "

Tutorial Outline

\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "z6YIFcOVkYkO" }, "source": [ "## Learning Objectives\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "bedYbW7PkYkP" }, "source": [ "- Work with Pandas DataFrames with real-world data.\n", "- Introducing basic functions to explore and understand the data.\n", "- Learn how to Make distribution plots of the data\n", "- Create a correlation map\n", "- Create new variables\n", "- Set up an OLS regression" ] }, { "cell_type": "markdown", "metadata": { "id": "s1CyoIv4kYkP" }, "source": [ "## 1. Introducing the packages\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "Cua_uDTHkYkQ" }, "source": [ "Within this tutorial, we are going to make use of the following packages:\n", "\n", "[**seaborn**](https://seaborn.pydata.org/index.html) is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.\n", "\n", "[**NumPy**](https://numpy.org/doc/stable/) is a Python library that provides a multidimensional array object, various derived objects, and an assortment of routines for fast operations on arrays.\n", "\n", "[**Pandas**](https://pandas.pydata.org/docs/) is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n", "\n", "[**statsmodels**](https://www.statsmodels.org/) is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration." ] }, { "cell_type": "markdown", "metadata": { "id": "9sLC9IB2kYkT" }, "source": [ "Now we will import these packages in the cell below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ks-i9aC0kYkT" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import statsmodels.api as sm\n", "\n", "sns.set_style(\"whitegrid\",{'axes.grid' : True})\n", "pd.options.mode.chained_assignment = None" ] }, { "cell_type": "markdown", "metadata": { "id": "-jcL-MqgkYkU" }, "source": [ "## 2. Reading the data and having a first look\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "K20OnXYFkYkV" }, "outputs": [], "source": [ "## Read the data using the pandas package\n", "data = pd.read_csv(r\"https://github.com/ElcoK/BigData_AED/raw/main/week2/usadataforweek2tut1.csv\")" ] }, { "cell_type": "markdown", "metadata": { "id": "uLNoPY0KkYkV" }, "source": [ "There are hundreds of variables included in the IPUMS USA database. In the next line of code we read in a file with a short description of the variables that we selected for this tutorial. The dataset contains socio-demographic variables such as age, gender, education, but also economic variables, such as income, mortgage payments, energy costs, etc. In these two weeks we are going to work towards two machine learning models that can predict energy costs per household." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "z6Ed5z6AkYkW", "outputId": "9858e000-4e5e-4cc9-f0b4-a74b90b2a5d2" }, "outputs": [], "source": [ "datadescription = pd.read_csv(r\"https://github.com/ElcoK/BigData_AED/raw/main/week2/shortdescription_ipumsusa_variables.csv\", sep = ';', encoding = 'unicode_escape')\n", "print(datadescription[['Variable_name','Short_description']])" ] }, { "cell_type": "markdown", "metadata": { "id": "5bLby1XrkYkW" }, "source": [ "Now let’s take a look at the dataset. A useful start is to explore the first rows through the `head()` function with Pandas." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 236 }, "id": "HK9eNsk_kYkX", "outputId": "016024d1-cc03-490b-ca5b-b2cebf3540bc" }, "outputs": [], "source": [ "data.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "aDxYNQfwkYkX" }, "source": [ "And let's have a look at the total size of this dataset. To do so, we can use the `shape()` function." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PrGXtSs3kYkY" }, "outputs": [], "source": [ "data.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "N0OEQbLskYkY" }, "source": [ "And some more information through the `describe()` function." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "g9kMFdAnkYkY" }, "outputs": [], "source": [ "data.describe()" ] }, { "cell_type": "markdown", "metadata": { "id": "0vXGOlUEkYkZ" }, "source": [ "
\n", "Question 1: What information do the functions .shape() and .describe() give you about the data?\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "kn4BwUG6kYkZ" }, "source": [ "Let's find out the data types of the data and if there are non-null counts." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2e-wS5q2kYkZ" }, "outputs": [], "source": [ "data.info()" ] }, { "cell_type": "markdown", "metadata": { "id": "dtC5Yt6akYka" }, "source": [ "Another way to check if there are missing values by using the following line of code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LXabn4ixkYka", "outputId": "f35e332f-07f8-486a-bace-44447f305a04" }, "outputs": [], "source": [ "data.isnull().sum().sort_values(ascending = False)" ] }, { "cell_type": "markdown", "metadata": { "id": "rELKVPGdkYka" }, "source": [ "
\n", "Question 2:
\n", " \n", "- Please describe what the following line of Python code does:\n", "
\n", " data.isnull().sum().sort_values(ascending = False)\n", "
\n", "Hint: Every dot (.) indicates a new function.\n", "
\n", "- Also describe what you see in the data. What does the output mean? Do you notice any surprises?\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "0yOner1XkYkb" }, "source": [ "## 3. Exploratory data analysis\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "-qKw69t7kYkb" }, "source": [ "Now we are going to take a look at some specific variables. We want to see how variables are distributed and how they are related to each other, and specifically how they are related to energy costs. We can do this in several ways." ] }, { "cell_type": "markdown", "metadata": { "id": "vIW9uRVkkYkb" }, "source": [ "Here we see the different values in the variable OWNERSHIP and how many times each value occurs. If you want to know what the values mean, you can search the variable on the ipums usa website." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "VjexcsBGkYkb", "outputId": "b52591f2-31a1-4a8c-c31f-8140714889a8" }, "outputs": [], "source": [ "data['OWNERSHP'].value_counts(dropna = False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "rQT7_acHkYkb", "outputId": "47ab5a18-0da0-4eb9-d60e-d112e5ed2e76" }, "outputs": [], "source": [ "data['HHINCOME'].value_counts(dropna = False)" ] }, { "cell_type": "markdown", "metadata": { "id": "JIuc6ddBkYkc" }, "source": [ "The function `value_counts()` is especially useful when the number of values in a variable is limited. For example, for income, which contains a lot of different values, value_counts is not so useful and a plot of the distribution would be more insightful. We can use the **seaborn** package to easily plot the distribution of the income. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 523 }, "id": "v5aDo2sqkYkc", "outputId": "a434e586-486b-403e-d0db-815719193fce" }, "outputs": [], "source": [ "%matplotlib inline\n", "sns.displot(data['HHINCOME'], bins=50,kde=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 523 }, "id": "CA3N-3D_kYkc", "outputId": "ce50a59e-42aa-4d60-e947-6180ce5e5047" }, "outputs": [], "source": [ "%matplotlib inline\n", "# Less bins\n", "sns.displot(data['HHINCOME'], bins=20,kde=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "pXYiwLDNkYkc" }, "source": [ "And we can also include the kernel density, by setting the `kde` parameter to **True**. Kernel Density Estimation (KDE) is a technique that let’s you create a smooth curve given a set of data. This can be useful if you want to visualize just the “shape” of some data, as a kind of continuous replacement for the discrete histogram." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 523 }, "id": "tMQEAQetkYkd", "outputId": "86bc96c4-f2d8-49c7-ef63-7b5d291983b5" }, "outputs": [], "source": [ "%matplotlib inline\n", "sns.displot(data['HHINCOME'], bins=50,kde=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "ofiJI98ekYkd" }, "source": [ "
\n", "Question 3: We just showed how to explore the OWNERSHP and HHINCOME variables. Choose a different variable and describe it. What are the units, and does it make sense to use value_counts() to understand this variable or would you rather use something different?\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0A7N1tM-kYkd" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "id": "0K30W0NTkYkd" }, "source": [ "Now we are going to take a look at the variable we want to predict in a later stage, energy costs." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 523 }, "id": "MGeyGthWkYke", "outputId": "373584ad-ee4d-491d-944f-5e5bb918cc28" }, "outputs": [], "source": [ "%matplotlib inline\n", "sns.displot(data['COSTENERGY'], bins=50,kde=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "dH7frrphkYkf" }, "source": [ "Next we want to see how different variables are related to each other. We are specifically interested in which variables are correlated to energy costs, as we are going to predict energy costs using a simple regression model in Step 6 of this tutorial. To identify potential predictors of energy costs, we can create a correlation matrix using the code below. You can add a few variables you are interested in yourself.\n", "As you can see, there are several options in the seaborn (imported as sns) heatmap. We have annot = True which makes sure that we include the numbers in the heatmap (try switching it off), fmt = .2f specifies that we want to have 2 decimals, with cmap we can specify the color palette and with center = 0, we scale the colors in such a way that a correlation of 0 corresponds to white color. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 607 }, "id": "5FD9ta4_kYkg", "outputId": "475e082e-9ba4-443b-a126-4d648fb4b564" }, "outputs": [], "source": [ "%matplotlib inline\n", "plt.figure(figsize=(8,6))\n", "sns.heatmap(data[[\"COSTENERGY\", \"ROOMS\", \"RENT\", \"INCTOT\", \"HHINCOME\", \"BUILTYR2\", \"VEHICLES\", \"BEDROOMS\", \"FARM\", \"OWNERSHP\", \"MORTGAGE\", \"AGE\", \"SEX\"]].corr(),\n", " annot=True, fmt = \".2f\", center=0, cmap = \"RdBu\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "yv55XfWGkYkg" }, "source": [ "
\n", " Question 4: Which variable is most correlated to COSTENERGY and which variable the least? \n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "cs781VNukYkg" }, "source": [ "## 4. Creating new variables\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "NwypAzjFkYkg" }, "source": [ "The dataset consists of households and individuals. Energy costs are a household variable, i.e. all members of the household have the same energy costs. Age and sex are individual variables, so the relationship between age or sex with energy costs is likely to be low. Think of a family with children, where a 3-year old daughter has the same energy costs as the 40-year old father. But maybe we can use these two variables, AGE and SEX in another way to make them more useful. We can for example create a variable with number of children in a household, or number of women, age of the oldest person in the household etc. To create these variables, we need to know which individuals belong to the same household and luckily this is exactly what the column SERIAL indicates, as SERIAL is the household ID. \n", "\n", "Before we go into the variables SEX and AGE, we are going to create a new variable household size based on the variable SERIAL." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Ow5FFfLxkYkg" }, "outputs": [], "source": [ "serialcount = data['SERIAL'].value_counts(dropna = False).reset_index()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CDrCdLWDkYkh" }, "outputs": [], "source": [ "print(serialcount)" ] }, { "cell_type": "markdown", "metadata": { "id": "5q9fLnlWkYkh" }, "source": [ "The names of the dataframe *serialcount* are a bit confusing, because the **index** column corresponds to the **SERIAL** number and the column **SERIAL** corresponds to the *household size*, so let's rename the **SERIAL** column to **HHSIZE** and the **index** column to **SERIAL**." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "wGCV5MM-kYkh" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "id": "-MSHB7T_kYkh" }, "source": [ "Then we add the column **HHSIZE** (household) size to our data using the `.merge()` function. Within the `merge()` function, you have to define on which column you want to merge the two dataframes using the `on` argument. Merge on the **SERIAL** column." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LvynYPTykYkh" }, "outputs": [], "source": [ "data = data.merge(serialcount, on = 'SERIAL', how = 'left')" ] }, { "cell_type": "markdown", "metadata": { "id": "hMyEZ2mjkYki" }, "source": [ "
\n", " Question 5: Please explain what the .merge() function does. Also explain the option 'on' and 'how'.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "p6TXXSpQkYki" }, "source": [ "Now we have merged a new variable HHSIZE to the dataframe. Next we will use the variable SEX to create a new variable. It is possible that women use more energy than men, or the other way around, so the number of women (or men) could be a useful variable for predicting energy costs. To obtain the household size, we simply counted the number of occurances of each serial number, but to obtain the number of women per household, we also need the column SEX. The variable SEX has two values, 1 represents a male and 2 represents a female." ] }, { "cell_type": "markdown", "metadata": { "id": "rFA3Lh2LkYki" }, "source": [ "We use the function `value_count()` to obtain the number of women and men per serial number, i.e., per unique household. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "UH67KE1TkYki" }, "outputs": [], "source": [ "gendercount = data[['SERIAL', 'SEX']].value_counts(dropna = False).reset_index()\n", "print(gendercount)" ] }, { "cell_type": "markdown", "metadata": { "id": "5qffPKNekYki" }, "source": [ "It is always good to realize that in many cases, multiple ways exist to obtain the same result. The following line of code produces the same results:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rCwZ8ItPkYkj" }, "outputs": [], "source": [ "gendercount = data[['SERIAL','SEX']].groupby(['SERIAL','SEX']).size().reset_index()\n", "print(gendercount)" ] }, { "cell_type": "markdown", "metadata": { "id": "wEnR34WykYkj" }, "source": [ "
\n", " Question 6: There are different ways to obtain the same result. Please describe the two lines of code above and explain how they take a different approach, but yet achieve the same outcome.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "nVjeWCw-kYkj" }, "source": [ "In the next steps you are going to write code that merges a new variable called **'NR_OF_WOMEN'** (number of women) to the existing dataframe. The following hints guide you through the steps.\n", "\n", "*Three lines of code is sufficient. Hint 1 is given for the first line of code, Hint 2 for the second line, and Hint 3a, 3b, 3c for the third line of code.*\n", "\n", "- **Hint 1**: You want to create a variable that indicates the number of women per household using the dataframe gendercount. The only thing you have to do is keep the number of the women in the dataframe gendercount, i.e., use SEX == 2. Store this dataframe as a new dataframe named womencount, hence your first line of code starts with womencount = . \n", "\n", "- **Hint 2**: Change the variable name of 0 to NR_OF_WOMEN in the dataframe womencount. 0 is not a string (so it's not '0'), but a number. \n", "\n", "- **Hint 3a**: In the third line of code you merge the dataframe womencount women to the dataframe data.\n", "- **Hint 3b**: Think about the column SEX in the womencount dataframe. It has no meaning anymore, since you only kept the women in the dataframe (all values in the column SEX are 2). Hence, it is not meaningful to merge the column SEX of the womencount dataframe to the data dataframe. Also, there is already a column SEX in the data dataframe, so when we merge another dataframe with the same column name, the column names are changed to SEX_x and SEX_y when we merge them. Hence, to avoid confusion you only want to include the columns NR_OF_WOMEN and SERIAL in the womencount dataframe when we merge it to data dataframe (Hint: you could for example use double square brackets). \n", "- **Hint 3c**: See the line of code where we merged serialcount to data. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "49QIROMMkYkj" }, "outputs": [], "source": [ "womencount =\n", "womencount =\n", "data = data.merge(" ] }, { "cell_type": "markdown", "metadata": { "id": "5GUTTDNgkYkk" }, "source": [ "When you succesfully merged the number of women the dataframe, what do you do with the nan values? Fill in the dots. The fillna function automatically replaces nan values with the number between the brackets." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "icrJJKKdkYkk" }, "outputs": [], "source": [ "data['NR_OF_WOMEN'] = data['NR_OF_WOMEN'].fillna(...)" ] }, { "cell_type": "markdown", "metadata": { "id": "w9a28h2NkYkk" }, "source": [ "
\n", " Question 7: Please provide the three lines of code you have written to succesfully create and merge the new variable NR_OF_WOMEN. Also describe how you solved/filled the NaN values.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "qfxEurYHkYkk" }, "source": [ "## 5. Nonlinear relationships\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "tB-ax0z4kYkk" }, "source": [ "We are going to create two new variables based on age. First we want to create a variable with the age of the oldest person." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fr-9BLVCkYkk" }, "outputs": [], "source": [ "oldest = data[['SERIAL', 'AGE']].groupby('SERIAL').max().reset_index()" ] }, { "cell_type": "markdown", "metadata": { "id": "K-o3cqeGkYkl" }, "source": [ "We want to change the name of the column **AGE** to **OLDEST**, because we already have a column **AGE** in the dataframe data. Then we merge it to the dataframe." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "mYWbZO8QkYkl" }, "outputs": [], "source": [ "oldest = oldest.rename(columns = {'AGE':'OLDEST'})\n", "data = data.merge(oldest, on = 'SERIAL', how = 'left')" ] }, { "cell_type": "markdown", "metadata": { "id": "waddrryFkYkl" }, "source": [ "Now create a variable yourself with the age of the youngest person, named **YOUNGEST**, and add it to the dataframe data. Follow the same approach as how you added the **OLDEST** column." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nxuzz2lOkYkl" }, "outputs": [], "source": [ "youngest = data[['SERIAL', 'AGE']].groupby(\n", "youngest = \n", "data = data.merge(" ] }, { "cell_type": "markdown", "metadata": { "id": "Ji7jYXUXkYkl" }, "source": [ "
\n", " Question 8: Please provide the three lines of code you have written to succesfully create and merge the new variable YOUNGEST.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "kTeLK-a4kYkl" }, "source": [ "Let's check if the correlation of the oldest and youngest person with energy costs is higher than the correlation of age with energy costs." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 452 }, "id": "sf-oPrCSkYkm", "outputId": "68470410-60f8-413b-84b5-a30b5471ef5e" }, "outputs": [], "source": [ "%matplotlib inline\n", "sns.heatmap(data[[\"COSTENERGY\",\"YOUNGEST\", \"OLDEST\", \"AGE\"]].corr(),\n", " annot=True, fmt = \".2f\", center=0, cmap = \"RdBu\")" ] }, { "cell_type": "markdown", "metadata": { "id": "Wwm6AqNnkYkm" }, "source": [ "A possible explanation for the low correlation could be that the relationship between age and energy costs is not linear. Let's see the average energy costs over the age of the oldest person in the household in a barplot." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 559 }, "id": "0H75bHnUkYkm", "outputId": "9906e923-fb72-4d2e-be06-83bf796dd8ce" }, "outputs": [], "source": [ "%matplotlib inline\n", "plt.figure()\n", "sns.barplot(x = 'OLDEST', y = 'COSTENERGY', data = data, ci = None)\n", "plt.xticks([15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], #position of xticks.\n", " [15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95]) #values of xticks.\n", "plt.xlim([15,95])\n", "plt.title('Oldest vs Cost of Energy')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "Selw6ilEkYkm" }, "source": [ "In the graph we see that especially households where the oldest member is young, energy costs are relatively low. Then energy costs increase until the age of 55 (maybe these are families with children) and then stabilizes." ] }, { "cell_type": "markdown", "metadata": { "id": "qRD_wrUjkYkn" }, "source": [ "## 6. Regression analysis\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "_vUMR2b4kYkn" }, "source": [ "In the last part we are going to estimate a first model to predict energy costs of households. We will use a simple Ordinary Least Squares regression model. To make life a bit easier, download the dataset for the OLS regression in the following line of code (it includes the new variables you created and it deals with missing values). We use the package statsmodels to estimate the OLS regression. We could also use the sklearn library, but the statsmodels package has the advantage that it includes standard errors of the coefficient estimates. This means that we can say something about the significance of the explanantory variables and the causality between the explanatory variables and the dependent variable, something which is not possible with a machine learning model. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LnHE5-_6kYkn" }, "outputs": [], "source": [ "dataOLS = pd.read_csv(r\"https://github.com/ElcoK/BigData_AED/raw/main/week2/usadataforOLS.csv\", sep = ',', encoding = 'unicode_escape')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BBwRaqJ_kYkn" }, "outputs": [], "source": [ "Y = dataOLS[['COSTENERGY']]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4hRMFUDvkYkn" }, "outputs": [], "source": [ "X = dataOLS[[\"ROOMS\", \"HHINCOME\", \"BUILTYR2\", \"VEHICLES\", \"FARM\", \"OWNERSHP\", \"YOUNGEST\", \"OLDEST\", \"HHSIZE\", \"NR_OF_WOMEN\"]]\n", "X['Constant'] = 1" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fOrBx0RXkYko" }, "outputs": [], "source": [ "regressionOLS = sm.OLS(Y, X)\n", "resultsOLS = regressionOLS.fit()\n", "print(resultsOLS.summary())" ] }, { "cell_type": "markdown", "metadata": { "id": "ksT_I6z7kYko" }, "source": [ "
\n", " Question 9: Interpret the results of the OLS regression. You can discuss the following questions in your answer. 1) Do you think that the coefficient estimates are plausible? You don't need to explain all coefficient estimates, but highlight a few. 2) Would you include other variables in the regression? Or do you want to exclude variables? Explain which ones. Make another correlation heatmap to support your explanation.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "8maQNAl8kYko" }, "source": [ "Interpret the results of the OLS regression. Make another correlation heatmap to support your explanation." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6lTpFlT9kYko" }, "outputs": [], "source": [ "# code for correlation heatmap." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Y5rNiv7akYkp" }, "outputs": [], "source": [] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" }, "vscode": { "interpreter": { "hash": "f323064ae63d54ed8d769390a968e914fbf7abacffc63e116cd2e04a08ed2d24" } } }, "nbformat": 4, "nbformat_minor": 4 }