PYSPARK - LINER REGRESSION.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "96816ed7-b08a-4ca3-abb9-f99880c3535d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook will show you how to create and query a table or DataFrame that you uploaded to DBFS. [DBFS](https://docs.databricks.com/user-guide/dbfs-databricks-file-system.html) is a Databricks File System that allows you to store data for querying inside of Databricks. This notebook assumes that you have a file already inside of DBFS that you would like to read from.\n",
    "\n",
    "This notebook is written in **Python** so the default cell type is Python. However, you can use different languages by using the `%LANGUAGE` syntax. Python, Scala, SQL, and R are all supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "6482be4c-f067-47c9-b0ac-35c938b94601",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+------+------+---+------+----+\n|total_bill| tip|   sex|smoker|day|  time|size|\n+----------+----+------+------+---+------+----+\n|     16.99|1.01|Female|    No|Sun|Dinner|   2|\n|     10.34|1.66|  Male|    No|Sun|Dinner|   3|\n|     21.01| 3.5|  Male|    No|Sun|Dinner|   3|\n|     23.68|3.31|  Male|    No|Sun|Dinner|   2|\n|     24.59|3.61|Female|    No|Sun|Dinner|   4|\n|     25.29|4.71|  Male|    No|Sun|Dinner|   4|\n|      8.77| 2.0|  Male|    No|Sun|Dinner|   2|\n|     26.88|3.12|  Male|    No|Sun|Dinner|   4|\n|     15.04|1.96|  Male|    No|Sun|Dinner|   2|\n|     14.78|3.23|  Male|    No|Sun|Dinner|   2|\n|     10.27|1.71|  Male|    No|Sun|Dinner|   2|\n|     35.26| 5.0|Female|    No|Sun|Dinner|   4|\n|     15.42|1.57|  Male|    No|Sun|Dinner|   2|\n|     18.43| 3.0|  Male|    No|Sun|Dinner|   4|\n|     14.83|3.02|Female|    No|Sun|Dinner|   2|\n|     21.58|3.92|  Male|    No|Sun|Dinner|   2|\n|     10.33|1.67|Female|    No|Sun|Dinner|   3|\n|     16.29|3.71|  Male|    No|Sun|Dinner|   3|\n|     16.97| 3.5|Female|    No|Sun|Dinner|   3|\n|     20.65|3.35|  Male|    No|Sat|Dinner|   3|\n+----------+----+------+------+---+------+----+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "# File location and type\n",
    "file_location = \"/FileStore/shared_uploads/astartupcto@gmail.com/tips.csv\"\n",
    "file_type = \"csv\"\n",
    "\n",
    "# The applied options are for CSV files. For other file types, these will be ignored.\n",
    "df =spark.read.csv(file_location,header=True,inferSchema=True)\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5e5b80f2-3426-44e1-b86e-171314f4827e",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n |-- total_bill: double (nullable = true)\n |-- tip: double (nullable = true)\n |-- sex: string (nullable = true)\n |-- smoker: string (nullable = true)\n |-- day: string (nullable = true)\n |-- time: string (nullable = true)\n |-- size: integer (nullable = true)\n\n"
     ]
    }
   ],
   "source": [
    "#show schema of dataframe\n",
    "df.printSchema()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0432b71c-b266-417d-b0d5-1c17afa0f090",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[27]: ['total_bill', 'tip', 'sex', 'smoker', 'day', 'time', 'size']"
     ]
    }
   ],
   "source": [
    "#show columns of dataframe\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0ae62ac1-81a6-4b1d-92b9-f85ec9cc93ff",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "### Handling Categorical Features\n",
    "from pyspark.ml.feature import StringIndexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "faa6f9b0-6f8b-4dbd-a5a2-dc074181f2e3",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+------+------+---+------+----+\n|total_bill| tip|   sex|smoker|day|  time|size|\n+----------+----+------+------+---+------+----+\n|     16.99|1.01|Female|    No|Sun|Dinner|   2|\n|     10.34|1.66|  Male|    No|Sun|Dinner|   3|\n|     21.01| 3.5|  Male|    No|Sun|Dinner|   3|\n|     23.68|3.31|  Male|    No|Sun|Dinner|   2|\n|     24.59|3.61|Female|    No|Sun|Dinner|   4|\n|     25.29|4.71|  Male|    No|Sun|Dinner|   4|\n|      8.77| 2.0|  Male|    No|Sun|Dinner|   2|\n|     26.88|3.12|  Male|    No|Sun|Dinner|   4|\n|     15.04|1.96|  Male|    No|Sun|Dinner|   2|\n|     14.78|3.23|  Male|    No|Sun|Dinner|   2|\n|     10.27|1.71|  Male|    No|Sun|Dinner|   2|\n|     35.26| 5.0|Female|    No|Sun|Dinner|   4|\n|     15.42|1.57|  Male|    No|Sun|Dinner|   2|\n|     18.43| 3.0|  Male|    No|Sun|Dinner|   4|\n|     14.83|3.02|Female|    No|Sun|Dinner|   2|\n|     21.58|3.92|  Male|    No|Sun|Dinner|   2|\n|     10.33|1.67|Female|    No|Sun|Dinner|   3|\n|     16.29|3.71|  Male|    No|Sun|Dinner|   3|\n|     16.97| 3.5|Female|    No|Sun|Dinner|   3|\n|     20.65|3.35|  Male|    No|Sat|Dinner|   3|\n+----------+----+------+------+---+------+----+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2ee7ab64-9804-4afb-852c-ee02eb5d3a20",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+------+------+---+------+----+-----------+\n|total_bill| tip|   sex|smoker|day|  time|size|sex_indexed|\n+----------+----+------+------+---+------+----+-----------+\n|     16.99|1.01|Female|    No|Sun|Dinner|   2|        1.0|\n|     10.34|1.66|  Male|    No|Sun|Dinner|   3|        0.0|\n|     21.01| 3.5|  Male|    No|Sun|Dinner|   3|        0.0|\n|     23.68|3.31|  Male|    No|Sun|Dinner|   2|        0.0|\n|     24.59|3.61|Female|    No|Sun|Dinner|   4|        1.0|\n|     25.29|4.71|  Male|    No|Sun|Dinner|   4|        0.0|\n|      8.77| 2.0|  Male|    No|Sun|Dinner|   2|        0.0|\n|     26.88|3.12|  Male|    No|Sun|Dinner|   4|        0.0|\n|     15.04|1.96|  Male|    No|Sun|Dinner|   2|        0.0|\n|     14.78|3.23|  Male|    No|Sun|Dinner|   2|        0.0|\n|     10.27|1.71|  Male|    No|Sun|Dinner|   2|        0.0|\n|     35.26| 5.0|Female|    No|Sun|Dinner|   4|        1.0|\n|     15.42|1.57|  Male|    No|Sun|Dinner|   2|        0.0|\n|     18.43| 3.0|  Male|    No|Sun|Dinner|   4|        0.0|\n|     14.83|3.02|Female|    No|Sun|Dinner|   2|        1.0|\n|     21.58|3.92|  Male|    No|Sun|Dinner|   2|        0.0|\n|     10.33|1.67|Female|    No|Sun|Dinner|   3|        1.0|\n|     16.29|3.71|  Male|    No|Sun|Dinner|   3|        0.0|\n|     16.97| 3.5|Female|    No|Sun|Dinner|   3|        1.0|\n|     20.65|3.35|  Male|    No|Sat|Dinner|   3|        0.0|\n+----------+----+------+------+---+------+----+-----------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "#The purpose of this code is to convert the categorical string column \"sex\" into a numerical column \"sex_indexed\". \n",
    "# This is done to prepare the data for machine learning models that require numerical input.\n",
    "indexer=StringIndexer(inputCol=\"sex\",outputCol=\"sex_indexed\")\n",
    "df_r=indexer.fit(df).transform(df)\n",
    "df_r.show()\n",
    "\n",
    "#Components of the Code\n",
    "#StringIndexer:\n",
    "\n",
    "#StringIndexer is a feature transformer in PySpark that encodes a string column of labels to a column of label indices.\n",
    "#This is particularly useful for converting categorical string columns into numerical indices, which are required for most machine learning algorithms.\n",
    "#Parameters:\n",
    "\n",
    "#inputCol=\"sex\": Specifies the input column that contains the string labels. In this case, the input column is \"sex\".\n",
    "#outputCol=\"sex_indexed\": Specifies the name of the output column that will contain the numerical indices corresponding to the string labels. In this case, the output column is \"sex_indexed\".\n",
    "#fit and transform:\n",
    "\n",
    "#indexer.fit(df): The fit method is used to compute the mapping from string labels to numerical indices based on the input DataFrame df.\n",
    "#indexer.transform(df): The transform method is used to apply the computed mapping to the DataFrame df, resulting in a new DataFrame df_r with the additional column \"sex_indexed\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "6b95d734-4c80-4762-bd9b-92b6a107dced",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+\n|total_bill| tip|   sex|smoker|day|  time|size|sex_indexed|smoker_indexed|day_indexed|time_index|\n+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+\n|     16.99|1.01|Female|    No|Sun|Dinner|   2|        1.0|           0.0|        1.0|       0.0|\n|     10.34|1.66|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|\n|     21.01| 3.5|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|\n|     23.68|3.31|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     24.59|3.61|Female|    No|Sun|Dinner|   4|        1.0|           0.0|        1.0|       0.0|\n|     25.29|4.71|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|\n|      8.77| 2.0|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     26.88|3.12|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|\n|     15.04|1.96|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     14.78|3.23|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     10.27|1.71|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     35.26| 5.0|Female|    No|Sun|Dinner|   4|        1.0|           0.0|        1.0|       0.0|\n|     15.42|1.57|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     18.43| 3.0|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|\n|     14.83|3.02|Female|    No|Sun|Dinner|   2|        1.0|           0.0|        1.0|       0.0|\n|     21.58|3.92|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|\n|     10.33|1.67|Female|    No|Sun|Dinner|   3|        1.0|           0.0|        1.0|       0.0|\n|     16.29|3.71|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|\n|     16.97| 3.5|Female|    No|Sun|Dinner|   3|        1.0|           0.0|        1.0|       0.0|\n|     20.65|3.35|  Male|    No|Sat|Dinner|   3|        0.0|           0.0|        0.0|       0.0|\n+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.feature import StringIndexer\n",
    "# Create a StringIndexer instance that will convert categorical columns into numerical indices\n",
    "indexer = StringIndexer(\n",
    "    inputCols=[\"smoker\", \"day\", \"time\"],        # Specify the input columns to be indexed\n",
    "    outputCols=[\"smoker_indexed\", \"day_indexed\", \"time_index\"]  # Specify the output columns to store the indices\n",
    ")\n",
    "\n",
    "# Fit the StringIndexer on the DataFrame and transform the DataFrame to add the indexed columns\n",
    "df_r = indexer.fit(df_r).transform(df_r)\n",
    "\n",
    "# Show the transformed DataFrame with the new indexed columns\n",
    "df_r.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a9909b0b-caee-4838-b477-47c3701dbfd4",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[32]: ['total_bill',\n 'tip',\n 'sex',\n 'smoker',\n 'day',\n 'time',\n 'size',\n 'sex_indexed',\n 'smoker_indexed',\n 'day_indexed',\n 'time_index']"
     ]
    }
   ],
   "source": [
    "df_r.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "61d875e5-71fa-4dc4-ae90-54924b00a632",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+--------------------+\n|total_bill| tip|   sex|smoker|day|  time|size|sex_indexed|smoker_indexed|day_indexed|time_index|Independent Features|\n+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+--------------------+\n|     16.99|1.01|Female|    No|Sun|Dinner|   2|        1.0|           0.0|        1.0|       0.0|[1.01,2.0,1.0,0.0...|\n|     10.34|1.66|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|[1.66,3.0,0.0,0.0...|\n|     21.01| 3.5|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|[3.5,3.0,0.0,0.0,...|\n|     23.68|3.31|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[3.31,2.0,0.0,0.0...|\n|     24.59|3.61|Female|    No|Sun|Dinner|   4|        1.0|           0.0|        1.0|       0.0|[3.61,4.0,1.0,0.0...|\n|     25.29|4.71|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|[4.71,4.0,0.0,0.0...|\n|      8.77| 2.0|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[2.0,2.0,0.0,0.0,...|\n|     26.88|3.12|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|[3.12,4.0,0.0,0.0...|\n|     15.04|1.96|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[1.96,2.0,0.0,0.0...|\n|     14.78|3.23|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[3.23,2.0,0.0,0.0...|\n|     10.27|1.71|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[1.71,2.0,0.0,0.0...|\n|     35.26| 5.0|Female|    No|Sun|Dinner|   4|        1.0|           0.0|        1.0|       0.0|[5.0,4.0,1.0,0.0,...|\n|     15.42|1.57|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[1.57,2.0,0.0,0.0...|\n|     18.43| 3.0|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|[3.0,4.0,0.0,0.0,...|\n|     14.83|3.02|Female|    No|Sun|Dinner|   2|        1.0|           0.0|        1.0|       0.0|[3.02,2.0,1.0,0.0...|\n|     21.58|3.92|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[3.92,2.0,0.0,0.0...|\n|     10.33|1.67|Female|    No|Sun|Dinner|   3|        1.0|           0.0|        1.0|       0.0|[1.67,3.0,1.0,0.0...|\n|     16.29|3.71|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|[3.71,3.0,0.0,0.0...|\n|     16.97| 3.5|Female|    No|Sun|Dinner|   3|        1.0|           0.0|        1.0|       0.0|[3.5,3.0,1.0,0.0,...|\n|     20.65|3.35|  Male|    No|Sat|Dinner|   3|        0.0|           0.0|        0.0|       0.0|(6,[0,1],[3.35,3.0])|\n+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+--------------------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "#The purpose of using VectorAssembler in this code is to combine multiple individual feature columns into a single vector column. This is an essential step in preparing data for machine learning algorithms, which typically require input features to be in vector form.\n",
    "\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "# Create an instance of the VectorAssembler\n",
    "featureassembler = VectorAssembler(\n",
    "    inputCols=['tip', 'size', 'sex_indexed', 'smoker_indexed', 'day_indexed', 'time_index'],  # List of input columns to combine\n",
    "    outputCol=\"Independent Features\"  # Name of the output column that will contain the combined feature vector\n",
    ")\n",
    "\n",
    "# Transform the DataFrame to add the new feature vector column\n",
    "output = featureassembler.transform(df_r)\n",
    "\n",
    "# Show the transformed DataFrame with the new feature vector column\n",
    "output.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d33d1178-95a2-468f-a94a-e0eebc67be86",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n|Independent Features|\n+--------------------+\n|[1.01,2.0,1.0,0.0...|\n|[1.66,3.0,0.0,0.0...|\n|[3.5,3.0,0.0,0.0,...|\n|[3.31,2.0,0.0,0.0...|\n|[3.61,4.0,1.0,0.0...|\n|[4.71,4.0,0.0,0.0...|\n|[2.0,2.0,0.0,0.0,...|\n|[3.12,4.0,0.0,0.0...|\n|[1.96,2.0,0.0,0.0...|\n|[3.23,2.0,0.0,0.0...|\n|[1.71,2.0,0.0,0.0...|\n|[5.0,4.0,1.0,0.0,...|\n|[1.57,2.0,0.0,0.0...|\n|[3.0,4.0,0.0,0.0,...|\n|[3.02,2.0,1.0,0.0...|\n|[3.92,2.0,0.0,0.0...|\n|[1.67,3.0,1.0,0.0...|\n|[3.71,3.0,0.0,0.0...|\n|[3.5,3.0,1.0,0.0,...|\n|(6,[0,1],[3.35,3.0])|\n+--------------------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "#The purpose of this code is to verify and visualize the combined feature vectors created by the VectorAssembler. \n",
    "# It helps in understanding how the individual feature columns have been combined into a single vector column.\n",
    "output.select('Independent Features').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f2646b66-7710-4297-a6e1-156a37e6582d",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+--------------------+\n|total_bill| tip|   sex|smoker|day|  time|size|sex_indexed|smoker_indexed|day_indexed|time_index|Independent Features|\n+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+--------------------+\n|     16.99|1.01|Female|    No|Sun|Dinner|   2|        1.0|           0.0|        1.0|       0.0|[1.01,2.0,1.0,0.0...|\n|     10.34|1.66|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|[1.66,3.0,0.0,0.0...|\n|     21.01| 3.5|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|[3.5,3.0,0.0,0.0,...|\n|     23.68|3.31|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[3.31,2.0,0.0,0.0...|\n|     24.59|3.61|Female|    No|Sun|Dinner|   4|        1.0|           0.0|        1.0|       0.0|[3.61,4.0,1.0,0.0...|\n|     25.29|4.71|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|[4.71,4.0,0.0,0.0...|\n|      8.77| 2.0|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[2.0,2.0,0.0,0.0,...|\n|     26.88|3.12|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|[3.12,4.0,0.0,0.0...|\n|     15.04|1.96|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[1.96,2.0,0.0,0.0...|\n|     14.78|3.23|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[3.23,2.0,0.0,0.0...|\n|     10.27|1.71|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[1.71,2.0,0.0,0.0...|\n|     35.26| 5.0|Female|    No|Sun|Dinner|   4|        1.0|           0.0|        1.0|       0.0|[5.0,4.0,1.0,0.0,...|\n|     15.42|1.57|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[1.57,2.0,0.0,0.0...|\n|     18.43| 3.0|  Male|    No|Sun|Dinner|   4|        0.0|           0.0|        1.0|       0.0|[3.0,4.0,0.0,0.0,...|\n|     14.83|3.02|Female|    No|Sun|Dinner|   2|        1.0|           0.0|        1.0|       0.0|[3.02,2.0,1.0,0.0...|\n|     21.58|3.92|  Male|    No|Sun|Dinner|   2|        0.0|           0.0|        1.0|       0.0|[3.92,2.0,0.0,0.0...|\n|     10.33|1.67|Female|    No|Sun|Dinner|   3|        1.0|           0.0|        1.0|       0.0|[1.67,3.0,1.0,0.0...|\n|     16.29|3.71|  Male|    No|Sun|Dinner|   3|        0.0|           0.0|        1.0|       0.0|[3.71,3.0,0.0,0.0...|\n|     16.97| 3.5|Female|    No|Sun|Dinner|   3|        1.0|           0.0|        1.0|       0.0|[3.5,3.0,1.0,0.0,...|\n|     20.65|3.35|  Male|    No|Sat|Dinner|   3|        0.0|           0.0|        0.0|       0.0|(6,[0,1],[3.35,3.0])|\n+----------+----+------+------+---+------+----+-----------+--------------+-----------+----------+--------------------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "output.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d1c1fa4c-c78a-441a-bed9-3bcfcc5af966",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "finalized_data=output.select(\"Independent Features\",\"total_bill\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3d14fe7b-bc59-4376-8139-142283af09b0",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----------+\n|Independent Features|total_bill|\n+--------------------+----------+\n|[1.01,2.0,1.0,0.0...|     16.99|\n|[1.66,3.0,0.0,0.0...|     10.34|\n|[3.5,3.0,0.0,0.0,...|     21.01|\n|[3.31,2.0,0.0,0.0...|     23.68|\n|[3.61,4.0,1.0,0.0...|     24.59|\n|[4.71,4.0,0.0,0.0...|     25.29|\n|[2.0,2.0,0.0,0.0,...|      8.77|\n|[3.12,4.0,0.0,0.0...|     26.88|\n|[1.96,2.0,0.0,0.0...|     15.04|\n|[3.23,2.0,0.0,0.0...|     14.78|\n|[1.71,2.0,0.0,0.0...|     10.27|\n|[5.0,4.0,1.0,0.0,...|     35.26|\n|[1.57,2.0,0.0,0.0...|     15.42|\n|[3.0,4.0,0.0,0.0,...|     18.43|\n|[3.02,2.0,1.0,0.0...|     14.83|\n|[3.92,2.0,0.0,0.0...|     21.58|\n|[1.67,3.0,1.0,0.0...|     10.33|\n|[3.71,3.0,0.0,0.0...|     16.29|\n|[3.5,3.0,1.0,0.0,...|     16.97|\n|(6,[0,1],[3.35,3.0])|     20.65|\n+--------------------+----------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "finalized_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "dbe03a38-e728-40f9-8a53-0b7968b8dc87",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "#This code is to train a linear regression model on the training data, where the independent features are combined into a vector column, and the target variable is total_bill. The trained model can then be used to make predictions on new data.\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "\n",
    "# Train-test split\n",
    "train_data, test_data = finalized_data.randomSplit([0.75, 0.25])\n",
    "\n",
    "# Initialize the LinearRegression model\n",
    "regressor = LinearRegression(featuresCol='Independent Features', labelCol='total_bill')\n",
    "\n",
    "# Fit the LinearRegression model on the training data\n",
    "regressor = regressor.fit(train_data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0fdc835a-96fb-4ab3-89be-6cfbc57c7ac6",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[39]: DenseVector([2.7985, 3.4757, -1.0544, 1.8106, -0.3856, -1.1182])"
     ]
    }
   ],
   "source": [
    "regressor.coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "fd1642d4-bb73-4fc0-a410-ada61d0f3410",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[40]: 2.6572337983204286"
     ]
    }
   ],
   "source": [
    "regressor.intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2e45a3d8-af1c-408b-b64f-fe466c3401bd",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "### Predictions\n",
    "pred_results=regressor.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "01d128d2-1a71-44d0-a14c-b0377693547b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----------+------------------+\n|Independent Features|total_bill|        prediction|\n+--------------------+----------+------------------+\n|(6,[0,1],[1.75,2.0])|     17.82| 14.50592690983163|\n| (6,[0,1],[2.0,2.0])|     12.69| 15.20554891453792|\n|(6,[0,1],[2.24,3.0])|     16.04|19.352855578339547|\n| (6,[0,1],[2.5,4.0])|     18.35| 23.55613200251767|\n|(6,[0,1],[2.72,2.0])|     13.28| 17.22046028809204|\n|(6,[0,1],[3.35,3.0])|     20.65|22.459177279235472|\n| (6,[0,1],[4.3,2.0])|      21.7| 21.64207135783579|\n|(6,[0,1],[6.73,4.0])|     48.27|  35.3937363221481|\n|[1.0,1.0,1.0,0.0,...|      7.25| 7.876964304410876|\n|[1.0,1.0,1.0,1.0,...|      3.07| 9.687578213722702|\n|[1.1,2.0,1.0,1.0,...|      12.9|13.443096554888802|\n|[1.44,2.0,0.0,0.0...|      7.56|11.748909539928615|\n|[1.48,2.0,0.0,0.0...|      8.52|11.860849060681623|\n|[1.5,2.0,0.0,0.0,...|     19.08|11.916818821058126|\n|[1.5,2.0,0.0,1.0,...|     12.03|14.459990630161906|\n|[1.5,2.0,1.0,0.0,...|     26.41|12.751877853107041|\n|[1.5,2.0,1.0,0.0,...|     10.65|10.862391769039828|\n|[1.71,2.0,0.0,0.0...|     10.27| 14.00834466098687|\n|[1.76,2.0,0.0,1.0...|     11.24|16.344525699331705|\n|[1.83,1.0,1.0,0.0...|     10.07| 8.310223275968548|\n+--------------------+----------+------------------+\nonly showing top 20 rows\n\n"
     ]
    }
   ],
   "source": [
    "## Final comparison\n",
    "pred_results.predictions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "4ca98d3f-a98b-449e-9a3f-85ad7a9c7d72",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "\n",
    "### Explanation of the Columns\n",
    "\n",
    "1. **Independent Features**:\n",
    "    - This column contains the feature vectors that were used as inputs to the linear regression model. Each vector combines multiple individual features into a single vector.\n",
    "    - The vectors are displayed in sparse format. For example, `(6,[0,1],[1.75,2.0])` means:\n",
    "        - The vector has 6 elements.\n",
    "        - The non-zero elements are at indices 0 and 1.\n",
    "        - The values of the non-zero elements are 1.75 and 2.0 respectively.\n",
    "\n",
    "2. **total_bill**:\n",
    "    - This column contains the actual values of the target variable (the true labels) from the test dataset. In this context, `total_bill` represents the actual bill amount.\n",
    "\n",
    "3. **prediction**:\n",
    "    - This column contains the predicted values of the target variable, calculated by the linear regression model based on the feature vectors in the `Independent Features` column. These predictions are the model's estimate of the `total_bill`.\n",
    "\n",
    "### Example Row Explanation\n",
    "\n",
    "Let's break down an example row to understand what each part means:\n",
    "\n",
    "```\n",
    "|(6,[0,1],[1.75,2.0])| 17.82 | 14.50592690983163 |\n",
    "```\n",
    "\n",
    "- **Independent Features**: `(6,[0,1],[1.75,2.0])`\n",
    "  - This means the feature vector has 6 elements, but only the first two elements (indices 0 and 1) are non-zero.\n",
    "  - The value at index 0 is 1.75 and the value at index 1 is 2.0.\n",
    "\n",
    "- **total_bill**: `17.82`\n",
    "  - This is the actual bill amount for this particular observation.\n",
    "\n",
    "- **prediction**: `14.50592690983163`\n",
    "  - This is the predicted bill amount made by the linear regression model based on the feature vector `(6,[0,1],[1.75,2.0])`.\n",
    "\n",
    "### Detailed Analysis of the Output\n",
    "\n",
    "Let's go through several rows to understand the predictions:\n",
    "\n",
    "| Independent Features     | total_bill | prediction         |\n",
    "|--------------------------|------------|--------------------|\n",
    "| (6,[0,1],[1.75,2.0])     | 17.82      | 14.50592690983163  |\n",
    "| (6,[0,1],[2.0,2.0])      | 12.69      | 15.20554891453792  |\n",
    "| (6,[0,1],[2.24,3.0])     | 16.04      | 19.352855578339547 |\n",
    "| (6,[0,1],[2.5,4.0])      | 18.35      | 23.55613200251767  |\n",
    "| (6,[0,1],[2.72,2.0])     | 13.28      | 17.22046028809204  |\n",
    "| (6,[0,1],[3.35,3.0])     | 20.65      | 22.459177279235472 |\n",
    "| (6,[0,1],[4.3,2.0])      | 21.7       | 21.64207135783579  |\n",
    "| (6,[0,1],[6.73,4.0])     | 48.27      | 35.3937363221481   |\n",
    "| [1.0,1.0,1.0,0.0,...]    | 7.25       | 7.876964304410876  |\n",
    "| [1.0,1.0,1.0,1.0,...]    | 3.07       | 9.687578213722702  |\n",
    "| [1.1,2.0,1.0,1.0,...]    | 12.9       | 13.443096554888802 |\n",
    "| [1.44,2.0,0.0,0.0,...]   | 7.56       | 11.748909539928615 |\n",
    "| [1.48,2.0,0.0,0.0,...]   | 8.52       | 11.860849060681623 |\n",
    "| [1.5,2.0,0.0,0.0,...]    | 19.08      | 11.916818821058126 |\n",
    "| [1.5,2.0,0.0,1.0,...]    | 12.03      | 14.459990630161906 |\n",
    "| [1.5,2.0,1.0,0.0,...]    | 26.41      | 12.751877853107041 |\n",
    "| [1.5,2.0,1.0,0.0,...]    | 10.65      | 10.862391769039828 |\n",
    "| [1.71,2.0,0.0,0.0,...]   | 10.27      | 14.00834466098687  |\n",
    "\n",
    "### Interpretation\n",
    "\n",
    "1. **Feature Vector Interpretation**:\n",
    "    - For each row, the `Independent Features` column shows a vector representation of the input features used for the prediction.\n",
    "    - For example, `(6,[0,1],[1.75,2.0])` means a 6-element vector with values 1.75 at index 0 and 2.0 at index 1.\n",
    "\n",
    "2. **Actual vs. Predicted Values**:\n",
    "    - The `total_bill` column shows the actual bill amount, while the `prediction` column shows the bill amount predicted by the model.\n",
    "    - By comparing these two columns, you can assess how well the model is performing. For instance, in the first row, the actual bill is 17.82, and the predicted bill is 14.51, which shows a small difference.\n",
    "\n",
    "3. **Model Accuracy**:\n",
    "    - The closer the `prediction` values are to the `total_bill` values, the more accurate the model is.\n",
    "    - Large discrepancies between these values indicate that the model may need improvement, either through better feature engineering, more data, or a different model.\n",
    "\n",
    "### Summary\n",
    "\n",
    "The `pred_results.predictions.show()` output provides a comparison between the actual and predicted values for the target variable (`total_bill`). It allows you to assess the performance of the linear regression model by showing how well the model's predictions match the actual values. Each row in the output shows the feature vector used for the prediction, the actual bill amount, and the predicted bill amount, helping you understand the model's accuracy and potential areas for improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "75e3e5b1-0bb4-4dbe-a1ca-08e5a31ee173",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[43]: (0.6715899791172986, 4.243246528600479, 33.83498887434812)"
     ]
    }
   ],
   "source": [
    "### PErformance Metrics\n",
    "pred_results.r2,pred_results.meanAbsoluteError,pred_results.meanSquaredError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "8492a128-58f6-465d-a932-45047a857404",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Performance Metrics Explanation\n",
    "\n",
    "The code `pred_results.r2, pred_results.meanAbsoluteError, pred_results.meanSquaredError` provides important performance metrics for evaluating the regression model. Here is a detailed explanation of each metric:\n",
    "\n",
    "1. **R² (R-Squared)**:\n",
    "   - **Value**: `0.6715899791172986`\n",
    "   - **Explanation**: R², also known as the coefficient of determination, indicates the proportion of the variance in the dependent variable (target) that is predictable from the independent variables (features). It ranges from 0 to 1, where:\n",
    "     - **0** means that the model explains none of the variability of the target variable.\n",
    "     - **1** means that the model explains all the variability of the target variable.\n",
    "   - **Interpretation**: An R² value of `0.6716` means that approximately 67.16% of the variance in the `total_bill` can be explained by the features in the model. This indicates a moderate level of explanatory power.\n",
    "\n",
    "2. **Mean Absolute Error (MAE)**:\n",
    "   - **Value**: `4.243246528600479`\n",
    "   - **Explanation**: MAE is the average of the absolute differences between the predicted values and the actual values. It provides a measure of the average magnitude of the errors in a set of predictions, without considering their direction.\n",
    "   - **Formula**: \n",
    "     \\[\n",
    "     \\text{MAE} = \\frac{1}{n} \\sum_{i=1}^{n} |y_i - \\hat{y}_i|\n",
    "     \\]\n",
    "     where \\( y_i \\) is the actual value and \\( \\hat{y}_i \\) is the predicted value.\n",
    "   - **Interpretation**: An MAE value of `4.2432` means that, on average, the predictions are off by approximately 4.24 units from the actual `total_bill`. Lower values of MAE indicate better predictive accuracy.\n",
    "\n",
    "3. **Mean Squared Error (MSE)**:\n",
    "   - **Value**: `33.83498887434812`\n",
    "   - **Explanation**: MSE is the average of the squared differences between the predicted values and the actual values. It provides a measure of the average magnitude of the errors, giving more weight to larger errors.\n",
    "   - **Formula**:\n",
    "     \\[\n",
    "     \\text{MSE} = \\frac{1}{n} \\sum_{i=1}^{n} (y_i - \\hat{y}_i)^2\n",
    "     \\]\n",
    "     where \\( y_i \\) is the actual value and \\( \\hat{y}_i \\) is the predicted value.\n",
    "   - **Interpretation**: An MSE value of `33.8350` means that, on average, the squared difference between the predicted and actual `total_bill` is 33.8350. Lower values of MSE indicate better predictive accuracy. However, because MSE squares the errors, it is more sensitive to outliers than MAE.\n",
    "\n",
    "### Code Example\n",
    "\n",
    "Let's put these metrics into context with the full code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "09383b66-0398-48d9-829c-5aa60bd6b9a6",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----------+------------------+\n|Independent Features|total_bill|        prediction|\n+--------------------+----------+------------------+\n| (6,[0,1],[4.5,4.0])|      45.0|35.833333333333336|\n+--------------------+----------+------------------+\n\nR²: -inf\nMean Absolute Error: 9.166666666666664\nMean Squared Error: 84.02777777777773\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.feature import VectorAssembler, StringIndexer\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "\n",
    "# Create a Spark session\n",
    "spark = SparkSession.builder.appName('LinearRegressionExample').getOrCreate()\n",
    "\n",
    "# Sample data\n",
    "data = [\n",
    "    (5.0, 2, \"Male\", \"Yes\", \"Sun\", \"Dinner\", 50.0),\n",
    "    (3.0, 3, \"Female\", \"No\", \"Sat\", \"Lunch\", 30.0),\n",
    "    (4.5, 4, \"Male\", \"Yes\", \"Fri\", \"Dinner\", 45.0)\n",
    "]\n",
    "columns = [\"tip\", \"size\", \"sex\", \"smoker\", \"day\", \"time\", \"total_bill\"]\n",
    "\n",
    "# Create a DataFrame\n",
    "df = spark.createDataFrame(data, columns)\n",
    "\n",
    "# Index categorical columns\n",
    "indexer = StringIndexer(\n",
    "    inputCols=[\"sex\", \"smoker\", \"day\", \"time\"],\n",
    "    outputCols=[\"sex_indexed\", \"smoker_indexed\", \"day_indexed\", \"time_index\"]\n",
    ")\n",
    "df_r = indexer.fit(df).transform(df)\n",
    "\n",
    "# Assemble features into a vector\n",
    "featureassembler = VectorAssembler(\n",
    "    inputCols=['tip', 'size', 'sex_indexed', 'smoker_indexed', 'day_indexed', 'time_index'],\n",
    "    outputCol=\"Independent Features\"\n",
    ")\n",
    "output = featureassembler.transform(df_r)\n",
    "\n",
    "# Select relevant columns for modeling\n",
    "finalized_data = output.select(\"Independent Features\", \"total_bill\")\n",
    "\n",
    "# Train-test split\n",
    "train_data, test_data = finalized_data.randomSplit([0.75, 0.25])\n",
    "\n",
    "# Check if the splits contain data\n",
    "if train_data.count() == 0 or test_data.count() == 0:\n",
    "    print(\"One of the train or test datasets is empty. Adjusting the split ratio.\")\n",
    "    train_data, test_data = finalized_data.randomSplit([0.9, 0.1])\n",
    "    if train_data.count() == 0 or test_data.count() == 0:\n",
    "        print(\"Still empty after adjustment. Using entire dataset for training.\")\n",
    "        train_data = finalized_data\n",
    "        test_data = finalized_data\n",
    "\n",
    "# Initialize and train the Linear Regression model\n",
    "regressor = LinearRegression(featuresCol='Independent Features', labelCol='total_bill')\n",
    "regressor = regressor.fit(train_data)\n",
    "\n",
    "# Make predictions on the test data\n",
    "predictions = regressor.transform(test_data)\n",
    "\n",
    "# Evaluate the model\n",
    "evaluator = RegressionEvaluator(\n",
    "    labelCol=\"total_bill\", predictionCol=\"prediction\", metricName=\"r2\")\n",
    "r2 = evaluator.evaluate(predictions)\n",
    "\n",
    "evaluator = RegressionEvaluator(\n",
    "    labelCol=\"total_bill\", predictionCol=\"prediction\", metricName=\"mae\")\n",
    "mae = evaluator.evaluate(predictions)\n",
    "\n",
    "evaluator = RegressionEvaluator(\n",
    "    labelCol=\"total_bill\", predictionCol=\"prediction\", metricName=\"mse\")\n",
    "mse = evaluator.evaluate(predictions)\n",
    "\n",
    "# Show predictions\n",
    "predictions.select(\"Independent Features\", \"total_bill\", \"prediction\").show()\n",
    "\n",
    "# Print performance metrics\n",
    "print(f\"R²: {r2}\")\n",
    "print(f\"Mean Absolute Error: {mae}\")\n",
    "print(f\"Mean Squared Error: {mse}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a44cb74d-330f-4730-a7f6-50a3aff15ddb",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Summary of Metrics\n",
    "\n",
    "- **R² (R-Squared)**: Indicates the proportion of variance in the dependent variable that is predictable from the independent variables. Higher values indicate better model performance.\n",
    "- **Mean Absolute Error (MAE)**: Measures the average magnitude of the errors in predictions, without considering their direction. Lower values indicate better model performance.\n",
    "- **Mean Squared Error (MSE)**: Measures the average magnitude of the errors, giving more weight to larger errors. Lower values indicate better model performance, but it is more sensitive to outliers than MAE.\n",
    "\n",
    "By understanding these metrics, you can better assess the performance of your regression model and make informed decisions about potential improvements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "b3f0d5d2-4c95-44c9-94d2-93d2d0a9ab5a",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "environmentMetadata": null,
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 2
   },
   "notebookName": "PYSPARK - LINER REGRESSION",
   "widgets": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}