Pysparking with iris dataset

Pyspark to cluster iris dataset

Before we start, you should have the following things in place :

1) oracle virtualbox

2) ubuntu mounted on the virtualbox

3) anaconda and spark installed the virtual environment

You can go through this blog to get ready with the above things : blog

Codes for pyspark are given below (note that the version of your spark might be different.

Mine was 2.0.1)

from numpy import array
from math import sqrt

from pyspark.mllib.clustering import KMeans, KMeansModel

# Load and parse the data
data = sc.textFile("data/mllib/kmeans_data.txt")
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))

# Build the model (cluster the data)
clusters = KMeans.train(parsedData, 2, maxIterations=10,
                        runs=10, initializationMode="random")

# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
    center = clusters.centers[clusters.predict(point)]
    return sqrt(sum([x**2 for x in (point - center)]))

WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))

# Save and load model
clusters.save(sc, "target/org/apache/spark/PythonKMeansExample/KMeansModel")
sameModel = KMeansModel.load(sc, "target/org/apache/spark/PythonKMeansExample/KMeansModel")

Kmeans On Jupyter

K Means Clustering on Iris Data set on Jupyter notebook

K Means clustering is an unsupervised machine learning algorithm.  During the learning process the error between the predicted outcome (predY) and actual outcome (y) is used to train the system. In an unsupervised method such as K Means clustering the outcome (y) variable is not used in the training process.

In this example we look at using the IRIS dataset.

Steps to go ahead with :-

• Importing the sample IRIS dataset
• Converting the dataset to a Pandas Dataframe
• Visualising the classifications using scatter plots
• Simple performance metrics

Requirements: I am using Anaconda Python Distribution which has everything you need including Pandas, NumPy, Matplotlib and importantly SciKit-Learn. I am also using iPython Notebook but you can use whatever IDE you want.

You can find the codes for the same in the below screenshot :

The plot showing the clusters of the flowers on the basis of petal width, sepal length and petal length:

So, we have clustered the iris dataset into 3 clusters using Kmeans algorithm.

Use can find the notebook for the same here : sushovan_notebook

For the iris dataset : sushovan's_iris

Enjoy Machine learning !!

😁😁