Machine Learning MATH1800: Selected Topics in Mathematics Clustering

Clustering is used if we believe our n data values fall naturally into k separate groups, or clusters. To classify the data, we want to assign a cluster index cluster(i) to each data item data(i,:). Assuming the clusters are roughly circular, then one way to do this is to pick a representative central point center(j,:) for each cluster and assign data items to the cluster with the nearest center. Mathematically, we imagine the vector cluster reports the assignment of each data item:

cluster(i) = arg min_{1≤j≤k}(data(i,:) − center(j,:))^{2}

In MATLAB, it may be more convenient to list all the data items that belong to a given cluster. Thus, the indices of the elements in cluster 1 might be listed in a vector c1, whose length also tells us how many elements are in cluster 1. For a two cluster problem, with d1 and d2 the distances of each data item to the two cluster centers, we could compute c1 in MATLAB by:

c1 = f i n d ( d1 < d2 ) ;

Given the centers, we can compute E, the cluster “energy” or cluster variance:

E = ∑_{i=1}^{n}(data(i,:) − center(cluster(i),:))^{2}

A good clustering has a low value of E and an optimal clustering minimizes it.

In MATLAB, we might compute the individual cluster energies, as in:

e1 = sum ( d1 ( c ) . ˆ 2 ) ;

and in our two-dimensional case, we can then easily compute E=e1+e2.

Notice that we are replacing n data values by k representatives, the cluster centers, so in some sense clustering is a kind of dimension reduction. In this case, we are not reducing the spatial dimension d, but the dimension n counting the number of data values.

Once we have the optimal cluster assignment, we can usually get better results by recomputing the centers as the average of the points belonging to them. By changing the centers, we might actually tempt some data points to switch clusters, and so the process of updating cluster assignments and centers can involve an iteration.

We will begin with some simple clustering efforts of our own, and then turn to the k-means algorithm, which can return the optimal cluster centers and cluster assignments by a single function call. Keep in mind that our measure of optimality is a small total cluster energy, E.

1 Read, separate, and plot the data

Timings have been made of the Old Faithful geyser, measuring the lengths of each eruption, and the pause that follows. We will work with a data file of n = 272 pairs of these measurements, hoping to find patterns.

1. Get a copy of the data file faithful.txt;

2. Create a data variable by reading the file;

3. Create x and y vectors from the two columns of data;

4. Create a scatterplot of x versus y;

2 Replot using an equal scale

The plot is autoscaled, that is, the data is shifted and squeezed to fit snugly in a fixed rectangle. This means it’s not immediately obvious that the ranges of the x and y variables are quite different. And that means that a vertical inch and a horizontal inch in the plot actually mean very different things. To see this, let’s redraw the plot, but this time use the same scale in both dimensions:

1. Repeat the commands that created a scatterplot of x versus y, but include one more command:

     plt.axes(’equal’);

We would much rather try to work with the data as it appears in picture 1, rather than picture 2! Similar scale problems occur all the time when working with data. If the appropriate action is not taken, the computational results can be seriously distorted.

3 Normalize the data

In order to remove the scale difference in our two sets of data, we can normalize. Given a set of values x that range from xmin to xmax, we can make a normalized set sx that ranges from 0 to 1, using the formula:

sx = \frac{x − xmin}{xmax − xmin}

Let’s make normalized copies of our x and y data:

1. Determine xmin, xmax, ymin and ymax for your data;

2. Create sx and sy, standardized versions of x and y, which vary between 0 and 1;

3. A scatterplot of sx versus sy will look the same as the original data, but the units of both axes will now run from 0 to 1;

4 Create two centers, and measure distances

We want to create two cluster, which we will call the blue and red clusters. We pick a blue center (bx,by) and a red center rx,ry. Then, just as students are bused to the nearest school, we will assign each data point (sx,sy) to the nearest center. The (Euclidean) distance between a point (bx,by) and sx,sy is

dist((bx, by), (sx, sy)) = \sqrt{(bx − sx)^{2} + (by − sy)^{2}}

First we will set our two centers. Then, for each data value, we will measure the distance to each center. This will allow us to split the data into two clusters.

1. Define bx = 0.4, by = 0.6, and rx=0.6, ry=0.3;

2. Define an n-vector bd, the distance of each data item to the blue cluster; similarly define rd;

5 Assign data to nearest center

1. Define vectors bc and rc, to list the data values that are closer to the blue or red cluster.

2. Use the function bc = np.where( bd < rd) to list the data for the blue cluster;

3. Use a similar command to set up rc for the red cluster;

4. Display the clusters in a scatterplot:

p l o t ( sx ( bc ) , sy ( bc ) , ’b . ’ )
p l o t ( sx ( rc ) , sy ( rc ) , ’ r . ’ )
p l o t ( bx , by , ’ bo ’ )
p l o t ( rx , ry , ’ ro ’ )

6 Replace centers by cluster averages

Notice that the cluster centers are not actually at the centers of their clusters. Better clusters could be obtained by averaging the data in each cluster.

1. Replace each cluster center by the average of its data elements;

2. Report these new centers in the table;

7 Use kmeans2() for best clustering

The process of clustering involves an iteration. After an initial choice of centers, we assign data to the nearest cluster. Then a better center is found by averaging the data in the cluster. Once we have new centers, we have to reeassign the data. Eventually, this process settles down. An algorithm for managing this process all the way to convergence is known as the kmeans algorithm. In fact, all the work we have done in this lab could be replaced by a single command:

from s c i p y . c l u s t e r . vq import kmeans2
data = l o a d t x t ( ’ f a i t h f u l . txt ’ )
centers , l a b e l s = kmeans2 ( data , 2 )
p l o t ( data [ bc , 0 ] , data [ bc , 2 ] , ’b . ’ )
p l o t ( data [ rc , 0 ] , data [ rc , 1 ] , ’ r . ’ )
p l o t ( c e n t e r s [ 0 , 0 ] , c e n t e r s [ 0 , 1 ] , ’ bo ’ )
p l o t ( c e n t e r s [ 1 , 0 ] , c e n t e r s [ 1 , 1 ] , ’ ro ’ )