Lecture #6 Logistic Regression ML 2022: Machine Learning

Location: https://people.sc.fsu.edu/∼jburkardt/classes/ml 2022/class06/lecture/logistic regression lecture.pdf

Egyptian god Maat weighed the sins of your heart against a feather.

The Logistic Regression Problem Logistic regression handles simple decision making.

• We often need to make a Yes or No choice using imperfect knowledge;

• Perhaps the only evidence we have is a record of our previous decisions;

• We seek a formula that can “explain” our previous decisions;

• This formula can be used to predict the right choice in new cases;

• We would also like to estimate how likely our decision is to be correct. As a classic example of decision making under uncertain knowledge, consider almost any medical diagnosis. The human body is a complicated and varied object, and few mathematical formulas are available to tell us exactly how to deal with a given case. After collecting a number of measurements, a doctor must often dedide Yes or No, perform a certain procedure or not. In the absence of a real scientific theory, this means the decision, even though it is based on the evidence of the measurements, is ultimately made simply by intuition built up over years of experience. So what’s a new doctor supposed to do?

Suppose that a doctor must decide whether a pregnant woman should have her baby by Caesarian section. A number of medical measurements would seem to be relevant before making the decision, but ultimately, the doctor has to make the choice. If the patient expresses concerns about this decision, the doctor would like to be able to explain whether the decision is strongly recommended or not.

This is an example of a classification problem. Our task is to look at a particular case, and decide which class it belongs to. In the logistic regression problem, there are only 2 such classes, which might be Yes/No, 0/1, Admit/Reject, Pass/Fail.

We will try to construct a formula that will produce a good decision based on our past experience. To do so, our first tool will be a mathematical function that does a reasonable job of representing a Yes/No function as a sequence of values that smoothly transition from 0 (No!) to 1 (Yes!).

1 The logistic function

To begin with, let us suppose that the output decision y is to made based on the value of just a single input x. Our decision will be simplified if we can somehow come up with a function y(x) that automatically returns a reasonable Yes or No value. The simplest such function might simply apply some cutoff value x^{∗}:

y(x) = \begin{cases}No & if x ≤ x^{∗} \\ Yes & if x^{∗} < x \end{cases}

However, this approach does not give us the additional information of how sure we are of our classification, and it does not generalize easily to situations involving multiple input variables x.

Consider the formula known as the logistic or sigmoid function:

y(x) = \frac{l}{1 + e^{−m(x−b)}}

which has parameters l (maximum value), m (slope) and b (cutoff). For our purposes, we will always set l = 1.

The logistic curve for l = 1, b = 2, m = 3.

The function y(x) is near 0 for values to the left of the cutoff, hits \frac{1}{2} at the cutoff value, and then rises to 1 on the right. The location of the cutoff is controlled by the value of b. The sharpness of the rise, and the width of the “uncertainty” region, depend on the slope m.

For a given problem, if we can determine good values of b and m, then we will have a way of classifying our data, as well as reporting how sure we are of our decision:

y(x) = \begin{cases}No! (90% sure) & if y(x) <= 0.10 \\ No & if 0.10 < y(x) < \frac{1}{2} \\ 50/50 & if y(x) = \frac{1}{2} \\ Yes & if \frac{1}{2} < y(x) < 0.90 \\ Yes! (90% sure) & if 0.90 < y(x) \end{cases}

We could tabulate the value of the logistic function with parameters l = 1, b = 2, m = 3, over the range −2.0 ≤ x ≤ 8.0.

1  x  y=l o g i s t i c ( 1 , 2 , 3 , x )

2

3  −2  0.000006
4  −1  0.000123
5  0  0.002472
6  1  0.047425
7  2  0.500000
8  3  0.952574
9  4  0.997527
10  5  0.999877
11  6  0.999994
12  7  1.000000
13  8  1.000000

Listing 1: Sample logistic function values.

The table verifies that the logistic function starts out essentially 0 and climbs to its maximum value l ( which here is 1), and at the cutoff point b (which here is 2) it reaches its halfway value l/2 (which here is 1/2). The role of the slope m is harder to see except in a plot. It controls how rapidly the function jumps from its low to high value.

For our purposes, l will always be 1, and, assuming we are working with a scalar variable x, we will rewrite the formula in terms of two weights stored in the vector w:

y(x) = \frac{1}{1 + e^{(−w_{0}−w_{1}x)}}

We will now investigate how to determine the values w that help us to systematize our decision making.

2 Apples and Oranges

A fruit company receives truckloads of apples and oranges mixed together. A robot is to be used to sort them. The robot can determine the weight x of each item, and then must compute y(x), which is 0 if the item should go in the orange box, and 1 if it should go in the apple box. This is an example of “logistic classification”. We have two categories or classes, and we need to assign each data item to one of those classes.

In order for the robot to do its job, we need to construct a formula for y(x). Instead of simply returning 0 or 1, our formula will return a number between 0 and 1, representing how strongly it thinks a given fruit is an orange or apple. All values below \frac{1}{2} would be regarded as oranges, while those above the limit are assumed to be apples.

To construct this formula, we will work with a version of the logistic function. As we have done in the past, we will modify our original data items x by inserting an initial value x_{0} = 1. In that case, our formula becomes

y(x) == \frac{1}{1 + e^{(−w_{0}x_{0}−w_{1}x_{1})}} = \frac{1}{1 + e^{−w′x}}

The missing information here is the value for the weight vector w.

Suppose we want to set up our formula based on a set of n data items X and 0/1 classification vector y. We can determine the values w if we assume we want to minimize the mean square error between the actual and computed class values for the data items, that is:

minimize mse(w) = \frac{1}{n} ∑_{i=0}^{i<n}(y_{i} − y(x_{i}))^{2}

We have solved this kind of problem before, using the method of gradient descent. We can construct a similar procedure for problems that have been posed with the logistic function.

Consider the following function: logistic regression(x,y,alpha,kmax), where

• x is our normalized input data, with an initial column of 1’s

• y is our output data, values of 0 or 1

• alpha is a learning rate, which might be 0.1 or 0.01

• kmax is the maximum number of iterations

The function uses the gradient descent method, and returns its best estimate for the weights w. Here is how it would be used for our gold coin problem:

1  import numpy as np

2

3  data = np . l o a d t x t ( ’ g o l d d a t a . t x t ’ )

4

5  g = data [ : , 0 ]
6  y = data [ : , 1 ]
7  m = len ( g )

8

9  gmin = np . min( g )
10  gmax = np .max( g )

11  g = ( g − np . min ( g ) ) / ( np .max ( g ) − np . min ( g ) )

12

13  x = np . z e r o s ( [ m, 2 ] )
14  x [ : , 0 ] = 1
15  x [ : , 1 ] = g

16

17  alpha = 0 . 0 2
18  kmax = 100000

19

20  from l o g i s t i c r e g r e s s i o n import l o g i s t i c r e g r e s s i o n

21

22  w = l o g i s t i c r e g r e s s i o n ( x , y , alpha , kmax )

Listing 2: Linear regression for gold coin data.

The weights w = [−2.311, 4.092] are returned for the normalized data values. This means that, if we actually want to compute the value of y for the first data value g[0], we need to do this by computing

1  gn = ( g [ 0 ] − gmin ) / ( gmax − gmin )
2  wtx = w [ 0 ] ∗ 1 . 0 + w [ 1 ] ∗ gn
3  y1 = 1 . 0 / ( 1 . 0 + np . exp ( − wtx ) )

In the following graph, we display the logistic function for our gold data, using the raw (un-normalized) values of the coin weights. This shows that the cutoff point is about 33.55 grams. Below this, a coin is more than 50% likely to be fake.

The logistic regression curve for the gold coin data.

3 Can we trust the logistic regression() function?

We will treat the logistic regression() function as a black box. It uses gradient descent, and we have already said that it is best for a gradient descent function that the data is normalized. Otherwise, badly scaled data can make it difficult to get convergence.

The variable alpha is a “learning rate”, a kind of step control variable that we saw in the gradient descent lab. For our problems, using a value of around 0.01 to 0.1 may be reasonable.

The variable kmax controls the number of steps of gradient descent to be carried out. For our problems, a value of 10,000 might be reasonable.

Because this function does not include any convergence tests, we have to guess whether or not it has done its job. One way to check that kmax is about the right size is to compare the results when you double it or cut it in half. If the results change significantly, this can indicate that more iterations would improve the accuracy. Similarly, if double or halving alpha makes a noticeable change in the results, this may indicate an issue with the convergence.

1  alpha = 0 . 0 2
2  kmax = 100000
3  w = l o g i s t i c r e g r e s s i o n ( x , y , alpha , kmax )
4  print ( w )

5

6  alpha = 0 . 0 2
7  kmax = 10000 # Try 1/10 the number o f i t e r a t i o n s
8  w = l o g i s t i c r e g r e s s i o n ( x , y , alpha , kmax )
9  print ( w )

10

11  alpha = 0 . 0 2
12  kmax = 200000 # Try t w i c e the number o f i t e r a t i o n s
13  w = l o g i s t i c r e g r e s s i o n ( x , y , alpha , kmax )

14  print ( w )

Listing 3: Is kmax about right?

4 Example: The comfort zone

The manager of an office complex has taken m = 44 measurements of humidity, h, and temperature t, and asked the office workers if they find the environmental comfortable (y = 1) or uncomfortable (y = 0).

Rather than keeping all this data, the manage would like a plot, or formula, or some simplified model that will generally suggest the comfortableness of any pair of humidity and temperature values.

Now we have two input variables to work with. We normalize the input values, create an m × 3 array x containing [1|h|t] and call logistic regression().

The logistic regression dividing line for the comfort data.

Instead of a cutoff value, we now have a cutoff line, of the form w′x = 0 or w_{0} 1 + w_{1} ĥ + w_{2} t̂= 0. Values to the left of this line represent comfortable mixtures of humidity and temperature. This plot was made using the normalized data ĥ, t̂.

1  import m a t p l o t l i b . pyplot as p l t
2  import numpy as np
3  from l o g i s t i c r e g r e s s i o n import l o g i s t i c r e g r e s s i o n

4

5  data = np . l o a d t x t ( ’ comfort data . t x t ’ )

6

7  h = data [ : , 0 ]
8  t = data [ : , 1 ]
9  y = data [ : , 2 ]
10  m = len ( h )

11

12  h = ( h − np . min ( h ) ) / ( np .max ( h ) − np . min ( h ) )
13  t = ( t − np . min ( t ) ) / ( np .max ( t ) − np . min ( t ) )

14

15  x = np . z e r o s ( [ m, 3 ] )
16  x [ : , 0 ] = 1
17  x [ : , 1 ] = h
18  x [ : , 2 ] = t

19

20  alpha = 1 . 0
21  kmax = 10000

22

23  w = l o g i s t i c r e g r e s s i o n ( x , y , alpha , kmax )

24

25  print ( ’ ’ )
26  print ( ’ Estimated weights W = (%g,%g,%g ) ’ % ( w[ 0 ] , w[ 1 ] , w [ 2 ] ) )

Listing 4: Linear regression for comfort data

5 Exercise: Admission Statistics

The file admit data.txt contains 100 records relating to college admission applications. Each record contains a student’s scores on an English exam (eng), on a math exam, (mat), and an admissions decision y, which is 0 (not admitted) or 1 (admitted). Our task is to find a logistic formula which, given eng and mat, can give a y value between 0 and 1, estimating the probability that the student will be admitted.

We will read the data, normalize it, pack the input values into an x array containing [1|eng|mat], and call

logistic regression() to get weights w.

1  import numpy as np
2  from l o g i s t i c r e g r e s s i o n import l o g i s t i c r e g r e s s i o n
3  data = np . l o a d t x t ( ’ admit data . t x t ’ )
4  eng = data [ : , 0 ]
5  mat = data [ : , 1 ]
6  admit = data [ : , 2 ]
7  m = len ( eng )

8

9  eng = ( eng − np . min ( eng ) ) / ( np .max ( eng ) − np . min ( eng ) )
10  mat = ( mat − np . min ( mat ) ) / ( np .max ( mat ) − np . min ( mat ) )

11

12  x = np . z e r o s ( [ m, 3 ] )
13  x [ : , 0 ] = 1
14  x [ : , 1 ] = eng [ : ]
15  x [ : , 2 ] = mat [ : ]

16

17  y = admit [ : ]

18

19  alpha = 0 . 0 1
20  kmax = 10000

21

22  w = l o g i s t i c r e g r e s s i o n ( x , y , alpha , kmax )

Listing 5: Linear regression for admissions data.

Three students apply late. We will use the weights w to decide whether to admit them. The raw data is:

1 eng mat
2  46  81
3  26  75
4  30  66

We need to use the normalized versions of these values, which are

1  eng  mat
2  0 . 6 6  0 . 8 8
3  0 . 2 2  0 . 7 4
4  0 . 3 1  0 . 5 5

We evaluate the exponent of the logistic function, and then the logistic function itself:

1  x1 = np . array ( [ 1 . 0 , 0 . 6 6 , 0 . 8 8 ] )
2  wx = np . dot ( w, x1 )
3  y1 = 1 . 0 / ( 1 . 0 + np . exp ( − wx ) )
4  print ( ’ Student 1 : wx = ’ , wx , ’ Y = ’ , y1 )

Listing 6: Evaluate formula for student #1

A y value near 0.5 is hard to decide, but values above that will signify admission, and lower values rejection.

6 Computing Assignment #7

The file caesarian data.txt contains 80 records; each record lists 6 values used by a doctor to determine whether or not to recommend that a baby be delivered by Caesarian section:

1. age, patient’s age in years;

2. num, number of previous deliveries by patient;

3. tim, delivery date (0=timely, 1 = premature, 2 = late); 4. pre: blood pressure (0=low, 1=normal, 2=high);

5. hrt: heart (0=healthy, 1=unhealthy);

6. cae: decision (0=normal delivery, 1=use Caesarian);

Write a program hw7.py which:

1): Reads the caesarian data file, normalizes the first 5 data items, creates an 80×6 data matrix x containing [1|age|num|tim|pre|hrt], calls logistic regression() to estimate the weights w, and prints w.

2): Uses the weights w to evaluate three new cases. The raw values are:

1 age num tim pre hrt
2  29  2  2  0  1
3  33  2  0  2  0
4  22  0  1  1  0

but when you evaluate the formula, you must use the normalized data values:

1 age num tim pre hrt
2 0 . 5 2 0 . 5 1 . 0 0 . 0 1 . 0
3 0 . 7 0 0 . 5 0 . 0 1 . 0 0 . 0
4 0 . 2 2 0 . 0 0 . 5 0 . 5 0 . 0

For new cases #1, 2, 3, return the values of w′ x (the exponent in the logistic function), y(x) (a number between 0 and 1), and your judgement of whether the patient should undergo a Caesarian operation: ’probably’, ’unsure’, ’probably not’.

Email a copy of your program, your results, and your decisions to Dr Schneier mhs64@pitt.edu by Wednesday, 30 October.