逻辑回归

概述:逻辑回归是一个分类算法,它可以处理二元分类以及多元分类。
逻辑回归的预测函数的sigmod函数,对于sigmod函数,当自变量趋于正无穷时,函数值趋向1,当自变量趋于负无穷时,函数趋向0。在二元逻辑回归中,我们取0.5作为分界线,大于0.5则y=1,否则为0

二元逻辑斯蒂回归推导

  • 推导的关键点 (3)
    1. 逻辑斯蒂回归的定义
    2. 损失函数(极大似然)
    3. 参数优化(梯度下降)
  • 给定训练集 T={(x1,y1),..,(xN,yN)},其中 x ∈ R^n, y ∈ {0, 1}
  1. 逻辑斯蒂回归的定义:
  2. 负对数函数作为损失函数:
    进一步代入 σ(x) 有:
  3. 求梯度
  4. 使用梯度下降法求解参数

多元逻辑回归

K元逻辑回归的概率分布如下:

代码:link1/link2

import numpy as np


def sigmoid(z):
    return 1 / (1 + np.exp(-z))


def cost(X, y, theta):
    first = np.multiply(-y, np.log(sigmoid(X @ theta)))
    second = np.multiply((1 - y), np.log(1 - sigmoid(X @ theta)))
    return np.sum(first - second) / len(X)


def gradient_descent(X, y, theta, alpha, iters):
    for i in range(iters):
        temp = (1 / len(X)) * X.T @ (sigmoid(X @ theta) - y)
        theta += -alpha * temp
    return theta


def predict(theta, X):
    probability = sigmoid(X @ theta)
    return [1 if x >= 0.5 else 0 for x in probability]


def cost_reg(X, y, theta, lambd):
    first = np.multiply(-y, np.log(sigmoid(X @ theta)))
    second = np.multiply((1 - y), np.log(1 - sigmoid(X @ theta)))
    reg = lambd * (theta.T @ theta) / len(X) / 2
    return np.sum(first - second) / len(X) + reg


def gradient_descent_reg(X, y, theta, alpha, iters, lambd):
    for i in range(iters):
        temp = (1 / len(X)) * X.T @ (sigmoid(X @ theta) - y) + 1 / len(X) * theta
        theta += -alpha * temp
    return theta


if __name__ == '__main__':
    X = np.array([[1, 2, 3], [2, 3, 4]])
    y = np.array([[1], [0]])
    theta = np.zeros((3, 1))
    # no reg
    print(cost(X, y, theta))
    theta = gradient_descent(X, y, theta, 0.01, 10000)
    print(cost(X, y, theta))

    # reg
    # print(cost_reg(X, y, theta, 1))
    # theta = gradient_descent_reg(X, y, theta, 0.01, 10000, 1)
    # print(cost_reg(X, y, theta, 1))
0