其他分享
首页 > 其他分享> > 【Tensorflow+Keras】学习率指数、分段、逆时间、多项式衰减及自定义学习率衰减的完整实例

【Tensorflow+Keras】学习率指数、分段、逆时间、多项式衰减及自定义学习率衰减的完整实例

作者:互联网

目录

1 引言

2 实现

2.1 方法一

在Optimizer中指定衰减策略即可,实现简单。

(1)指数衰减

lr_scheduler = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_steps=10000,
    decay_rate=0.96)
optimizer = tf.keras.optimizers.SGD(learning_rate=lr_scheduler)

(2)分段衰减

[0~1000]steps,学习率为1.0,[10001~9000]steps,学习率为0.5,其他steps,学习率为0.1

step = tf.Variable(0, trainable=False)
boundaries = [1000, 10000]
values = [1.0, 0.5, 0.1]
learning_rate_fn = tf.keras.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)
lr_scheduler = learning_rate_fn(step)
optimizer = tf.keras.optimizers.SGD(learning_rate=lr_scheduler)

(3)多项式衰减

在10000步中从0.1衰减到0.001,使用开根式( power=0.5)

start_lr = 0.1
end_lr = 0.001
decay_steps = 10000
lr_scheduler = tf.keras.optimizers.schedules.PolynomialDecay(
    start_lr,
    decay_steps,
    end_lr,
    power=0.5)
optimizer = tf.keras.optimizers.SGD(learning_rate=lr_scheduler)

(4)逆时间衰减

initial_lr = 0.1
decay_steps = 1.0
decay_rate = 0.5
lr_scheduler = keras.optimizers.schedules.InverseTimeDecay(
  initial_lr, decay_steps, decay_rate)
optimizer = tf.keras.optimizers.SGD(learning_rate=lr_scheduler)

(5)完整实例

from sklearn import datasets
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.callbacks import LearningRateScheduler,ModelCheckpoint
from math import pow,floor

dataset =datasets.load_iris()

X = dataset.data
Y = dataset.target
# 设定随机种子
seed =7
np.random.seed(seed)

def create_model(init = 'glorot_uniform'):
    #构造模型
    model = Sequential()
    model.add(Dense(units=4,activation='relu',input_dim=4,kernel_initializer=init))
    model.add(Dense(units=6,activation='relu',kernel_initializer=init))
    model.add(Dense(units=3,activation='softmax',kernel_initializer=init))
    lr_scheduler = tf.keras.optimizers.schedules.ExponentialDecay(
                                                        initial_learning_rate=1e-2,
                                                        decay_steps=10000,
                                                        decay_rate=0.96)
    optimizer = SGD(lr=lr_scheduler,momentum=0.9,decay=0.0,nesterov=False)
    model.compile(loss='categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])
if __name__=="__main__":    
    checkpoint = ModelCheckpoint(filepath="weight.h5",monitor='val_acc',berbose=1,save_best_only=True,mode='max')
	model = KerasClassifier(build_fn = create_model,epochs=200,batch_size=5,verbose=1,callbacks=[checkpoint])
    model.fit(X,Y)

2.2 方法二

可以自定义学习率的衰减,该方法灵活。

(1)自定义指数衰减

前100epoch学习率不变,之后的epoch指数衰减。在以下程序的注释中,共三个步骤。

from sklearn import datasets
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.callbacks import LearningRateScheduler,ModelCheckpoint
from math import pow,floor

dataset =datasets.load_iris()

X = dataset.data
Y = dataset.target
seed =7
np.random.seed(seed)
# 第一步:自定义指数衰减策略
def step_decay(epoch):
    init_lr = 0.1
    drop=0.5
    epochs_drop=10
    if epoch<100:
        return init_lr
   	else:
        return init_lr*pow(drop,floor(1+epoch)/epochs_drop)
def create_model(init = 'glorot_uniform'):
    model = Sequential()
    model.add(Dense(units=4,activation='relu',input_dim=4,kernel_initializer=init))
    model.add(Dense(units=6,activation='relu',kernel_initializer=init))
    model.add(Dense(units=3,activation='softmax',kernel_initializer=init))
    lr_scheduler = tf.keras.optimizers.schedules.ExponentialDecay(
                                                        initial_learning_rate=1e-2,
                                                        decay_steps=10000,
                                                        decay_rate=0.96)
    optimizer = SGD(lr=lr_scheduler,momentum=0.9,decay=0.0,nesterov=False)
    model.compile(loss='categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])
if __name__=="__main__":    
    checkpoint = ModelCheckpoint(filepath="weight.h5",monitor='val_acc',berbose=1,save_best_only=True,mode='max')
    # 第二步:用LearningRateScheduler封装学习率衰减策略
    lr_callback = LearningRateScheduler(step_decay)
    # 第三步:加入callbacks
	model = KerasClassifier(build_fn = create_model,epochs=200,batch_size=5,verbose=1,callbacks=[checkpoint,lr_callback])
    model.fit(X,Y)

(2)动态修改学习率

ReduceLROnPlateau(monitor=‘val_acc’, mode=‘max’,min_delta=0.1,factor=0.2,patience=5, min_lr=0.001)

训练集连续patience个epochs的val_acc小于min_delta时,学习率将会乘以factor。mode可以选择max或者min,根据monitor的选择而灵活设定。min_lr是学习率的最低值。

from sklearn import datasets
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.callbacks import LearningRateScheduler,ModelCheckpoint
from math import pow,floor

dataset =datasets.load_iris()

X = dataset.data
Y = dataset.target
seed =7
np.random.seed(seed)

def create_model(init = 'glorot_uniform'):
    model = Sequential()
    model.add(Dense(units=4,activation='relu',input_dim=4,kernel_initializer=init))
    model.add(Dense(units=6,activation='relu',kernel_initializer=init))
    model.add(Dense(units=3,activation='softmax',kernel_initializer=init))
    lr_scheduler = tf.keras.optimizers.schedules.ExponentialDecay(
                                                        initial_learning_rate=1e-2,
                                                        decay_steps=10000,
                                                        decay_rate=0.96)
    optimizer = SGD(lr=lr_scheduler,momentum=0.9,decay=0.0,nesterov=False)
    model.compile(loss='categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])
if __name__=="__main__":    
    checkpoint = ModelCheckpoint(filepath="weight.h5",monitor='val_acc',berbose=1,save_best_only=True,mode='max')
    # 第一步:ReduceLROnPlateau定义学习动态变化策略
    reduce_lr_callback = ReduceLROnPlateau(monitor='val_acc', factor=0.2,patience=5, min_lr=0.001)
    # 第二步:加入callbacks
	model = KerasClassifier(build_fn = create_model,epochs=200,batch_size=5,verbose=1,callbacks=[checkpoint,reduce_lr_callback])
    model.fit(X,Y)

标签:自定义,keras,decay,Keras,lr,import,model,衰减
来源: https://blog.csdn.net/weixin_43935696/article/details/120602024