1、pre-training2、random initialization3、Xavier initialization因为Xavier的推导过程是基于2个假设的其中一个是激活函数是线性的。这并不适用于ReLU激活函数。另一个是激活值关于0对称这个不适用于sigmoid函数和ReLU函数实际情况是有可能在sigmoid函数上获得较好的效果。方法来源论文《Understanding the difficulty of training deep feedforward neural networks》。4、He initializationXavier初始化的变种适用于ReluHe initialization的思想是在ReLU网络中假定每一层有一半的神经元被激活另一半为0所以要保持variance不变只需要在Xavier的基础上再除以2这个方法试用 RELUwithout BN 激活函数时最好选用 He 初始化方法。这个初始化方法也主要是为relu激活函数设计的论文Kaiming He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classfication5、Batch Normalization目的使得每次输入参数分布相同方式通过BN公式计算keras代码测试import numpy as np import matplotlib.pyplot as plt import keras from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D,BatchNormalization,PReLU from keras.layers import Conv2D, MaxPool2D, AveragePooling2D, Activation, Embedding import keras.backend as K from keras.callbacks import LearningRateScheduler from keras.utils import np_utils from keras.callbacks import LearningRateScheduler, ModelCheckpoint, TensorBoard from keras.utils import plot_model from keras.preprocessing.image import ImageDataGenerator from keras.models import Model, Input from keras.optimizers import SGD from keras.applications.resnet50 import ResNet50 import tensorflow as tf def ObtainLayerOutput(input_model,input_layer_name,input_data): target_layer Model(inputsinput_model.input, outputsinput_model.get_layer(input_layer_name).output) layer_output target_layer.predict(input_data) return layer_output def ObtainLayerWeightsAndBias(model,input_layer_name): # 获得某一层的权重和偏置 weights model.get_layer(input_layer_name).get_weights() return weights def My_Initializer(): m_Zeros keras.initializers.Zeros() m_Ones keras.initializers.Ones() m_Constant keras.initializers.Constant(value1.1) m_RandomNormal keras.initializers.RandomNormal(mean0, stddev2.0, seed0) return m_RandomNormal def func_model(): # 定义一个8-16-2的感知器 IN keras.layers.Input(shape(3,)) m_kerrnel_initialMy_Initializer() m_beta_initializer keras.initializers.Constant(value0.1) m_gamma_initializer keras.initializers.Constant(value0.2) m_moving_mean_initializer keras.initializers.Constant(value0.3) m_moving_variance_initializer keras.initializers.Constant(value0.4) m_bn BatchNormalization(beta_initializerm_beta_initializer, gamma_initializerm_gamma_initializer, moving_mean_initializerm_moving_mean_initializer, moving_variance_initializerm_moving_variance_initializer)(IN) m_bn PReLU()(m_bn) HIDDEN keras.layers.Dense(5,use_biasFalse, activationrelu, kernel_initializerm_kerrnel_initial, bias_initializerones)(m_bn) OUT keras.layers.Dense(2, activationsigmoid, kernel_initializerones, bias_initializerzeros)(HIDDEN) model1 keras.models.Model(inputsIN, outputsOUT) model1.summary() return model1 对BatchNormalization层进行测试 计算符合如下公式 output (x - mean) / sqrt(var epsilon) * gamma beta def TestBatchNormalization(): model func_model() input_data np.ones((2, 3), dtypenp.float32) input_data[0][0]1.0 input_data[0][1]2.0 input_data[0][2]3.0 input_data[1][0] 1.1 input_data[1][1] 2.1 input_data[1][2] 3.1 print(input_data, input_data, np.shape(input_data)) m_batch_normalization_1 ObtainLayerWeightsAndBias(model, batch_normalization_1) print(m_batch_normalization_1, m_batch_normalization_1, np.shape(m_batch_normalization_1)) m_outputObtainLayerOutput(model,batch_normalization_1,input_data) print(m_output,m_output) if __name__ __main__: #2验证bath_normalization_1 TestBatchNormalization()参考文献1在网络中如何应用https://blog.csdn.net/appleml/article/details/791666952原理解析讲解较好https://blog.csdn.net/hjimce/article/details/508663133讲解较好https://www.cnblogs.com/hellcat/articles/7220040.html6、初始化为0一般只在训练线性回归/逻辑回归模型时才使用0初始化所有参数导致所有输出全部都一样参考为什么使用1https://www.cnblogs.com/lky-learning/p/10830223.html为什么在逻辑回归中可以使用0进行初始化2https://www.jianshu.com/p/02b52963486831、初始化方法优特点https://blog.csdn.net/mzpmzk/article/details/79839047https://www.cnblogs.com/WayneZeng/p/9290701.html2、使用不同激活函数时候应该使用的初始化值策略https://blog.csdn.net/shuibuzhaodeshiren/article/details/886978903、各种初始化方法讲解较为清晰
1、pre-training2、random initialization3、Xavier initialization因为Xavier的推导过程是基于2个假设的其中一个是激活函数是线性的。这并不适用于ReLU激活函数。另一个是激活值关于0对称这个不适用于sigmoid函数和ReLU函数实际情况是有可能在sigmoid函数上获得较好的效果。方法来源论文《Understanding the difficulty of training deep feedforward neural networks》。4、He initializationXavier初始化的变种适用于ReluHe initialization的思想是在ReLU网络中假定每一层有一半的神经元被激活另一半为0所以要保持variance不变只需要在Xavier的基础上再除以2这个方法试用 RELUwithout BN 激活函数时最好选用 He 初始化方法。这个初始化方法也主要是为relu激活函数设计的论文Kaiming He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classfication5、Batch Normalization目的使得每次输入参数分布相同方式通过BN公式计算keras代码测试import numpy as np import matplotlib.pyplot as plt import keras from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D,BatchNormalization,PReLU from keras.layers import Conv2D, MaxPool2D, AveragePooling2D, Activation, Embedding import keras.backend as K from keras.callbacks import LearningRateScheduler from keras.utils import np_utils from keras.callbacks import LearningRateScheduler, ModelCheckpoint, TensorBoard from keras.utils import plot_model from keras.preprocessing.image import ImageDataGenerator from keras.models import Model, Input from keras.optimizers import SGD from keras.applications.resnet50 import ResNet50 import tensorflow as tf def ObtainLayerOutput(input_model,input_layer_name,input_data): target_layer Model(inputsinput_model.input, outputsinput_model.get_layer(input_layer_name).output) layer_output target_layer.predict(input_data) return layer_output def ObtainLayerWeightsAndBias(model,input_layer_name): # 获得某一层的权重和偏置 weights model.get_layer(input_layer_name).get_weights() return weights def My_Initializer(): m_Zeros keras.initializers.Zeros() m_Ones keras.initializers.Ones() m_Constant keras.initializers.Constant(value1.1) m_RandomNormal keras.initializers.RandomNormal(mean0, stddev2.0, seed0) return m_RandomNormal def func_model(): # 定义一个8-16-2的感知器 IN keras.layers.Input(shape(3,)) m_kerrnel_initialMy_Initializer() m_beta_initializer keras.initializers.Constant(value0.1) m_gamma_initializer keras.initializers.Constant(value0.2) m_moving_mean_initializer keras.initializers.Constant(value0.3) m_moving_variance_initializer keras.initializers.Constant(value0.4) m_bn BatchNormalization(beta_initializerm_beta_initializer, gamma_initializerm_gamma_initializer, moving_mean_initializerm_moving_mean_initializer, moving_variance_initializerm_moving_variance_initializer)(IN) m_bn PReLU()(m_bn) HIDDEN keras.layers.Dense(5,use_biasFalse, activationrelu, kernel_initializerm_kerrnel_initial, bias_initializerones)(m_bn) OUT keras.layers.Dense(2, activationsigmoid, kernel_initializerones, bias_initializerzeros)(HIDDEN) model1 keras.models.Model(inputsIN, outputsOUT) model1.summary() return model1 对BatchNormalization层进行测试 计算符合如下公式 output (x - mean) / sqrt(var epsilon) * gamma beta def TestBatchNormalization(): model func_model() input_data np.ones((2, 3), dtypenp.float32) input_data[0][0]1.0 input_data[0][1]2.0 input_data[0][2]3.0 input_data[1][0] 1.1 input_data[1][1] 2.1 input_data[1][2] 3.1 print(input_data, input_data, np.shape(input_data)) m_batch_normalization_1 ObtainLayerWeightsAndBias(model, batch_normalization_1) print(m_batch_normalization_1, m_batch_normalization_1, np.shape(m_batch_normalization_1)) m_outputObtainLayerOutput(model,batch_normalization_1,input_data) print(m_output,m_output) if __name__ __main__: #2验证bath_normalization_1 TestBatchNormalization()参考文献1在网络中如何应用https://blog.csdn.net/appleml/article/details/791666952原理解析讲解较好https://blog.csdn.net/hjimce/article/details/508663133讲解较好https://www.cnblogs.com/hellcat/articles/7220040.html6、初始化为0一般只在训练线性回归/逻辑回归模型时才使用0初始化所有参数导致所有输出全部都一样参考为什么使用1https://www.cnblogs.com/lky-learning/p/10830223.html为什么在逻辑回归中可以使用0进行初始化2https://www.jianshu.com/p/02b52963486831、初始化方法优特点https://blog.csdn.net/mzpmzk/article/details/79839047https://www.cnblogs.com/WayneZeng/p/9290701.html2、使用不同激活函数时候应该使用的初始化值策略https://blog.csdn.net/shuibuzhaodeshiren/article/details/886978903、各种初始化方法讲解较为清晰
