遺傳人工神經(jīng)網(wǎng)絡Python實現(xiàn)

南氏珍藏 2019-06-23

展開全文

介紹

人工神經(jīng)網(wǎng)絡的靈感來自我們的大腦。遺傳算法受到進化的啟發(fā)。本文提出了一種新型的輔助訓練的神經(jīng)網(wǎng)絡:遺傳神經(jīng)網(wǎng)絡。這些神經(jīng)網(wǎng)絡具有適應度等特性，并使用遺傳算法訓練隨機生成的權重。遺傳優(yōu)化發(fā)生在任何形式的反向傳播之前，以給梯度下降提供一個更好的起點。

序列神經(jīng)網(wǎng)絡

序列神經(jīng)網(wǎng)絡接受一個輸入矩陣，在模型外部與一個真實輸出值的向量配對。然后通過遍歷每一層，通過權重和激活函數(shù)來變換矩陣。

這是一個序列神經(jīng)網(wǎng)絡，具有一個輸入矩陣，兩個隱藏層，一個輸出層，三個權重矩陣和一種激活函數(shù)。

訓練算法

最初的預測很可能是不準確的，所以為了訓練一個序列神經(jīng)網(wǎng)絡做出更好的預測，我們把它看作一個復合函數(shù)。

創(chuàng)建一個損失函數(shù)，輸入矩陣和真實輸出向量(X和y)保持不變。

現(xiàn)在所有的東西都是關于函數(shù)的，并且有一個明確的目標(最小化損失)，我們得到一個多變量微積分的優(yōu)化問題。

隨著模型顯示出越來越多的復雜性，梯度下降的計算成本可能變得非常昂貴。遺傳神經(jīng)網(wǎng)絡提供了一個可供選擇的初始訓練過程，以提供一個更好的起點，在反向傳播過程中允許更少的epochs。

遺傳神經(jīng)網(wǎng)絡

在遺傳神經(jīng)網(wǎng)絡中，網(wǎng)絡被視為具有fields和適應度的計算對象。這些fields被認為是在反向傳播之前通過遺傳算法優(yōu)化的基因。這使得梯度下降具有更好的起始位置，并且允許更少的訓練時間，并具有更高的模型測試準確度?？紤]以下遺傳神經(jīng)網(wǎng)絡，其中權重被視為計算對象中的fields。

這些fields是相對于遺傳神經(jīng)網(wǎng)絡的每個實例的基因。就像序列神經(jīng)網(wǎng)絡一樣，它可以表示為復合函數(shù)。

然而，在使用微積分之前，我們將使用遺傳算法采取進化方法來優(yōu)化權重。

遺傳算法

在自然界中，染色體交叉看起來是這樣的…

如果我們把染色體簡化成塊…

這與遺傳算法用于改變權重矩陣的邏輯相同。這個想法將是創(chuàng)建一個初始種群的n個遺傳神經(jīng)網(wǎng)絡，經(jīng)過正向傳播計算出一個適應度得分，最后選擇最適合的個體來創(chuàng)建孩子。這個過程將重復，直到找到最優(yōu)的初始權值進行反向傳播。

應用遺傳神經(jīng)網(wǎng)絡

首先，我們必須建立遺傳神經(jīng)網(wǎng)絡。我們使用的是具有四個輸入節(jié)點，兩個隱藏層和一個輸出層的訓練模型（以匹配上圖），這可以擴展到任何類型的神經(jīng)網(wǎng)絡。

import pandas as pd
import numpy as np
import random
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from keras.models import Sequential
from keras.layers import Dense
# New Type of Neural Network
class GeneticNeuralNetwork(Sequential):
 # Constructor
 def __init__(self, child_weights=None):
 # Initialize Sequential Model Super Class
 super().__init__()
 # If no weights provided randomly generate them
 if child_weights is None:
 # Layers are created and randomly generated
 layer1 = Dense(4, input_shape=(4,), activation='sigmoid')
 layer2 = Dense(2, activation='sigmoid')
 layer3 = Dense(1, activation='sigmoid')
 # Layers are added to the model
 self.add(layer1)
 self.add(layer2)
 self.add(layer3)
 # If weights are provided set them within the layers
 else:
 # Set weights within the layers
 self.add(
 Dense(
 4,
 input_shape=(4,),
 activation='sigmoid',
 weights=[child_weights[0], np.zeros(4)])
 )
 self.add(
 Dense(
 2,
 activation='sigmoid',
 weights=[child_weights[1], np.zeros(2)])
 )
 self.add(
 Dense(
 1,
 activation='sigmoid',
 weights=[child_weights[2], np.zeros(1)])
 )
 # Function for forward propagating a row vector of a matrix
 def forward_propagation(self, X_train, y_train):
 # Forward propagation
 y_hat = self.predict(X_train.values)
 # Compute fitness score
 self.fitness = accuracy_score(y_train, y_hat.round())
 # Standard Backpropagation
 def compile_train(self, epochs):
 self.compile(
 optimizer='rmsprop',
 loss='binary_crossentropy',
 metrics=['accuracy']
 )
 self.fit(X_train.values, y_train.values, epochs=epochs)

現(xiàn)在我們已經(jīng)建立了遺傳神經(jīng)網(wǎng)絡，我們可以開發(fā)出一種交叉算法。我們將使用類似于上面給出的生物圖示的單點交叉。每一個矩陣列都有相同的機會被選擇為一個交叉點，讓每一個父母組合他們的基因并將它們傳遞給孩子。

# Crossover traits between two Genetic Neural Networks
def dynamic_crossover(nn1, nn2):
 # Lists for respective weights
 nn1_weights = []
 nn2_weights = []
 child_weights = []
 # Get all weights from all layers in the first network
 for layer in nn1.layers:
 nn1_weights.append(layer.get_weights()[0])
 # Get all weights from all layers in the second network
 for layer in nn2.layers:
 nn2_weights.append(layer.get_weights()[0])
 # Iterate through all weights from all layers for crossover
 for i in range(0, len(nn1_weights)):
 # Get single point to split the matrix in parents based on # of cols
 split = random.randint(0, np.shape(nn1_weights[i])[1]-1)
 # Iterate through after a single point and set the remaing cols to nn_2
 for j in range(split, np.shape(nn1_weights[i])[1]-1):
 nn1_weights[i][:, j] = nn2_weights[i][:, j]
 # After crossover add weights to child
 child_weights.append(nn1_weights[i])
 # Add a chance for mutation
 mutation(child_weights)
 # Create and return child object
 child = GeneticNeuralNetwork(child_weights)
 return child

為了確保種群探索解空間，應該會發(fā)生突變。在這種情況下，因為解空間非常大，突變的概率顯著高于大多數(shù)其他遺傳算法。沒有特定的方法來改變矩陣，我們在矩陣上隨機執(zhí)行標量乘法，幅度為2-5。

# Chance to mutate weights
def mutation(child_weights):
 # Add a chance for random mutation
 selection = random.randint(0, len(child_weights)-1)
 mut = random.uniform(0, 1)
 if mut >= .5:
 child_weights[selection] *= random.randint(2, 5)
 else:
 # No mutation
 pass

最后，模擬遺傳神經(jīng)網(wǎng)絡的演化。我們需要網(wǎng)絡數(shù)據(jù)來學習，因此我們將使用眾所周知的 banknote機器學習數(shù)據(jù)集。

# Read Data
data = pd.read_csv('banknote.csv')
# Create Matrix of Independent Variables
X = data.drop(['Y'], axis=1)
# Create Vector of Dependent Variable
y = data['Y']
# Create a Train Test Split for Genetic Optimization
X_train, X_test, y_train, y_test = train_test_split(X, y)
# Create a List of all active GeneticNeuralNetworks
networks = []
pool = []
# Track Generations
generation = 0
# Initial Population
n = 20
# Generate n randomly weighted neural networks
for i in range(0, n):
 networks.append(GeneticNeuralNetwork())
# Cache Max Fitness
max_fitness = 0
# Max Fitness Weights
optimal_weights = []
# Evolution Loop
while max_fitness < .9:
 # Log the current generation
 generation += 1
 print('Generation: ', generation)
 # Forward propagate the neural networks to compute a fitness score
 for nn in networks:
 # Propagate to calculate fitness score
 nn.forward_propagation(X_train, y_train)
 # Add to pool after calculating fitness
 pool.append(nn)
 # Clear for propagation of next children
 networks.clear()
 # Sort based on fitness
 pool = sorted(pool, key=lambda x: x.fitness)
 pool.reverse()
 # Find Max Fitness and Log Associated Weights
 for i in range(0, len(pool)):
 # If there is a new max fitness among the population
 if pool[i].fitness > max_fitness:
 max_fitness = pool[i].fitness
 print('Max Fitness: ', max_fitness)
 # Reset optimal_weights
 optimal_weights = []
 # Iterate through all layers, get weights, and append to optimal
 for layer in pool[i].layers:
 optimal_weights.append(layer.get_weights()[0])
 print(optimal_weights)
 # Crossover, top 5 randomly select 2 partners for child
 for i in range(0, 5):
 for j in range(0, 2):
 # Create a child and add to networks
 temp = dynamic_crossover(pool[i], random.choice(pool))
 # Add to networks to calculate fitness score next iteration
 networks.append(temp)
# Create a Genetic Neural Network with optimal initial weights
gnn = GeneticNeuralNetwork(optimal_weights)
gnn.compile_train(10)
# Test the Genetic Neural Network Out of Sample
y_hat = gnn.predict(X_test.values)
print('Test Accuracy: %.2f' % accuracy_score(y_test, y_hat.round()))