An Application of Generative Adversarial Networks (GAN)

If you are planning to start your own religion the first thigh you would need is a holy book! Now, writing a Holy book is from scratch is a sedulous task. To end this problem you can train the GANs model and it will do the job for you.

Basically, What I am trying to imply is that you train a GANs model to learn from any text and generate similar txt out of it.

Here in this example, we are going to use the Holly Bible as training data, which is in plain text format and can be downloaded from given link. (https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt)

Here is the output(with start string=’A’) after training a single layer GRU for 20 epochs with the default settings below:

And the third part of the law of the Lord Jesus Christ, the God of Israel, and the son of Joseph, and the son of Joseph were scattered and said, The first called the things which are in the first day, and the son of man, and the heavens and of the earth, and the sons of Joseph was a company of the Lord Jesus Christ, and the son of Joseph said unto him, Lord, what thou hast seen in the LORD our God, and the son of man is not afraid.

1:15 And the son of man, and the son of Joseph, which is in the faith of the Lord Jesus Christ, and the son of Joseph said unto him, Lord, what thou hast seen in the LORD our God, and the son of man is not afraid.

1:15 And the son of man, and the son of Joseph, which is in the faith of the Lord Jesus Christ, and the son of Joseph said unto him, Lord, what thou hast seen in the LORD our God, and the son of man is not afraid.

1:15 And the son of man, and the son of Joseph, which is in the faith of the Lord Jesus Christ, and the son of Joseph said unto him, L

Of course, some of the sentences do not make sense. But, considering they appear to very similar to the Bible itself.

Download File 1 (Codes)

Download File 2 (Codes)

Unicode library is required

We need to convert Unicode to ASCII.

In [0]:
!pip install unidecode

Import tensorflow and other libraries.

In [0]:
# Import TensorFlow
import tensorflow as tf

# Enable eager execution
tf.enable_eager_execution()

import numpy as np
import os
import re
import random
import unidecode
import time

Download the dataset

In this example, As already mentioned we will use The Holly Bible.

In [0]:
path_to_file = tf.keras.utils.get_file('bible.txt', 'https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt')

Read the dataset

In [0]:
text = unidecode.unidecode(open(path_to_file).read())
# length of text is the number of characters in it
print (len(text))
Create dictionaries to map from characters to their indices.

In [0]:
# unique contains all the unique characters in the file
unique = sorted(set(text))

# creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(unique)}
idx2char = {i:u for i, u in enumerate(unique)}
In [0]:
# On average there are 60 characters per line in our text file. set the maximum length sentence we want for a single input in characters
max_length = 60

# length of the vocabulary in chars
vocab_size = len(unique)

# the embedding dimension
embedding_dim = 256

# number of RNN (here GRU) units
units = 1024

# batch size
BATCH_SIZE = 64

# buffer size to shuffle our dataset
BUFFER_SIZE = 10000

Creating the input and output tensors

In [0]:
input_text = []
target_text = []

for f in range(0, len(text)-max_length, max_length):
inps = text[f:f+max_length]
targ = text[f+1:f+1+max_length]

input_text.append([char2idx[i] for i in inps])
target_text.append([char2idx[t] for t in targ])

print (np.array(input_text).shape)
print (np.array(target_text).shape)

Creating batches and shuffling them using tf.data

In [0]:
dataset = tf.data.Dataset.from_tensor_slices((input_text, target_text)).shuffle(BUFFER_SIZE)
dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)

Creating the model

In [0]:
class Model(tf.keras.Model):
def __init__(self, vocab_size, embedding_dim, units, batch_size):
super(Model, self).__init__()
self.units = units
self.batch_sz = batch_size

self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)

if tf.test.is_gpu_available():
self.gru = tf.keras.layers.CuDNNGRU(self.units,
return_sequences=True,
return_state=True,
recurrent_initializer='glorot_uniform')
else:
self.gru = tf.keras.layers.GRU(self.units,
return_sequences=True,
return_state=True,
recurrent_activation='sigmoid',
recurrent_initializer='glorot_uniform')

self.fc = tf.keras.layers.Dense(vocab_size)

def call(self, x, hidden):
x = self.embedding(x)

# output shape == (batch_size, max_length, hidden_size)
# states shape == (batch_size, hidden_size)

# states variable to preserve the state of the model
# this will be used to pass at every step to the model while training
output, states = self.gru(x, initial_state=hidden)

# reshaping the output so that we can pass it to the Dense layer
# after reshaping the shape is (batch_size * max_length, hidden_size)
output = tf.reshape(output, (-1, output.shape[2]))

# The dense layer will output predictions for every time_steps(max_length)
# output shape after the dense layer == (max_length * batch_size, vocab_size)
x = self.fc(output)

return x, states

Call the model and set the optimizer and the loss function

In [0]:
model = Model(vocab_size, embedding_dim, units, BATCH_SIZE)
In [0]:
optimizer = tf.train.AdamOptimizer()

# using sparse_softmax_cross_entropy so that we don't have to create one-hot vectors
def loss_function(real, preds):
return tf.losses.sparse_softmax_cross_entropy(labels=real, logits=preds)

Checkpoints (Object-based saving)

In [0]:
checkpoint_dir = './training_checkpoints'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(optimizer=optimizer,
model=model)

Train the model

This most time taking the task, and dependents upon the number of epochs for which you want to train your model. For this example, we will set epochs to only 20. For each epoch, it took about 100 seconds for me.

In [0]:
# Training step

EPOCHS = 20

for epoch in range(EPOCHS):
start = time.time()

# initializing the hidden state at the start of every epoch
hidden = model.reset_states()

for (batch, (inp, target)) in enumerate(dataset):
with tf.GradientTape() as tape:
# feeding the hidden state back into the model
# This is the interesting step
predictions, hidden = model(inp, hidden)

# reshaping the target because that's how the
# loss function expects it
target = tf.reshape(target, (-1,))
loss = loss_function(target, predictions)

grads = tape.gradient(loss, model.variables)
optimizer.apply_gradients(zip(grads, model.variables))

if batch % 100 == 0:
print ('Epoch {} Batch {} Loss {:.4f}'.format(epoch+1,
batch,
loss))
# saving (checkpoint) the model every 5 epochs
if (epoch + 1) % 5 == 0:
checkpoint.save(file_prefix = checkpoint_prefix)

print ('Epoch {} Loss {:.4f}'.format(epoch+1, loss))
print('Time taken for 1 epoch {} secn'.format(time.time() - start))

Restore the latest checkpoint

In [0]:
# restoring the latest checkpoint in checkpoint_dir
checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))

Predicting using our trained model

The below code block is used to generated the text

In [0]:
# Evaluation step(generating text using the model learned)

# number of characters to generate
num_generate = 1000

# You can change the start string to experiment
start_string = 'A'
# converting our start string to numbers(vectorizing!)
input_eval = [char2idx[s] for s in start_string]
input_eval = tf.expand_dims(input_eval, 0)

# empty string to store our results
text_generated = ''

# low temperatures results in more predictable text.
# higher temperatures results in more surprising text
# experiment to find the best setting
temperature = 1.0

# hidden state shape == (batch_size, number of rnn units); here batch size == 1
hidden = [tf.zeros((1, units))]
for i in range(num_generate):
predictions, hidden = model(input_eval, hidden)

# using a multinomial distribution to predict the word returned by the model
predictions = predictions / temperature
predicted_id = tf.argmax(predictions[0]).numpy()

# We pass the predicted word as the next input to the model
# along with the previous hidden state
input_eval = tf.expand_dims([predicted_id], 0)

text_generated += idx2char[predicted_id]

print (start_string + text_generated)

---

Download File 1 (Codes)

Download File 2 (Codes)

---

Note: This is a guest post, and opinion in this article is of the guest writer. If you have any issues with any of the articles posted at www.marktechpost.com please contact at [email protected]