programming assignment transformers architecture with tensorflow github

• TensorFlow Core
• TensorFlow.js
• TensorFlow Lite

May 23, 2019 — A guest article by Bryan M. Li , FOR.ai The use of artificial neural networks to create chatbots is increasingly popular nowadays, however, teaching a computer to have natural conversations is very difficult and often requires large and complicated language models. With all the changes and improvements made in TensorFlow 2.0 we can build complicated models with ease. In this post, we will demonstr…

• Preprocessing the Cornell Movie-Dialogs Corpus using TensorFlow Datasets and creating an input pipeline using tf.data
• Implementing MultiHeadAttention with Model subclassing
• Implementing a Transformer with Functional API

Transformer

• It makes no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects.
• Layer outputs can be calculated in parallel, instead of a series like an RNN.
• Distant items can affect each other’s output without passing through many recurrent steps, or convolution layers.
• It can learn long-range dependencies.
• For a time-series, the output for a time-step is calculated from the entire history instead of only the inputs and current hidden-state. This may be less efficient.
• If the input does have a temporal/spatial relationship, like text, some positional encoding must be added or the model will effectively see a bag of words.
• Extract a list of conversation pairs from move_conversations.txt and movie_lines.txt
• Preprocess each sentence by removing special characters in each sentence.
• Build tokenizer (map text to ID and ID to text) with TensorFlow Datasets SubwordTextEncoder .
• Tokenize each sentence and add START_TOKEN and END_TOKEN to indicate the start and end of each sentence.
• Filter out sentences that contain more than MAX_LENGTH tokens.
• Pad tokenized sentences to MAX_LENGTH
• Build tf.data.Dataset with the tokenized sentences

Scaled dot product attention

Multi-head Attention Layer

• Final linear layer.

Positional Encoding

Transformer with functional api, encoding layer.

• 2 dense layers followed by dropout
• N of encoder layers

Decoder Layer

• N decoder layers

Train the model

• Skip to primary navigation
• Skip to main content
• Skip to primary sidebar
• Skip to footer

PyImageSearch

You can master Computer Vision, Deep Learning, and OpenCV - PyImageSearch

Attention Deep Learning Transformers Tutorial

A Deep Dive into Transformers with TensorFlow and Keras: Part 1

by Aritra Roy Gosthipaty and Ritwik Raha on September 5, 2022

Table of Contents

A deep dive into transformers with tensorflow and keras: part 1, introduction, the transformer architecture, evolution of attention, scaling of the dot product, version 4 (cross-attention), version 5 (self-attention), version 6 (multi-head attention), citation information.

While we look at gorgeous futuristic landscapes generated by AI or use massive models to write our own tweets , it is important to remember where all this started.

Data, matrix multiplications, repeated and scaled with non-linear switches. Maybe that simplifies things a lot, but even today, most architectures boil down to these principles. Even the most complex systems, ideas, and papers can be boiled down to just that:

Data, matrix multiplications, repeated and scaled with non-linear switches.

Over the past few months, we have covered Natural Language Processing (NLP) through our tutorials. We started from the very history and foundation of NLP and discussed Neural Machine Translation with attention .

Here are all the tutorials chronologically.

• Introduction to Natural Language Processing (NLP)
• Introduction to the Bag-of-Words (BoW) Model
• Word2Vec: A Study of Embeddings NLP
• Comparison Between BagofWords and Word2Vec
• Introduction to Recurrent Neural Networks with Keras and TensorFlow
• Long Short-Term Memory Networks
• Neural Machine Translation
• Neural Machine Translation with Bahdanau’s Attention Using TensorFlow and Keras

Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras

Now, the progression of NLP, as discussed, tells a story. We begin with tokens and then build representations of these tokens. We use these representations to find similarities between tokens and embed them in a high-dimensional space. The same embeddings are also passed into sequential models that can process sequential data. Those models are used to build context and, through an ingenious way, attend to parts of the input sentence that are useful to the output sentence in translation .

Phew! That was a lot of research. We are almost something of a scientist ourselves .

But what lies ahead? A group of real scientists got together to answer that question and formulate a genius plan (as shown in Figure 1 ) that would shake the field of Deep Learning to its very core.

In this tutorial, you will learn about the evolution of the attention mechanism that led to the seminal architecture of Transformers.

This lesson is the 1st in a 3-part series on NLP 104 :

• A Deep Dive into Transformers with TensorFlow and Keras: Part 1 (today’s tutorial)
• A Deep Dive into Transformers with TensorFlow and Keras: Part 2
• A Deep Dive into Transformers with TensorFlow and Keras: Part 3

To learn how the attention mechanism evolved into the Transformer architecture, just keep reading.

In our previous blog post , we covered Neural Machine Translation models based on Recurrent Neural Network architectures that include an encoder and a decoder . In addition, to facilitate better learning, we also introduce the attention module .

Vaswani et al. proposed a simple yet effective change to the Neural Machine Translation models. An excerpt from the paper best describes their proposal.

We propose a new simple network architecture, the Transformer , based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

In today’s tutorial, we will cover the theory behind this neural network architecture called the Transformer. We will focus on the following in this tutorial:

We take a top-down approach in building the intuitions behind the Transformer architecture. Let us first look at the entire architecture and break down individual components later.

The Transformer consists of two individual modules, namely the Encoder and the Decoder , as shown in Figure 2 .

The first is a multi-head self-attention mechanism , and the second is a simple, position-wise, fully connected feed-forward network .

The authors also employ residual connections (red lines) and a normalization operation around the two sub-layers.

The source tokens are first embedded into a high-dimensional space. The input embeddings are added with positional encoding (we will cover positional encodings in depth later in the tutorial series). The summed embeddings are then fed into the encoder.

In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack.

The decoder also has residual connections and a normalization operation around the three sub-layers.

Notice that the first sublayer of the decoder is a masked multi-head attention layer instead of a multi-head attention layer.

The target tokens are offset by one. Like the encoder, the tokens are first embedded into a high-dimensional space. The embeddings are then added with positional encodings . The summed embeddings are then fed into the decoder.

This masking , combined with the fact that the target tokens are offset by one position, ensures that the predictions for position can depend only on the known outputs at positions less than .

The encoder and decoder have been built around a central piece called the Multi-Head Attention module. This piece of the architecture is the formula X that has placed Transformers at the top of the Deep Learning food chain. But Multi-Head Attention (MHA) did not always exist in its present form.

We have studied a very basic form of attention in the prior blog posts covering the Bahdanau and Luong attentions. However, the journey from the early form of attention to the one that is actually used in the Transformers architecture is long and full of monstrous notations.

But do not fear. Our quest will be to navigate the different versions of attention and counter any problems we might face. At the end of our journey, we shall emerge with an intuitive understanding of how attention works in the Transformer architecture.

To understand the intuition of attention, we start with an input and a query . Then, we attend to parts of the input based on the query. So if you have an image of a landscape and someone asks you to decipher the weather there, you would attend to the sky first. The image is the input, while the query is “how is the weather there?”

In terms of computation, attention is given to parts of the input matrix which is similar to the query vector . We compute the similarity between the input matrix and the query vector. After we obtain the similarity score , we transform the input matrix into an output vector. The output vector is the weighted summation (or average) of the input matrix.

Intuitively the weighted summation (or average) should be richer in representation than the original input matrix. It includes the “where and what to attend to.” The diagram of this baseline version (version 0) is shown in Figure 5 .

The two most commonly used attention functions are additive attention and dot-product (multiplicative) attention . Additive attention computes the compatibility function using a feed-forward network.

The first change we make to the mechanism is swapping out the feed-forward network with a dot product operation. Turns out that this is highly efficient with reasonably good results. While we use the dot product, notice how the shape of the input vectors now changes to incorporate the dot product. The diagram of version 1 is shown in Figure 6 .

Similarity function: Dot Product

Here let us pose some problems and devise the solutions ourselves. The scaling factor will be hidden inside the solution.

• Vanishing Gradient Problem: The weights of a Neural Network update in proportion to the gradient of the loss. The problem is that, in some cases, the gradient will be small, effectively preventing the weight from changing its value at all. This, in turn, prohibits the network from learning any further. This is often referred to as the vanishing gradient problem.
• Unnormalized softmax: Consider a normal distribution. The softmax of the distribution is heavily dependent on its standard deviation . With a huge standard deviation, the softmax will result in a peak with zeros all around. Figures 7-10 help visualize the problem.

• Unnormalized softmax leading to the vanishing gradient: Consider if your logits pass through softmax and then we have a loss (cross-entropy). The errors that backpropagate will be dependent on the softmax output. Now assume that you have an unnormalized softmax function, as mentioned above. The error corresponding to the peak will definitely be back-propagated, while the others (corresponding to zeros in the softmax) will not flow at all. This gives rise to the vanishing gradient problem.

To counter the problem of vanishing gradients due to unnormalized softmax, we need to find a way to have a better softmax output.

It turns out that the standard deviation of a distribution largely influences the softmax output. Let’s create a normal distribution with a standard deviation of 100. We also scale the distribution so that the standard deviation is unity. The code to create the distribution and scale it can be found in Figure 11 . Figure 12 visualizes the histograms of the distributions.

The histograms of both distributions seem alike. One is the scaled version of the other (look at the x -axis).

Let’s calculate the softmax of both and visualize them as shown in Figures 13 and 14 .

Scaling the distribution to unit standard deviation provides a distributed softmax output. This softmax allows the gradients to backpropagate, saving our model from collapsing .

We came across the vanishing gradient problem , the unnormalized softmax output , and also a way we can counter it. We are yet to understand the relationship between the above-mentioned problems and solutions to that of the scaled dot product proposed by the authors.

The attention layers consist of a similarity function that takes two vectors and performs a dot product. This dot product is then passed through a softmax to create the attention weights. This recipe is perfect for a vanishing gradient problem. The way to counter the problem is to transform the dot product result into a unit standard deviation distribution.

Previously we looked at a single query vector. Let us scale this implementation to multiple query vectors. We calculate the similarities of the input matrix with all the query vectors (query matrix) we have. The visualization of Version 3 is shown in Figure 17 .

To build cross-attention, we make some changes. The changes are specific to the input matrix. As we already know, attention needs an input matrix and a query matrix. Suppose we projected the input matrix into a pair of matrices, namely the key and value matrices.

The key matrix is attended to with respect to the query matrix. This results in attention weights. Here the value matrix is transformed with the attention weights as opposed to the input matrix transformation, as seen earlier.

This is done to decouple the complexity. The input matrix can now have a better projection that takes care of building attention weights and better output matrices as well. The visualization of Cross Attention is shown in Figure 18 .

With cross-attention, we learned that there are three matrices in the attention module: key, value, and query. The key and value matrix are projected versions of the input matrix. What if the query matrix also was projected from the input?

This results in what we call self-attention. Here the main motivation is to build a richer implementation of self with respect to self. This sounds funny, but it is highly important and forms the basis of the Transformer architecture. The visualization of Self-Attention is shown in Figure 19 .

This is the last stage of evolution. We have come a long way. We started by building the intuition of attention, and now we will discuss multi-head (self) attention.

The authors wanted to decouple relations further by introducing multiple heads of attention. This means that the key, value, and query matrices are now split into a number of heads and projected. The individual splits are then passed into a (self) attention module (described above).

All the splits are then concatenated into a single representation. The visualization of Multi-Head Attention is shown in Figure 20 .

If you have come this far, take a pause and congratulate yourselves. The journey has been long and filled with monstrous notations and numerous matrix multiplications. But as promised, we now have an intuitive sense of how Multi-Head Attention evolved. To recap:

• Version 0 started with the baseline, where the similarity function is computed between an input and a query using a feed-forward network.
• Version 1 saw us swap that feed-forward network for a simple dot product.
• Due to problems like vanishing gradients and unnormalized probability distribution, we use a scaled dot product in Version 2.
• In Version 3, we use multiple query vectors rather than just one.
• In Version 4, we build the cross-attention layer by breaking the input vector into key and value matrices.
• Whatever is found outside can also be found inside . Thus in Version 5, we obtain the query vector from the input as well, calling this the self-attention layer.
• Version 6 is the last and final form, where we see all relations between query, key, and value being further decoupled by using multiple heads.

Transformers might have multiple heads, but we have only one, and if it is spinning right now, we do not blame you. Here is an interactive demo to visually recap whatever we have learned thus far.

What's next? We recommend PyImageSearch University .

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do . My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 84 courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 84 Certificates of Completion
• ✓ 114+ hours of on-demand video
• ✓ Brand new courses released regularly , ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 536+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Attention is all you need , was published in the year 2017. Since then, it has absolutely revolutionized Deep Learning. Almost all tasks and novel architectures have leveraged Transformers as a whole or in parts.

The novelty of the architecture stands out when we study the evolution of the attention mechanism rather than singularly focusing on the version used in the paper.

This tutorial focused on developing this central piece: the Multi-Head Attention layer. In upcoming tutorials, we will learn about the connecting wires (feed-forward layers, positional encoding, and others) that hold the architecture together and also how to code the architecture in TensorFlow and Keras.

A. R. Gosthipaty and R. Raha. “A Deep Dive into Transformers with TensorFlow and Keras: Part 1,” PyImageSearch , P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, https://pyimg.co/8kdj1

Unleash the potential of computer vision with Roboflow - Free!

• Step into the realm of the future by signing up or logging into your Roboflow account . Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
• Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
• Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
• Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
• Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

Join Roboflow Now

Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to join the PyImageSearch Newsletter and download my FREE 17-page Resource Guide PDF on Computer Vision, OpenCV, and Deep Learning.

About the Author

Hey there, fellow developer! Howdy doo!

We're Aritra Roy Gosthipaty and Ritwik Raha, two quirky Deep Learning (DL) engineers at PyImageSearch who just love teaching, academic research, and developing cool stuff in the fields of DL and Computer Vision. We're like detectives, always on the hunt for the most complex topics and novel ideas in research papers. And once we find them, we love breaking them down into easy-to-understand blogs that anyone could read, learn from, and implement on their own.

If you like what you see and want to connect with us, feel free to drop Aritra a line on Twitter at @ariG23498, or Ritwik at @ritwik_raha. We promise we are not bots....well, not completely!

Previous Article:

Next Article:

CycleGAN: Unpaired Image-to-Image Translation (Part 1)

Comment section.

Hey, Adrian Rosebrock here, author and creator of PyImageSearch. While I love hearing from readers, a couple years ago I made the tough decision to no longer offer 1:1 help over blog post comments.

At the time I was receiving 200+ emails per day and another 100+ blog post comments. I simply did not have the time to moderate and respond to them all, and the sheer volume of requests was taking a toll on me.

Instead, my goal is to do the most good for the computer vision, deep learning, and OpenCV community at large by focusing my time on authoring high-quality blog posts, tutorials, and books/courses.

If you need help learning computer vision and deep learning, I suggest you refer to my full catalog of books and courses — they have helped tens of thousands of developers, students, and researchers just like yourself learn Computer Vision, Deep Learning, and OpenCV.

Click here to browse my full catalog.

Similar articles

Ml days in tashkent — day 1: city tour, opencv fast fourier transform (fft) for blur detection in images and video streams, covid-19: face mask detector with opencv, keras/tensorflow, and deep learning.

You can learn Computer Vision, Deep Learning, and OpenCV.

Get your FREE 17 page Computer Vision, OpenCV, and Deep Learning Resource Guide PDF. Inside you’ll find our hand-picked tutorials, books, courses, and libraries to help you master CV and DL.

• Deep Learning
• Dlib Library
• Embedded/IoT and Computer Vision
• Face Applications
• Image Processing
• OpenCV Install Guides
• Machine Learning and Computer Vision
• Medical Computer Vision
• Optical Character Recognition (OCR)
• Object Detection
• Object Tracking
• OpenCV Tutorials
• Raspberry Pi

Books & Courses

• PyImageSearch University
• FREE CV, DL, and OpenCV Crash Course
• Practical Python and OpenCV
• Deep Learning for Computer Vision with Python
• PyImageSearch Gurus Course
• Raspberry Pi for Computer Vision
• Get Started
• Privacy Policy

Armin Norouzi

Data Dcientist and Machine Learning Engineer

• waterloo, Canada
• ResearchGate
• Google Scholar

Transformer with TensorFlow

88 minute read

Published: May 26, 2023

This notebook provides an introduction to the Transformer, a deep learning model introduced in the paper “Attention Is All You Need” by Vaswani et al. The Transformer has revolutionized natural language processing and is now a fundamental building block of many state-of-the-art models.

The notebook also includes a TensorFlow implementation of the Transformer. It covers the essential components of the Transformer, including the self-attention mechanism, the feedforward network, and the encoder-decoder architecture. The implementation uses the Keras API in TensorFlow and demonstrates how to train the model on a toy dataset for machine translation.

By the end of the notebook, readers should have a good understanding of the Transformer architecture and be able to implement it in TensorFlow. The post is compatible with Google Colaboratory with Pytorch version 1.12.1+cu113 and can be accessed through this link:

Table of Contents:

• Intorduction to Transformer
• Tensorflow implementation of Transformer

1. Intorduction to Transformer

1.1. generative adversarial network vs transformer:.

In the previous section, we learn what GAN is and how it works. Now let’s see what is transformers and why we need such a thing.

We learned GAN is a type of neural network that consists of two networks, a generator and a discriminator . The generator tries to create new data samples that are similar to the input data, while the discriminator tries to distinguish between the real and fake data samples. The two networks are trained together in a way that the generator learns to create more realistic samples while the discriminator gets better at distinguishing between the real and fake samples.

On the other hand, the Transformer is a type of neural network architecture that was introduced in the field of natural language processing (NLP). It is mainly used for tasks such as language translation , text summarization , and language modelling . The Transformer model consists of an encoder and a decoder that work together to process input sequences and generate output sequences. The encoder processes the input sequence and produces a hidden representation of the input. The decoder then takes the hidden representation and generates the output sequence. The Transformer uses a self-attention mechanism that allows the model to focus on different parts of the input sequence while processing it. Additionally, the Transformer uses a positional encoding technique to preserve the order of the input sequence, which is important for language tasks.

While both models are used for different tasks, they do share some similarities. Both GAN and Transformer are deep learning models that are based on neural networks and use backpropagation to train their parameters. Additionally, they both have been used for generating realistic images and natural language text.

However, the key difference between the two models is that GAN is used for generative tasks, while the Transformer is used for tasks related to natural language processing. GANs generate new samples, while Transformers transform input sequences into output sequences.

1.2. We have RNN and LSTM; why do we need transformers?

While RNNs and LSTMs are powerful models that have been used successfully in many natural language processing tasks, they have certain limitations that can make them less effective for certain tasks. Here are a few reasons why Transformers have emerged as an important alternative to RNNs and LSTMs:

Long-term dependencies: RNNs and LSTMs are designed to capture sequential dependencies in data, which makes them well-suited for modeling time-series data or sequences of variable length. However, they can struggle to capture long-term dependencies in data, particularly when the distance between the relevant elements in the sequence is large. Transformers are designed to explicitly model long-range dependencies using self-attention mechanisms, which allow them to attend to different parts of the input sequence and capture long-term relationships.

Parallelization: RNNs and LSTMs process data sequentially, which can make them slower and more computationally expensive than other models. Transformers, on the other hand, can process the entire input sequence in parallel, which makes them more efficient and faster to train. This is particularly important for large-scale natural language processing tasks that involve processing large amounts of data.

Handling variable-length inputs: RNNs and LSTMs are designed to handle input sequences of variable length, but they can struggle with very long sequences or sequences that contain significant amounts of noise or irrelevant information. Transformers are better suited for handling variable-length inputs and can effectively filter out noise or irrelevant information using their attention mechanisms.

Attention-based mechanisms: Transformers are designed to use attention-based mechanisms, which allow them to dynamically focus on different parts of the input sequence based on the context of the task. This makes them particularly well-suited for tasks that require the model to selectively attend to different parts of the input sequence, such as machine translation or question answering.

1.2. Transformer components

We briefly talked about the transformer component; let’s dive into it in more detail.

This figure shows transformers architecture schematically. The Transformer architecture consists of an encoder and decoder, which are composed of multiple layers that use attention and self-attention mechanisms to process the input and output sequences. The positional encoding technique is used to encode the position of tokens in the input sequence. These components work together to enable the Transformer to achieve state-of-the-art performance on various natural language processing tasks.

Encoder: The encoder is the part of the Transformer architecture that processes the input sequence and produces a hidden representation of the sequence. The input sequence is first transformed into a sequence of embeddings, which are then fed into a stack of identical layers. Each layer in the encoder stack consists of two sublayers: a self-attention layer and a feedforward layer. The self-attention layer allows the encoder to attend to different parts of the input sequence and capture long-range dependencies, while the feedforward layer applies a nonlinear transformation to the hidden representation.

Decoder: The decoder is the part of the Transformer architecture that generates the output sequence based on the hidden representation produced by the encoder. Like the encoder, the decoder also consists of a stack of identical layers, but each layer has three sublayers: a self-attention layer, an encoder-decoder attention layer, and a feedforward layer. The self-attention layer allows the decoder to attend to different parts of the output sequence, while the encoder-decoder attention layer allows the decoder to attend to different parts of the input sequence.

Attention: Attention is a mechanism in neural networks that allows the model to selectively attend to different parts of the input when making a prediction. In the Transformer architecture, attention is used in both the encoder and decoder. The attention mechanism calculates a weighted sum of the values of the input sequence, where the weights are determined by the similarity between the query and the keys. The attention mechanism allows the model to focus on different parts of the input sequence depending on the task at hand.

Self-Attention Mechanism: Self-attention is a specific type of attention mechanism that is used in the Transformer architecture. In self-attention, the input sequence is transformed into a sequence of query, key, and value vectors. The query vectors are used to calculate the attention weights for each position in the input sequence, based on the similarity between the query vector and the key vectors. The value vectors are then weighted by the attention weights and summed up to produce a weighted representation of the input sequence. This weighted representation is then used as the input for the next layer of the model. Self-attention allows the model to attend to different parts of the input sequence and capture long-range dependencies.

Positional Encoding: Positional encoding is a technique used in the Transformer architecture to encode the position of the tokens in the input sequence. Since the Transformer does not have a recurrence or convolutional structure that can capture the order of the input sequence, the positional encoding is added to each token’s embedding to provide the model with information about the position of the token in the sequence. The positional encoding is calculated using a fixed function that takes into account the position of the token in the sequence and the dimension of the embedding. The result is then added to the token’s embedding, allowing the model to differentiate between tokens that appear in different positions in the input sequence.

For learning how these componenet working in a big picture, I refer you to Google AI Blog post :

Neural networks for machine translation typically contain an encoder reading the input sentence and generating a representation of it. A decoder then generates the output sentence word by word while consulting the representation generated by the encoder. The Transformer starts by generating initial representations, or embeddings, for each word… Then, using self-attention, it aggregates information from all of the other words, generating a new representation per word informed by the entire context, represented by the filled balls. This step is then repeated multiple times in parallel for all words, successively generating new representations.

Applying the Transformer to machine translation. Source: Google AI Blog .

Now let’s dive into the code. I used most of these codes based on official tutorials from Tensorflow. I suggest going through that if you are more experienced in the machine learning world. I changed it slightly to explain the codes better so I can understand and teach!

You can find tutorials here: https://www.tensorflow.org/text/tutorials/transformer

2. Tensorflow implementation of Transformer

2.1. setting environement and training data prepration.

First let’s import the necessary libraries for building and training a transformer model

Let’s donwload TED Talks dataset for Portuguese-to-English translation using TensorFlow Datasets (TFDS) :

The tfds.load() function is used to load the dataset. The arguments passed to the function are:

• 'ted_hrlr_translate/pt_to_en': This specifies the name of the dataset to load, which is the TED Talks dataset for Portuguese-to-English translation.
• with_info=True: This specifies that additional metadata about the dataset should be returned along with the dataset itself.
• as_supervised=True: This specifies that the dataset should be returned as a tuple of (input, target) pairs, where input is a Portuguese sentence and target is the corresponding English translation.

Let’s print out the first batch of examples from the Portuguese-to-English translation dataset loaded using TensorFlow Datasets (TFDS).

The train_examples.batch(3).take(1) function call batches the dataset into groups of three examples and then takes the first batch. This means that the code will print out the first three examples in the dataset.

The code then loops over the examples in the batch and prints out each example in both Portuguese and English. The .decode('utf-8') function call is used to convert the byte strings in the dataset to human-readable text.

2.2. Set up the tokenizer

Now time to tokenize our text.

Let’s download and load the tokenizers used for the Portuguese-to-English translation model provided by TensorFlow.

This tutorial follows main tuterial from tensorflow website and uses the tokenizers built in the subword tokenizer tutorial. That tutorial optimizes two text.BertTokenizer objects (one for English, one for Portuguese) for this dataset and exports them in a TensorFlow saved_model format.

The tf.keras.utils.get_file() function is used to download a zipped version of the tokenizers from the TensorFlow website. The first argument specifies the name of the downloaded file, while the second argument specifies the URL from which to download the file. The cache_dir argument specifies the directory in which to cache the downloaded file, while cache_subdir specifies the subdirectory in which to store the file. The extract argument specifies whether to extract the contents of the downloaded zip file.

Now we can use the tf.saved_model.load() function to load the tokenizers from the saved model. The model_name argument specifies the name of the saved model to load, which in this case is ted_hrlr_translate_pt_en_converter .

The tokenize function is used to convert a group of strings into a batch of token IDs with padding. Prior to tokenization, the function splits punctuation, converts all letters to lowercase, and normalizes the input to Unicode format. However, since the input data has already been standardized, these steps are not apparent in the code. Let’s check an exampple before and after tokenizer:

The detokenize method tries to transform the token IDs into text that can be easily read and understood by humans.

The lower level lookup method converts from token-IDs to token text:

Now let’s take a closer look at data by ploting the distribution of token lengths.

First, an empty list called lengths is created to store the token lengths. Then, for each batch of 1024 examples in the training set, we can use the tokenizers.pt.tokenize() and tokenizers.en.tokenize() functions to tokenize the Portuguese and English examples, respectively. The row_lengths() function is then used to compute the number of tokens in each row of the tokenized data, and the resulting lengths are appended to the lengths list.

After processing all of the batches, the np.concatenate() function is used to concatenate all of the token lengths into a single numpy array called all_lengths. This array is then used to create a histogram of token lengths using the plt.hist() function.

2.3. Set up a data pipeline

Let’s write prepare_batch() function that prepares a batch of examples for training a machine translation model. The input to the function is a batch of Portuguese and English sentences, and the output is a tuple of two tensors representing the input and output sequences for the model.

First, the Portuguese sentences are tokenized using the tokenizers.pt.tokenize() method, which returns a ragged tensor representing the tokenized sentences. The code then trims the tensor to a maximum length of MAX_TOKENS using the pt[:, :MAX_TOKENS] syntax, which selects the first MAX_TOKENS tokens from each sentence. The resulting tensor is converted to a dense tensor with zero padding using the pt.to_tensor() method.

The English sentences are tokenized and trimmed in a similar way, but with an additional step. The en[:, :(MAX_TOKENS+1)] syntax selects the first MAX_TOKENS+1 tokens from each sentence, which includes the start token [START] and end token [END]. The en_inputs tensor is created by selecting all but the last token from each sentence, which drops the end token. The en_labels tensor is created by selecting all but the first token from each sentence, which drops the start token.

Finally, the function returns a tuple of two tensors, pt, en_inputs and en_labels , which represent the input and output sequences for the machine translation model. These tensors can be used to train the model using techniques such as teacher forcing, where the model is trained to predict the next token in the output sequence given the input sequence and the ground truth output sequence up to that point.

Now let’s take dataset and converted it into batches that are ready to be fed to the model.

The following function shuffles the examples in the dataset and batches them into batches of size BATCH_SIZE . It then applies the prepare_batch function to each batch, which tokenizes the text and prepares the input and output sequences for the model. Finally, it prefetches the batches to improve performance during training. The BUFFER_SIZE parameter determines the number of examples to load into memory for shuffling. The tf.data.AUTOTUNE argument allows TensorFlow to automatically tune the input pipeline for optimal performance.

Let’s see if we did everything write by testing dataset

The function make_batches prepares tf.data.Dataset objects for training a Keras model. The model is expected to take input in the form of pairs of tokenized Portuguese and English sequences (pt, en) , and predict the English sequences shifted by one token. This is known as “teacher forcing” because at each timestep, the model receives the true value as input for the next timestep regardless of its previous output. This is a simple and efficient way to train a text generation model as the outputs can be computed in parallel.

While one might expect the input, output pairs to simply be the Portuguese, English sequences, this setup adds “context” to the model by conditioning it on the Portuguese sequence. It is possible to train a model without conditioning it on the Portuguese sequence, but that would require writing an inference loop and passing the model’s output back to the input. This is slower and harder to learn but can result in a more stable model as the model has to learn to correct its own errors during training.

The en and en_labels are the same, just shifted by 1:

2.4. Define the components

2.4.1. the embedding and positional encoding layer.

Both the encoder and decoder components use the same logic to convert input tokens to vectors. This is done using a tf.keras.layers.Embedding layer, which creates a vector representation for each token in the input sequence.

The attention layers in the model don’t rely on the order of the tokens in the input sequence, because the model doesn’t contain any recurrent or convolutional layers that would inherently capture the sequence order. Without a way to identify the word order, the model would see the input sequence as a “bag of words”, where the order of the tokens doesn’t matter. For example, the sequences “how are you”, “how you are”, and “you how are” would all be seen as identical by the model.

To overcome this issue, a Transformer model adds a “Positional Encoding” to the embedding vectors. The Positional Encoding uses a set of sines and cosines at different frequencies across the sequence. Each token in the input sequence has a unique positional encoding that captures its position in the sequence. The nearby tokens in the sequence will have similar positional encodings. By incorporating this information into the input representation, the model can maintain the sequential order of the input tokens and better understand the meaning of the sentence.

The formula for calculating the positional encoding is as follows:

\(\Large{PE_{(pos, 2i)} = \sin(pos / 10000^{2i / d_{model}})}\) \(\Large{PE*{(pos, 2i+1)} = \cos(pos / 10000^{2i / d*{model}})}\)

Now, let’s implement it:

The positional_encoding function generates a matrix of position encodings for the input sequence. The purpose of positional encoding is to add information about the position of each token in the sequence, so that the self-attention mechanism in the transformer can distinguish between the different positions of the tokens.

The function takes two arguments: length , which specifies the length of the input sequence, and depth , which specifies the dimensionality of the encoding.

The function first creates two matrices: positions and depths . positions has shape (length, 1) and contains the indices of the positions in the input sequence. depths has shape (1, depth/2) and contains values ranging from 0 to (depth/2)-1 , which are then normalized by depth/2 .

The function then calculates the angle rates using the formula 1 / (10000**depths) , which has shape (1, depth/2) . The angle rates are used to calculate the angle radians using the formula positions * angle_rates , which has shape (length, depth/2) .

Finally, the function concatenates the sine and cosine values of the angle radians along the last axis to create the position encoding matrix, which has shape (length, depth) . The resulting matrix is then cast to tf.float32 and returned.

The position encoding function uses a series of sines and cosines that oscillate at various frequencies based on where they are positioned along the depth of the embedding vector. These oscillations occur across the position axis. Let’s visualize it here:

The purpose of this plot is to visualize the positional encoding matrix and see how it changes across different positions and depths in the sequence. It also helps to ensure that the encoding values are properly normalized and distributed across the matrix

Let’s visualize the cosine similarity between the positional encoding vector at index 1000 and all other vectors in the positional encoding matrix.

The positional encoding vectors are first normalized using L2 normalization. The code then calculates the dot product between the positional encoding vector at index 1000 and all other vectors in the matrix using the einsum function. The resulting dot products are plotted in a graph with the y-axis representing the cosine similarity values between the vectors.

The first plot shows the entire cosine similarity graph, while the second plot zooms in on the cosine similarity values between index 950 and 1050.

This visualization helps to illustrate how the positional encoding vectors encode the position information of each token in the sequence. The cosine similarity values are highest for vectors that are close to each other along the position axis, indicating that they have similar positional information.

Now let’s put things togher and create PositionEmbedding class. This is a tf.keras.layers.Layer class that combines an embedding layer and a positional encoding layer to create a layer that can be used to encode input sequences in a transformer model.

The class takes two arguments: vocab_size which is the size of the vocabulary of the input sequences and d_model which is the size of the embedding and positional encoding vectors.

In the constructor, it creates an Embedding layer that maps input tokens to their corresponding embedding vectors, and a positional encoding matrix of shape (max_length, d_model) using the positional_encoding function.

The compute_mask method of this class returns a mask with the same shape as the input tensor to the embedding layer.

In the call method, the input tensor is first passed through the embedding layer, and then scaled by the square root of the d_model value. Then, the positional encoding matrix is added to the embedding output corresponding to each input token. Finally, the encoded input sequence is returned.

Note: According to tensflow tutorial , the original paper , section 3.4 and 5.1, uses a single tokenizer and weight matrix for both the source and target languages. This tutorial uses two separate tokenizers and weight matrices.

Let’s create two instances of the PositionalEmbedding class, one for the Portuguese tokenizer and one for the English tokenizer. We pass the vocabulary size of each tokenizer and a value for d_model which is the dimensionality of the embedding vector.

Then we call these instances on our tokenized Portuguese and English sentences ( pt and en ), respectively. The output of each call is an embedded representation of the sentence, where each token is represented as a vector with a positional encoding added to it, as described in the PositionalEmbedding class.

The resulting embeddings can be used as input to the encoder and decoder of a Transformer model.

In Keras, masking is used to indicate timesteps that should be ignored during processing, for example, padding timesteps. The _keras_mask attribute returns a boolean tensor with the same shape as en_emb that indicates which timesteps should be masked (True for masked timesteps, False for unmasked timesteps). If a timestep is masked, it means that its corresponding values will be ignored during computation.”

2.4.5. Add and normalize

The “Add & Norm” blocks are used in the Transformer model and help with efficient training. These blocks consist of a residual connection, which provides a direct path for the gradient and ensures that vectors are updated instead of replaced by the attention layers, and a LayerNormalization layer that maintains a reasonable scale for the outputs. These blocks are scattered throughout the model, and the code is organized around them. Custom layer classes are defined for each block. The Add layer is used in the implementation to ensure that Keras masks are propagated since the + operator does not do that.

Note: In the case of residual addition, the original input to a layer is added to the output of that layer, creating a “residual” connection that allows the gradient to bypass the layer during backpropagation. This helps to prevent the gradients from vanishing and allows the weights to continue updating during training. In the case of the Transformer model, residual connections are used in combination with layer normalization to help with training efficiency and maintain a reasonable scale for the outputs.

2.4.6. Attention layer

The model includes Attention blocks, each containing a layers.MultiHeadAttention, a layers.LayerNormalization , and a layers.Add . To create these attention layers, we first define a base class that includes these three components, and then we create specific subclasses for each use case. Although it requires writing more code, this approach helps keep the implementation organized and easy to understand.

The class contains three layers, tf.keras.layers.MultiHeadAttention , tf.keras.layers.LayerNormalization , and tf.keras.layers.Add .

• The MultiHeadAttention layer is responsible for computing the attention weights between the input and output sequences.
• The LayerNormalization layer normalizes the activations of the layer across the batch and feature dimensions.
• The Add layer adds the output of the MultiHeadAttention layer to the original input sequence using a residual connection.

By creating a base class with these layers, we can reuse this code to create different attention mechanisms by inheriting from this class and defining the specific implementation details. This helps keep the code organized and clear.

How attention works?

In an attention layer, there are two inputs the query sequence and the context sequence . The query sequence is the sequence being processed, while the context sequence is the sequence being attended to. The output has the same shape as the query sequence.

The operation of an attention layer is often compared to that of a dictionary lookup, but with fuzzy, differentiable, and vectorized characteristics. Just like a dictionary lookup, a query is used to search for relevant information, which is represented as keys and values . When searching for a query in a regular dictionary, the matching key and its corresponding value are returned. However, in a fuzzy dictionary, a query does not need to match perfectly with a key for the value to be returned.

For example, if we searched for the key “species” in the dictionary {'color': 'blue', 'age': 22, 'type': 'pickup'} , it might return the value “pickup” as the best match for the query.

An attention layer works similarly to a fuzzy dictionary lookup, but instead of returning a single value, it combines multiple values based on how well they match with the query. The query, key, and value in an attention layer are each represented as vectors. Instead of using hash lookup, the attention layer combines the query and key vectors to determine how well they match, which is known as the attention score. The values are then combined by taking the weighted average of all values, where the weights are determined by the attention scores.

In the context of NLP, the query sequence can provide a query vector at each location, while the context sequence serves as the dictionary, with a key and value vector at each location. Before using the input vectors, the layers.MultiHeadAttention layer includes layers.Dense layers to project the input vectors.

So now let’s use this class to create other attention layers. We will create:

• The cross attention layer: Decoder-encoder attention
• The global self attention layer: Encoder self-attention
• The causal self attention layer: Decoder self-attention

2.4.6.1. The cross attention layer: Decoder-encoder attention

Let’s write CrossAttention class by inheriting it from the BaseAttention class, which contains a multi-head attention layer, a layer normalization layer, and an add layer.

The call method takes two input arguments, x and context . x is the query sequence, which is being processed and doing the attending, and context is the context sequence, which is being attended to.

The call method passes x and context to the self.mha (multi-head attention) layer, which returns an attention output tensor and attention scores tensor. The self.last_attn_scores attribute is set to the attention scores tensor for plotting later.

Next, the attention output tensor is added to the original x tensor using the self.add layer, and the result is normalized using the self.layernorm layer. The final output is then returned.

The output length is the length of the query sequence, and not the length of the context key/value sequence.

2.4.6.2. The global self attention layer: Encoder self-attention

This layer is responsible for processing the context sequence, and propagating information along its length. Now let’s write GlobalSelfAttention by inheriting from baseAttention layer.

In GlobalSelfAttention , there is only one input x , which is a sequence of vectors that represents the sequence being processed. This input is used as the query, key and value input for the multi-head attention (MHA) mechanism. The MHA computes a weighted average of the values based on how well the query matches the keys, where the attention scores determine the weight of each value.

In other words, the MHA learns to selectively focus on different parts of the input sequence, which can help the model capture relevant information for a particular task. In GlobalSelfAttention , since the input sequence is used for both query and key , it captures the relationship between each position and all the other positions in the sequence.

Finally, the output of the MHA is added back to the original input, followed by layer normalization, to obtain the final output of the attention layer. The normalization helps to stabilize the training process and improves the performance of the model.

Output tensor has the same shape as the input

2.4.6.3. The causal self attention layer: Decoder self-attention

This layer does a similar job as the global self attention layer, for the output sequence. Now let’s write CausalSelfAttention by inheriting from baseAttention layer.

The CausalSelfAttention class is a type of self-attention layer used in neural networks for sequence modeling tasks where the output at each time step can only depend on previous time steps, and not on future time steps. In such tasks, the causal self-attention layer is used to enforce the constraint that the model can only attend to the previous time steps during the decoding process.

The call method of this class takes a tensor x as input, and applies the causal self-attention mechanism to it. Specifically, the method uses the mha method (multi-head attention) of the BaseAttention class with the query , key , and value inputs set to x . Additionally, the use_causal_mask argument of the mha method is set to True , which applies a causal mask to the attention scores to ensure that the model can only attend to previous time steps .

After applying the causal self-attention mechanism, the method adds the output to the original input tensor x , and normalizes the result using layer normalization. Finally, the normalized tensor is returned as the output of the method.

The output for early sequence elements doesn’t depend on later elements, so it shouldn’t matter if you trim elements before or after applying the layer:

basiclly the difference between before and after triming tf.reduce_max(abs(out1 - out2)).numpy() is zero!

2.4.7. The feed forward network

Now let’s implement feedforward network.

The FeedForward class is a custom layer in TensorFlow that implements a feedforward neural network. It is commonly used in transformer-based models like BERT and GPT-2 to process each token’s representation.

The layer takes as input a tensor x with shape (batch_size, seq_len, d_model) , where d_model is the size of the last dimension. It passes the tensor x through a feedforward network consisting of two dense layers with dff hidden units and a relu activation function. A dropout_rate is also applied after the first dense layer to prevent overfitting. The output of the feedforward network is added to the original input x via the Add() layer. Finally, the output is normalized using the LayerNormalization() layer.

The FeedForward layer can learn a more complex function than a simple linear layer, which makes it useful for modeling non-linear relationships between the input and output.

Test the layer, the output is the same shape as the input:

2.4.8. The encoder

The encoder consists of a PositionalEmbedding layer at the input and a stack of EncoderLayer layers. Where each EncoderLayer contains a GlobalSelfAttention and FeedForward layer.

Let’s first write class for EncoderLayer and out togheter GlobalSelfAttention and FeedForward , then use stack of EncoderLayer and PositionalEmbedding to build Encoder .

The EncoderLayer class represents a single layer in the transformer encoder stack. It consists of two sub-layers: a self-attention layer and a feedforward neural network layer.

The __init__ function initializes the ncoderLayer object by creating its sub-layers. The self_attention layer is an instance of the GlobalSelfAttention class, which performs self-attention over the input sequence. The num_heads and key_dim parameters determine the number of attention heads and the dimensionality of the keys and values in each head, respectively. The dropout_rate parameter specifies the dropout rate to be applied after the self-attention sub-layer. The ffn sub-layer is an instance of the FeedForward class, which consists of two dense layers with ReLU activation, followed by a dropout layer.

The call function is called to apply the forward pass of the EncoderLayer . The input sequence x is passed through the self_attention sub-layer, followed by the ffn sub-layer, and the resulting output sequence is returned.

This code defines the Encoder class that is used in the Transformer architecture for natural language processing tasks such as language translation and language modeling.

The Encoder class is a subclass of the tf.keras.layers.Layer class, which is a base class for implementing new layers in Keras.

The __init__ method initializes the Encoder object by defining the model parameters such as d_model (the size of the output space), num_heads (the number of heads in the multi-head attention mechanism), dff (the dimension of the feedforward network), vocab_size (the size of the vocabulary of input tokens), and dropout_rate (the rate of dropout to be applied to the outputs of the layer).

The pos_embedding attribute initializes a PositionalEmbedding layer that adds positional information to the input tokens to take into account their position in the sequence.

The enc_layers attribute initializes a list of EncoderLayer objects, which each implement the EncoderLayer functionality. The number of layers in the encoder is determined by the num_layers parameter.

The dropout attribute initializes a dropout layer to apply dropout to the output of the layer.

The call method is called when the layer is called on input data. It applies the positional embedding to the input tokens and then applies the dropout layer. It then iteratively applies the EncoderLayer object to the output of the previous layer. The final output is returned as a tensor of shape (batch_size, seq_len, d_model) .

Let’s test encoder:

2.4.9. The decoder

Similar to the Encoder , the Decoder consists of a PositionalEmbedding , and a stack of DecoderLayer . And the decoder’s stack is slightly more complex, with each DecoderLayer containing a CausalSelfAttention , a CrossAttention , and a FeedForward layer.

Let’s first write DecoderLayer then write Encoder .

Let’s define DecoderLayer class which is a building block for a transformer-based decoder in a sequence-to-sequence model. The class inherits from tf.keras.layers.Layer .

The class has an __init__ method that initializes the layer’s parameters and sub-layers. It takes the following arguments:

• d_model : The number of expected features in the input and output.
• num_heads : The number of parallel attention heads.
• dff : The number of neurons in the feedforward sub-layer.
• dropout_rate : The dropout rate to be applied.

The call method defines how to use the layer in the forward pass. It takes two arguments: x and context . x is the input to the decoder layer, which is passed through causal self-attention, cross-attention, and feedforward sub-layers to produce the output x . context is the output from the encoder layer which is used as the attention context for the cross-attention mechanism.

The class contains the following sub-layers:

• causal_self_attention : The causal self-attention layer that attends to the input sequence in a causal manner, i.e., predicting future tokens based on the previous ones.
• cross_attention : The cross-attention layer that attends to the encoder output context to align the decoder output with the input sequence.
• ffn : A feedforward sub-layer that applies a non-linear transformation to the output of the attention sub-layers.

The call method also caches the last attention scores computed by the cross_attention sub-layer, which can be used for visualization and debugging purposes.

Now let’s put this inside Decoder and write decoder class. This class is responsible for decoding the encoded input sequences to generate the target sequences in sequence-to-sequence models. The Decoder layer consists of multiple DecoderLayer blocks, with each block containing a self-attention mechanism, a cross-attention mechanism, and a feedforward network. The Decoder layer also includes positional embedding and dropout layers.

The Decoder class expects as input a sequence of token-IDs representing the target sequence, and the encoded input sequence, or context, to which the decoder should attend to. The class consists of a stack of DecoderLayer instances, each of which applies a series of operations on the input sequence to generate an output sequence.

In the constructor, the Decoder class initializes several layer instances, including a PositionalEmbedding layer, which adds positional encoding to the input token-IDs, a dropout layer, and a stack of DecoderLayer instances.

During a forward pass, the input token-IDs are first passed through the positional embedding and dropout layers. Then, for each DecoderLayer , the input is passed through a causal self-attention layer, followed by a cross-attention layer, and finally through a feed-forward neural network layer. The output of the last DecoderLayer is returned as the output of the Decoder .

The last_attn_scores attribute of the Decoder instance contains the attention scores from the last decoder layer, which can be useful for visualization and debuggin

2.5. The Transformer

The Encoder and Decoder are the key components of the Transformer model, but they need to be combined and followed by a final Dense layer to output token probabilities. Now let’s put toghther these two classes and create the Transformer by extending tf.keras.Model :

The Transformer class is a Keras model that combines the Encoder and Decoder to implement the Transformer architecture.

The Encoder is an instance of the Encoder class that takes a sequence of tokens as input and outputs a sequence of vectors that represent the contextual information for each token in the sequence.

The Decoder is an instance of the Decoder class that takes a sequence of target tokens and the contextual information from the Encoder as input and outputs a sequence of vectors that represent the contextual information for each target token in the sequence.

The final_layer is a Keras Dense layer that takes the output of the Decoder and maps it to a sequence of target token probabilities.

The call method of the Transformer class takes an input tensor inputs , which is a tuple of two tensors: the context tensor, which is the input sequence to the Encoder , and the x tensor, which is the target sequence to the Decoder . The method passes the context tensor through the Encoder to obtain the contextual information for each token in the sequence, and then passes the x tensor and the Encoder output to the Decoder to generate the output sequence. Finally, the method passes the output of the Decoder through the final_layer to obtain the target token probabilities. The method returns the logits , which are the target token probabilities, as well as the attention weights.

In order to maintain a relatively quick and compact example, the size of the layers, embeddings, and internal dimensionality of the FeedForward layer in the Transformer model have been decreased. The original Transformer paper utilized a base model with num_layers=6 , d_model=512 , and dff=2048 . However, the number of self-attention heads in this example remains the same, set at num_heads=8.

Now let’s instantiate the Transformer model:

We can test the model before move forward to training part:

Let’s print summary of model to visualize it better.

The total number of trainable parameters in this model is 10,184,162. There are no non-trainable parameters.

2.6. Training

2.6.1. optimizer.

Use the Adam optimizer with a custom learning rate scheduler according to the formula in the original Transformer paper .

The CustomSchedule class is a subclass of tf.keras.optimizers.schedules.LearningRateSchedule . It takes in two arguments d_model and warmup_steps . The d_model represents the dimensionality of the model, which is cast into a float32 . The warmup_steps is the number of steps to increase the learning rate linearly before decaying it.

The __call__ method takes in a single argument step , which represents the current training step. It casts the step into a float32 and calculates two arguments arg1 and arg2 . arg1 is calculated by taking the reciprocal square root of the step . arg2 is calculated by multiplying the step with the reciprocal square root of the warmup_steps raised to the power of -1.5 .

Finally, the method returns the product of the reciprocal square root of d_model and the minimum of arg1 and arg2 . This method is used to define the learning rate schedule for the optimizer used in the training process of the Transformer model.

Now we can instantiate the optimizer (in this example it’s tf.keras.optimizers.Adam ):

Let’s see how that looks like

2.6.1. Loss

Let’s define loss and accuracy function. We are using cross-entropy loss function using tf.keras.losses.SparseCategoricalCrossentropy :

The masked_loss function calculates the masked sparse categorical cross-entropy loss between the predicted values and the true labels. In this function, the label and pred inputs are the true labels and predicted values, respectively.

First, the function creates a boolean mask to exclude padded values (0’s) from the loss calculation. Then, it defines the loss object as SparseCategoricalCrossentropy from tf.keras.losses , which computes the cross-entropy loss between the true and predicted labels.

The next step multiplies the loss by the mask to exclude any loss contribution from the padded values. The function then reduces the loss by computing the sum of the loss over the non-padded values and dividing it by the sum of the mask to obtain the average loss over non-padded values.

Now let’s write a function to calculate accuracy:

The masked_accuracy function computes the masked accuracy of the predicted values given the true labels. The inputs of the function are label and pred , which are the true labels and predicted values, respectively.

First, the function uses tf.argmax to find the index of the maximum value in pred along the last dimension, which represents the predicted class. Then, the true labels label are cast to the same data type as the predicted values pred.

The function then creates a boolean mask to exclude padded values from the calculation. The function uses the & operator to compare the predicted and true labels and create a boolean matrix match. The values in match indicate whether the predicted and true labels match for non-padded values.

The match matrix and the mask matrix are then cast to float32 and used to compute the average accuracy over non-padded values. The function returns the sum of match divided by the sum of mask.

2.6.3. Training

Now let’s compile the model and use model.fit to train model! I use Colab A100 GPU!

2.7. Testing

Now we trained our model for 20 Epochs, let’s try to use this model for translating! For this, let’s write Translator class.

Translator takes tokenizers and transformer as inputs in its constructor. It has a __call__ method that takes a sentence in Portuguese and translates it to English using the transformer model.

The input sentence is first tokenized using the Portuguese tokenizer and converted to a tensor. The encoder input is set to be the tokenized sentence.

The English [START] token is added to the output to initialize it. The output is stored in a tf.TensorArray .

For each token in the output, the transformer model is called with the encoder input and the current output. The last token from the seq_len dimension of the predictions is selected and appended to the output. If the last token is the [END] token, the loop is terminated. The output is converted to text using the English tokenizer and returned along with the attention weights.

Create an instance of this Translator class, and try it out a few times:

Let’s wrrite a function to translate sentences for us:

We can also write a function to return attention wights. Let’s write that!

plot_attention_weights is a function that plots the attention weights of all the attention heads in a multi-head attention mechanism. It takes in the input sentence, the translated output tokens, and the attention weights and creates a figure where each subplot represents one attention head.

plot_attention_head is a helper function used by plot_attention_weights . It takes in the input tokens, the output tokens, and the attention matrix of a single attention head and plots a heatmap where the x-axis represents the input tokens and the y-axis represents the output tokens. It is called once for each attention head to create the subplots in the plot_attention_weights figure.

now let’s put togheter some sentences to test:

2.8. Export the model

For this, let’s create class called ExportTranslator , then wrap translator in the ExportTranslator:

ExportTranslator class takes a translator object as input and exports it as a TensorFlow module. It has a __call__ method that takes a single string argument sentence and returns the translation result of the translator object for that input sentence.

The __call__ method is decorated with tf.function and input_signature that specify the data type and shape of the input tensor. The sentence tensor has an empty shape and string data type. The __call__ method calls the translator object with the input sentence and max_length argument set to MAX_TOKENS, and returns the translation result as a tensor.

Since the model is decoding the predictions using tf.argmax the predictions are deterministic. The original model and one reloaded from its SavedModel should give identical predictions:

Now we can use .save method to save model!

We can reload model and test the prediction!

2.9. Experimenting with the Model

Unfortunately, I don’t have access to a lot of computing power, but it would be ideal to train the model for more epochs. I have tested the model with more than 20 epochs, so let’s take a look at the results here for model with 50 more epochs and compare the results!

Let’s export model.

So it seems we have slightly better translation!

Final Note before goodbye: This tutorial is heavily based on https://www.tensorflow.org/text/tutorials/transformer . I tried to make the explanation clear and adapt it based on my student taking this series of courses! I highly recommend going through the main material!

[1] Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017).

[2] https://python.plainenglish.io/image-captioning-with-an-end-to-end-transformer-network-8f39e1438cd4

[3] https://www.tensorflow.org/text/tutorials/transformer

You May Also Enjoy

Mastering dynamic programming: 4 essential algorithms for optimal solutions.

19 minute read

Published: June 29, 2023

Dive into the world of dynamic programming with this insightful post. Explore four essential algorithms that leverage dynamic programming to solve complex problems optimally. Enhance your understanding of data structure and algorithms with this comprehensive guide.

Cracking the Code of Sorting and Searching: 11 Algorithms for Enhanced Efficiency

18 minute read

Published: June 21, 2023

Unravel the secrets of sorting and searching algorithms in this comprehensive post. Discover eleven algorithms that will equip you with the tools to sort and search data efficiently. Elevate your data structure and algorithms expertise with this invaluable resource. In this post, we will learn about sorting algorithms in Python. Let’s dive in! You can run this post in Google Colab using this link:

Mastering Heaps: 5 Essential Algorithms for Optimal Performance

27 minute read

Published: June 16, 2023

Immerse yourself in the world of heaps with this informative post. Explore five essential algorithms that leverage heaps to optimize performance and efficiency. Unlock the potential of heaps in your data structure and algorithms toolkit with this indispensable guide.

Mastering Linked Lists: 11 Algorithms for Efficient Data Manipulation

37 minute read

Published: June 13, 2023

Unlock the full potential of linked lists in this comprehensive post. Explore eleven powerful algorithms that will revolutionize the way you manipulate and operate on linked list data structures. Elevate your data structure and algorithms expertise with this transformative content. This post focuses on linked lists and demonstrates various operations that can be performed on linked lists using Python. A linked list is a fundamental data structure composed of nodes, where each node contains a value and a reference to the next node in the list. This notebook covers the basics of linked lists, including insertion, deletion, traversal, and searching operations. You can run this post in Google Colab using this link:

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

Programming Assignment: Transformers Architecture with TensorFlow: Transformer course 5 week 4 Deep learning speciality coursera

abdelkadergelany/Transformer-course-5-week-4-Deep-learning-speciality-by-coursera

Folders and files, repository files navigation, transformer-course-5-week-4-deep-learning-speciality-by-coursera (november 2022 ).

• Jupyter Notebook 86.7%
• Python 13.3%

Training a language model with 🤗 Transformers using TensorFlow and TPUs

*]:break-words" href="#introduction" title="Introduction"> Introduction *]:break-words" href="#motivation" title="Motivation"> Motivation *]:break-words" href="#what-to-expect" title="What to expect"> What to expect *]:break-words" href="#getting-the-data-and-training-a-tokenizer" title="Getting the data and training a tokenizer"> Getting the data and training a tokenizer *]:break-words" href="#tokenizing-the-data-and-creating-tfrecords" title="Tokenizing the data and creating TFRecords"> Tokenizing the data and creating TFRecords *]:break-words" href="#training-a-model-on-data-in-gcs" title="Training a model on data in GCS"> Training a model on data in GCS *]:break-words" href="#conclusion" title="Conclusion"> Conclusion Introduction

TPU training is a useful skill to have: TPU pods are high-performance and extremely scalable, making it easy to train models at any scale from a few tens of millions of parameters up to truly enormous sizes: Google’s PaLM model (over 500 billion parameters!) was trained entirely on TPU pods.

We’ve previously written a tutorial and a Colab example showing small-scale TPU training with TensorFlow and introducing the core concepts you need to understand to get your model working on TPU. This time, we’re going to step that up another level and train a masked language model from scratch using TensorFlow and TPU, including every step from training your tokenizer and preparing your dataset through to the final model training and uploading. This is the kind of task that you’ll probably want a dedicated TPU node (or VM) for, rather than just Colab, and so that’s where we’ll focus.

As in our Colab example, we’re taking advantage of TensorFlow's very clean TPU support via XLA and TPUStrategy . We’ll also be benefiting from the fact that the majority of the TensorFlow models in 🤗 Transformers are fully XLA-compatible . So surprisingly, little work is needed to get them to run on TPU.

Unlike our Colab example, however, this example is designed to be scalable and much closer to a realistic training run -- although we only use a BERT-sized model by default, the code could be expanded to a much larger model and a much more powerful TPU pod slice by changing a few configuration options.

Why are we writing this guide now? After all, 🤗 Transformers has had support for TensorFlow for several years now. But getting those models to train on TPUs has been a major pain point for the community. This is because:

• Many models weren’t XLA-compatible
• Data collators didn’t use native TF operations

We think XLA is the future: It’s the core compiler for JAX, it has first-class support in TensorFlow, and you can even use it from PyTorch . As such, we’ve made a big push to make our codebase XLA compatible and to remove any other roadblocks standing in the way of XLA and TPU compatibility. This means users should be able to train most of our TensorFlow models on TPUs without hassle.

There’s also another important reason to care about TPU training right now: Recent major advances in LLMs and generative AI have created huge public interest in model training, and so it’s become incredibly hard for most people to get access to state-of-the-art GPUs. Knowing how to train on TPU gives you another path to access ultra-high-performance compute hardware, which is much more dignified than losing a bidding war for the last H100 on eBay and then ugly crying at your desk. You deserve better. And speaking from experience: Once you get comfortable with training on TPU, you might not want to go back.

What to expect

We’re going to train a RoBERTa (base model) from scratch on the  WikiText dataset (v1) . As well as training the model, we’re also going to train the tokenizer, tokenize the data and upload it to Google Cloud Storage in TFRecord format, where it’ll be accessible for TPU training. You can find all the code in this directory . If you’re a certain kind of person, you can skip the rest of this blog post and just jump straight to the code. If you stick around, though, we’ll take a deeper look at some of the key ideas in the codebase.

Many of the ideas here were also mentioned in our Colab example , but we wanted to show users a full end-to-end example that puts it all together and shows it in action, rather than just covering concepts at a high level. The following diagram gives you a pictorial overview of the steps involved in training a language model with 🤗 Transformers using TensorFlow and TPUs:

Getting the data and training a tokenizer

As mentioned, we used the WikiText dataset (v1) . You can head over to the dataset page on the Hugging Face Hub to explore the dataset.

Since the dataset is already available on the Hub in a compatible format, we can easily load and interact with it using 🤗 datasets. However, for this example, since we’re also training a tokenizer from scratch, here’s what we did:

• Loaded the train split of the WikiText using 🤗 datasets.
• Leveraged 🤗 tokenizers to train a Unigram model .
• Uploaded the trained tokenizer on the Hub.

You can find the tokenizer training code here and the tokenizer here . This script also allows you to run it with any compatible dataset from the Hub.

💡 It’s easy to use 🤗 datasets to host your text datasets. Refer to this guide to learn more.

Tokenizing the data and creating TFRecords

Once the tokenizer is trained, we can use it on all the dataset splits ( train , validation , and test in this case) and create TFRecord shards out of them. Having the data splits spread across multiple TFRecord shards helps with massively parallel processing as opposed to having each split in single TFRecord files.

We tokenize the samples individually. We then take a batch of samples, concatenate them together, and split them into several chunks of a fixed size (128 in our case). We follow this strategy rather than tokenizing a batch of samples with a fixed length to avoid aggressively discarding text content (because of truncation).

We then take these tokenized samples in batches and serialize those batches as multiple TFRecord shards, where the total dataset length and individual shard size determine the number of shards. Finally, these shards are pushed to a Google Cloud Storage (GCS) bucket .

If you’re using a TPU node for training, then the data needs to be streamed from a GCS bucket since the node host memory is very small. But for TPU VMs, we can use datasets locally or even attach persistent storage to those VMs. Since TPU nodes are still quite heavily used, we based our example on using a GCS bucket for data storage.

You can see all of this in code in this script . For convenience, we have also hosted the resultant TFRecord shards in this repository on the Hub.

Training a model on data in GCS

If you’re familiar with using 🤗 Transformers, then you already know the modeling code:

But since we’re in the TPU territory, we need to perform this initialization under a strategy scope so that it can be distributed across the TPU workers with data-parallel training:

Similarly, the optimizer also needs to be initialized under the same strategy scope with which the model is going to be further compiled. Going over the full training code isn’t something we want to do in this post, so we welcome you to read it here . Instead, let’s discuss another key point of — a TensorFlow-native data collator — DataCollatorForLanguageModeling .

DataCollatorForLanguageModeling is responsible for masking randomly selected tokens from the input sequence and preparing the labels. By default, we return the results from these collators as NumPy arrays. However, many collators also support returning these values as TensorFlow tensors if we specify return_tensor="tf" . This was crucial for our data pipeline to be compatible with TPU training.

Thankfully, TensorFlow provides seamless support for reading files from a GCS bucket:

If args.dataset contains the gs:// identifier, TensorFlow will understand that it needs to look into a GCS bucket. Loading locally is as easy as removing the gs:// identifier. For the rest of the data pipeline-related code, you can refer to this section in the training script.

Once the datasets have been prepared, the model and the optimizer have been initialized, and the model has been compiled, we can do the community’s favorite - model.fit() . For training, we didn’t do extensive hyperparameter tuning. We just trained it for longer with a learning rate of 1e-4. We also leveraged the PushToHubCallback for model checkpointing and syncing them with the Hub. You can find the hyperparameter details and a trained model here: https://huggingface.co/tf-tpu/roberta-base-epochs-500-no-wd .

Once the model is trained, running inference with it is as easy as:

If there’s one thing we want to emphasize with this example, it’s that TPU training is powerful, scalable and easy. In fact, if you’re already using Transformers models with TF/Keras and streaming data from tf.data , you might be shocked at how little work it takes to move your whole training pipeline to TPU. They have a reputation as somewhat arcane, high-end, complex hardware, but they’re quite approachable, and instantiating a large pod slice is definitely easier than keeping multiple GPU servers in sync!

Diversifying the hardware that state-of-the-art models are trained on is going to be critical in the 2020s, especially if the ongoing GPU shortage continues. We hope that this guide will give you the tools you need to power cutting-edge training runs no matter what circumstances you face.

As the great poet GPT-4 once said:

If you can keep your head when all around you Are losing theirs to GPU droughts, And trust your code, while others doubt you, To train on TPUs, no second thoughts;

If you can learn from errors, and proceed, And optimize your aim to reach the sky, Yours is the path to AI mastery, And you'll prevail, my friend, as time goes by.

Sure, it’s shamelessly ripping off Rudyard Kipling and it has no idea how to pronounce “drought”, but we hope you feel inspired regardless.

More Articles from our Blog

Welcome Llama 3 - Meta's new open LLM

By  philschmid April 18, 2024 • 221

Introducing Idefics2: A Powerful 8B Vision-Language Model for the community

By  Leyo April 15, 2024 • 87

tf-transformers 2.0.0

pip install tf-transformers Copy PIP instructions

Released: Apr 8, 2022

NLP with Transformer based models on Tensorflow 2.0

Verified details

Maintainers.

Unverified details

Github statistics.

View statistics for this project via Libraries.io , or by using our public dataset on Google BigQuery

License: MIT License (MIT)

Author: Sarath R Nair

Maintainer: Sarath R Nair

Tags tensorflow, transformers, nlp, keras, bert, deep learning

Requires: Python >=3.7, <4.0

Classifiers

• OSI Approved :: MIT License
• Python :: 3
• Python :: 3.7
• Python :: 3.8
• Python :: 3.9
• Python :: 3.10

Project description

Tensorflow Transformers

Website: https://legacyai.github.io/tf-transformers, tf-transformers: faster and easier state-of-the-art transformer in tensorflow 2.0.

Imagine auto-regressive generation to be 90x faster. tf-transformers (Tensorflow Transformers) is designed to harness the full power of Tensorflow 2, designed specifically for Transformer based architecture.

These models can be applied on:

• 📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.
• 🖼️ Images, for tasks like image classification, object detection, and segmentation.
• 🗣️ Audio, for tasks like speech recognition and audio classification. (Coming Soon)

Unique Features

• Faster AutoReggressive Decoding
• TFlite support
• Creating TFRecords is simple .
• Auto-Batching tf.data.dataset or tf.ragged tensors
• Everything is dictionary (inputs and outputs)
• Multiple mask modes like causal , user-defined , prefix .
• tensorflow-text tokenizer support
• Supports GPU, TPU, multi-GPU trainer with wandb, multiple callbacks, auto tensorboard

Benchmark on GPT2 text generation

GPT2 text generation with max_length=64 , num_beams=3 .

From 83 minutes to 31 minutes is a significant speedup. 167 %1 speedup. On an average, tf-transformers is 80-90 times faster than HuggingFace Tensorflow implementation and in most cases it is comparable or faster than PyTorch .

More benchmarks can be found in benchmark

Installation

This repository is tested on Python 3.7+ and TensorFlow 2.7.

Recommended prerequistes

Install tensorflow >= 2.7.0 [CPU or GPU] as per your machine. You should install tf-transformers in a virtual environment . If you're unfamiliar with Python virtual environments, check out the user guide .

First, create a virtual environment with the version of Python you're going to use and activate it.

Then, you will need to install at least one of TensorFlow. Please refer to TensorFlow installation page , installation pages regarding the specific install command for your platform. We highly recommend to install [tensorflow-text] ( https://www.tensorflow.org/text ).

When one of those backends has been installed, tf-transformers can be installed using pip as follows:

From source

tf-transformers API is very simple and minimalistic.

For text-generation, it is very important to add :obj: use_auto_regressive=True . This is required for all the models.

To serialize save and load model

We have covered tutorials covering pre-training, finetuning, classfication, QA, NER so much more.

• Read and Write TFRecords using tft
• Text Classification using Albert
• Dynamic MLM (on the fly pre-processing using tf-text) in TPU
• Image Classification ViT multi GPU mirrored
• Sentence Embedding train from scratch using Quoara on Roberta + Zeroshot STS-B
• Prompt Engineering using CLIP
• Question Answering as Generation - Squad v1 using GPT2
• Code to Code translation (CodexGLUE - Java to C#) using T5

Model usage

• Text Generation using GPT2
• Text Generation using T5
• Sentence Transformers

TFlite Tutorials

• Albert TFlite
• Bert TFlite
• Roberta TFlite

Why should I use tf-transformers?

Use state-of-the-art models in Production, with less than 10 lines of code.

• High performance models, better than all official Tensorflow based models
• Very simple classes for all downstream tasks
• Complete TFlite support for all tasks.

Make industry based experience to avaliable to students and community with clear tutorials

Train any model on GPU , multi-GPU , TPU with amazing tf.keras.Model.fit

• Train state-of-the-art models in few lines of code.
• All models are completely serializable.

Customize any models or pipelines with minimal or no code change.

The Research section has codes for pre-training different models ranging from **MLM, T5, CLIP etc **. All these scripts are designed to harness full power of tensorflow-io pipeline and tested on TPU V2 and TPU V3. Bugs are expected in those, but it serves as a purpose for practioners to start or modifying what we have already done.

Contributions

Joint albert (smallest and best transformer based model ever) on glue ..

We have conducted few experiments to squeeze the power of Albert base models ( concept is applicable to any models and in tf-transformers, it is out of the box.)

The idea is minimize the loss for specified task in each layer of your model and check predictions at each layer. as per our experiments, we are able to get the best smaller model (thanks to Albert ), and from layer 4 onwards we beat all the smaller model in GLUE benchmark. By layer 6 , we got a GLUE score of 81.0 , which is 4 points ahead of Distillbert with GLUE score of 77 and MobileBert GLUE score of 78 .

The Albert model has 14 million parameters, and by using layer 6 , we were able to speed up the compuation by 50% .

The concept is applicable to all the models and tasks.

Codes + Read More

Long Block Sequence Transformer

By splitting input sequence into block attention and merge using FFN layer we have shown that, smaller machines will be able to perform sequence processing up to 4096 tokens in a single V100 GPU machine. The model has outperforms Pegasus Base (128 million) in PubMed summarisation despite being 60 million parameter.

Supported Models architectures

tf-transformers currently provides the following architectures .

• ALBERT (from Google Research and the Toyota Technological Institute at Chicago) released with the paper ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
• BERT (from Google) released with the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
• BERT For Sequence Generation (from Google) released with the paper Leveraging Pre-trained Checkpoints for Sequence Generation Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.
• ELECTRA (from Google Research/Stanford University) released with the paper ELECTRA: Pre-training text encoders as discriminators rather than generators by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
• GPT-2 (from OpenAI) released with the paper Language Models are Unsupervised Multitask Learners by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
• MT5 (from Google AI) released with the paper mT5: A massively multilingual pre-trained text-to-text transformer by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel.
• RoBERTa (from Facebook), released together with the paper a Robustly Optimized BERT Pretraining Approach by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
• T5 (from Google AI) released with the paper Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
• Vision Transformer (ViT) (from Google AI) released with the paper An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. 10 CLIP (from OpenAI) released with the paper Learning Transferable Visual Models From Natural Language Supervision by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever.

We now have a page you can cite for the tf-transformers library.

Project details

Release history release notifications | rss feed.

Apr 8, 2022

Apr 3, 2022

Apr 2, 2022

Mar 10, 2022

Jan 11, 2022

Mar 21, 2021

Mar 15, 2021

Mar 14, 2021

Jan 8, 2021

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages .

Source Distribution

Uploaded Apr 8, 2022 Source

Built Distribution

Uploaded Apr 8, 2022 Python 3

Hashes for tf-transformers-2.0.0.tar.gz

Hashes for tf_transformers-2.0.0-py3-none-any.whl.

• português (Brasil)

Supported by

• Español – América Latina
• Português – Brasil
• Tiếng Việt

Neural machine translation with attention

This tutorial demonstrates how to train a sequence-to-sequence (seq2seq) model for Spanish-to-English translation roughly based on Effective Approaches to Attention-based Neural Machine Translation (Luong et al., 2015).

While this architecture is somewhat outdated, it is still a very useful project to work through to get a deeper understanding of sequence-to-sequence models and attention mechanisms (before going on to Transformers ).

This example assumes some knowledge of TensorFlow fundamentals below the level of a Keras layer:

• Working with tensors directly
• Writing custom keras.Model s and keras.layers

After training the model in this notebook, you will be able to input a Spanish sentence, such as " ¿todavia estan en casa? ", and return the English translation: " are you still at home? "

The resulting model is exportable as a tf.saved_model , so it can be used in other TensorFlow environments.

The translation quality is reasonable for a toy example, but the generated attention plot is perhaps more interesting. This shows which parts of the input sentence has the model's attention while translating:

This tutorial uses a lot of low level API's where it's easy to get shapes wrong. This class is used to check shapes throughout the tutorial.

Toggle code

The tutorial uses a language dataset provided by Anki . This dataset contains language translation pairs in the format:

They have a variety of languages available, but this example uses the English-Spanish dataset.

Download and prepare the dataset

For convenience, a copy of this dataset is hosted on Google Cloud, but you can also download your own copy. After downloading the dataset, here are the steps you need to take to prepare the data:

• Add a start and end token to each sentence.
• Clean the sentences by removing special characters.
• Create a word index and reverse word index (dictionaries mapping from word → id and id → word).
• Pad each sentence to a maximum length.

Create a tf.data dataset

From these arrays of strings you can create a tf.data.Dataset of strings that shuffles and batches them efficiently:

Text preprocessing

One of the goals of this tutorial is to build a model that can be exported as a tf.saved_model . To make that exported model useful it should take tf.string inputs, and return tf.string outputs: All the text processing happens inside the model. Mainly using a layers.TextVectorization layer.

Standardization

The model is dealing with multilingual text with a limited vocabulary. So it will be important to standardize the input text.

The first step is Unicode normalization to split accented characters and replace compatibility characters with their ASCII equivalents.

The tensorflow_text package contains a unicode normalize operation:

Unicode normalization will be the first step in the text standardization function:

Text Vectorization

This standardization function will be wrapped up in a tf.keras.layers.TextVectorization layer which will handle the vocabulary extraction and conversion of input text to sequences of tokens.

The TextVectorization layer and many other Keras preprocessing layers have an adapt method. This method reads one epoch of the training data, and works a lot like Model.fit . This adapt method initializes the layer based on the data. Here it determines the vocabulary:

That's the Spanish TextVectorization layer, now build and .adapt() the English one:

Now these layers can convert a batch of strings into a batch of token IDs:

The get_vocabulary method can be used to convert token IDs back to text:

The returned token IDs are zero-padded. This can easily be turned into a mask:

Process the dataset

The process_text function below converts the Datasets of strings, into 0-padded tensors of token IDs. It also converts from a (context, target) pair to an ((context, target_in), target_out) pair for training with keras.Model.fit . Keras expects (inputs, labels) pairs, the inputs are the (context, target_in) and the labels are target_out . The difference between target_in and target_out is that they are shifted by one step relative to eachother, so that at each location the label is the next token.

Here is the first sequence of each, from the first batch:

The encoder/decoder

The following diagrams shows an overview of the model. In both the encoder is on the left, the decoder is on the right. At each time-step the decoder's output is combined with the encoder's output, to predict the next word.

The original [left] contains a few extra connections that are intentionally omitted from this tutorial's model [right], as they are generally unnecessary, and difficult to implement. Those missing connections are:

• Feeding the state from the encoder's RNN to the decoder's RNN
• Feeding the attention output back to the RNN's input.

Before getting into it define constants for the model:

The encoder

The goal of the encoder is to process the context sequence into a sequence of vectors that are useful for the decoder as it attempts to predict the next output for each timestep. Since the context sequence is constant, there is no restriction on how information can flow in the encoder, so use a bidirectional-RNN to do the processing:

The encoder:

• Takes a list of token IDs (from context_text_processor ).
• Looks up an embedding vector for each token (Using a layers.Embedding ).
• Processes the embeddings into a new sequence (Using a bidirectional layers.GRU ).
• Returns the processed sequence. This will be passed to the attention head.

Try it out:

The attention layer

The attention layer lets the decoder access the information extracted by the encoder. It computes a vector from the entire context sequence, and adds that to the decoder's output.

The simplest way you could calculate a single vector from the entire sequence would be to take the average across the sequence ( layers.GlobalAveragePooling1D ). An attention layer is similar, but calculates a weighted average across the context sequence. Where the weights are calculated from the combination of context and "query" vectors.

The attention weights will sum to 1 over the context sequence, at each location in the target sequence.

Here are the attention weights across the context sequences at t=0 :

Because of the small-random initialization the attention weights are initially all close to 1/(sequence_length) . The model will learn to make these less uniform as training progresses.

The decoder

The decoder's job is to generate predictions for the next token at each location in the target sequence.

• It looks up embeddings for each token in the target sequence.
• It uses an RNN to process the target sequence, and keep track of what it has generated so far.
• It uses RNN output as the "query" to the attention layer, when attending to the encoder's output.
• At each location in the output it predicts the next token.

When training, the model predicts the next word at each location. So it's important that the information only flows in one direction through the model. The decoder uses a unidirectional (not bidirectional) RNN to process the target sequence.

When running inference with this model it produces one word at a time, and those are fed back into the model.

Here is the Decoder class' initializer. The initializer creates all the necessary layers.

Next, the call method, takes 3 arguments:

• context - is the context from the encoder's output.
• x - is the target sequence input.
• state - Optional, the previous state output from the decoder (the internal state of the decoder's RNN). Pass the state from a previous run to continue generating text where you left off.
• return_state - [Default: False] - Set this to True to return the RNN state.

That will be sufficient for training. Create an instance of the decoder to test out:

In training you'll use the decoder like this:

Given the context and target tokens, for each target token it predicts the next target token.

To use it for inference you'll need a couple more methods:

With those extra functions, you can write a generation loop:

Since the model's untrained, it outputs items from the vocabulary almost uniformly at random.

Now that you have all the model components, combine them to build the model for training:

During training the model will be used like this:

For training, you'll want to implement your own masked loss and accuracy functions:

Configure the model for training:

The model is randomly initialized, and should give roughly uniform output probabilities. So it's easy to predict what the initial values of the metrics should be:

That should roughly match the values returned by running a few steps of evaluation:

Now that the model is trained, implement a function to execute the full text => text translation. This code is basically identical to the inference example in the decoder section , but this also captures the attention weights.

Here are the two helper methods, used above, to convert tokens to text, and to get the next token:

Use that to generate the attention plot:

Translate a few more sentences and plot them:

The short sentences often work well, but if the input is too long the model literally loses focus and stops providing reasonable predictions. There are two main reasons for this:

• The model was trained with teacher-forcing feeding the correct token at each step, regardless of the model's predictions. The model could be made more robust if it were sometimes fed its own predictions.
• The model only has access to its previous output through the RNN state. If the RNN state looses track of where it was in the context sequence there's no way for the model to recover. Transformers improve on this by letting the decoder look at what it has output so far.

The raw data is sorted by length, so try translating the longest sequence:

The translate function works on batches, so if you have multiple texts to translate you can pass them all at once, which is much more efficient than translating them one at a time:

So overall this text generation function mostly gets the job done, but so you've only used it here in python with eager execution. Let's try to export it next:

If you want to export this model you'll need to wrap the translate method in a tf.function . That implementation will get the job done:

Run the tf.function once to compile it:

Now that the function has been traced it can be exported using saved_model.save :

[Optional] Use a dynamic loop

It's worth noting that this initial implementation is not optimal. It uses a python loop:

The python loop is relatively simple but when tf.function converts this to a graph, it statically unrolls that loop. Unrolling the loop has two disadvantages:

• It makes max_length copies of the loop body. So the generated graphs take longer to build, save and load.
• You have to choose a fixed value for the max_length .
• You can't break from a statically unrolled loop. The tf.function version will run the full max_length iterations on every call. That's why the break only works with eager execution. This is still marginally faster than eager execution, but not as fast as it could be.

To fix these shortcomings, the translate_dynamic method, below, uses a tensorflow loop:

It looks like a python loop, but when you use a tensor as the input to a for loop (or the condition of a while loop) tf.function converts it to a dynamic loop using operations like tf.while_loop .

There's no need for a max_length here it's just in case the model gets stuck generating a loop like: the united states of the united states of the united states... .

On the down side, to accumulate tokens from this dynamic loop you can't just append them to a python list , you need to use a tf.TensorArray :

This version of the code can be quite a bit more efficient:

With eager execution this implementation performs on par with the original:

But when you wrap it in a tf.function you'll notice two differences.

First, it's much quicker to trace, since it only creates one copy of the loop body:

The tf.function is much faster than running with eager execution, and on small inputs it's often several times faster than the unrolled version, because it can break out of the loop.

So save this version as well:

• Download a different dataset to experiment with translations, for example, English to German, or English to French.
• Experiment with training on a larger dataset, or using more epochs.
• Try the transformer tutorial which implements a similar translation task but uses transformer layers instead of RNNs. This version also uses a text.BertTokenizer to implement word-piece tokenization.
• Visit the tensorflow_addons.seq2seq tutorial , which demonstrates a higher-level functionality for implementing this sort of sequence-to-sequence model, such as seq2seq.BeamSearchDecoder .

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2023-12-07 UTC.

course-deep-learning

Deep learning: northwestern university cs 396/496 spring 2024, class day/time.

Tuesdays and Thursdays, 9:30am - 10:50am Central Time

Tech Lecture Room 5

Instructors

Professor: Bryan Pardo

TAs: Hugo Flores Garcia, Weijan Li

Peer Mentors: Conor Kotwasinski, Cameron Churchwell, Nathan Pruyne, Finn Wintz, Ben Ferreira

Office hours

Monday: Weijan Li 3-5pm on Weijan’s zoom link , Conor Kotwasinski 5-6pm in Mudd 3532

Tuesday: Hugo Flores Garcia 1-2pm in Mudd 3532, Cameron Churchwell 1-2pm on Cameron’s zoom link , Bryan Pardo 3-5pm in Mudd 3115

Wednesday: Cameron Churchwell 9-10am in Mudd 3532, Ben Ferreira 1-3pm on Ben’s zoom link , Conor Kotwasinsky 3 - 4pm in Mudd 3108, Finn Wintz 4-5pm on Finn’s zoom link

Thursday: Finn Wintz 4-5pm on Finn’s zoom link , Nathan Pruyne 6pm - 8pm in Mudd 3532

Course Description

This is a first course in Deep Learning. We will study deep learning architectures: perceptrons, multi-layer perceptrons, convolutional networks, recurrent neural networks (LSTMs, GRUs), attention networks, transformers, autoencoders, and the combination of reinforcement learning with deep learning. Other covered topics include regularization, loss functions and gradient descent.

Learning will be in the practical context of implementing networks using these architectures in a modern programming environment: Pytorch. Homework consists of a mixture of programming assignments, review of research papers, running experiments with deep learning architectures, and theoretical questions about deep learning.

Students completing this course should be able to reason about deep network architectures, build a deep network from scratch in Python, modify existing deep networks, train networks, and evaluate their performance. Students completing the course should also be able to understand current research in deep networks.

Course Prerequisites

This course presumes prior knowledge of machine learning equivalent to having taken CS 349 Machine Learning.

Course textbook

The primary text is the Deep Learning book . This reading will be supplemented by reading key papers in the field.

Course Policies

Questions outside of class.

Please use CampusWire for class-related questions.

Submitting assignments

Assignments must be submitted on the due date by the time specified on Canvas. If you are worried you can’t finish on time, upload a safety submission an hour early with what you have. I will grade the most recent item submitted before the deadline. Late submissions will not be graded.

Grading Policy

You will be graded on a 100 point scale (e.g. 93 to 100 = A, 90-92 = A-, 87-89 = B+, 83-86 = B, 80-82 = B-…and so on).

Homework and reading assignments are solo assignments and must be original work.

Extra Credit

You can earn up to 8 points of extra credit in the final reading example

Course Calendar

Back to top

Helpful Programming Packages

Anaconda is the most popular python distro for machine learning.

Pytorch Facebook’s popular deep learning package. My lab uses this. Tensorboard is what my lab uses to visualize how experiments are going.

Tensorflow is Google’s most popular python DNN package

Keras A nice programming API that works with Tensorflow

JAX Is an alpha package from Gogle that allows differentiation of numpy and also an optimizing compiler for working on tensor processing units

Trax Is Google Brain’s DNN package. It focuses on transformers and is implemented on top of JAX

MXNET is Apache’s open source DL package.

Helpful Books on Deep Learning

Deep Learning is THE book on Deep Learning. One of the authors won the Turing prize due to his work on deep learning.

Dive Into Deep Learning provides example code and instruction for how to write DL models in Pytorch, Tensorflow and MXNet.

Computing Resources

Google’s Colab offers free GPU time and a nice environment for running Jupyter notebook-style projects. For $10 per month, you also get priority access to GPUs and TPUs.

Amazon’s SageMaker offers hundres of free hours for newbies.

The CS Department Wilkinson Lab just got 22 new machines that each have a graphics card suitable for deep learning, and should be remote-accessable and running Linux with all the python packages needed for deep learning.

Course Reading

The history.

The Organization of Behavior : Hebb’s 1949 book that provides a general framework for relating behavior to synaptic organization through the dynamics of neural networks.

The Perceptron : This is the 1st neural networks paper, published in 1958. The algorithm won’t be obvious, but the thinking is interesting and the conclusions are worth reading.

The Perceptron: A perceiving and recognizing automoton : This one is an earlier paper by Rosenblatt that is, perhaps, even more historical than the 1958 paper and a bit easer for an engineer to follow, I think.

The basics (1st reading topic)

* Chapter 4 of Machine Learning : This is Tom Mitchell’s book. Historical overview + explanation of backprop of error. It’s a good starting point for actually understanding deep nets. START HERE. IT’S WORTH 2 READINGS. WHAT THAT MEANS IS…GIVE ME 2 PAGES OF REACTIONS FOR THIS READING AND GET CREDIT FOR 2 READINGS

Chapter 6 of Deep Learning : Modern intro on deep nets. To me, this is harder to follow than Chapter 4 of Machine Learning, though. Certainly, it’s longer.

Optimization (2nd reading topic)

This reading is NOT worth points, but… …if you don’t know what a gradient, Jacobian or Hessian is, you should read this before you read Chapter 4 of the Deep Learning book.

Chapter 4 of the Deep Learning Book : This covers basics of gradient-based optimization. Start here for optimization

Chapter 8 of the Deep Learning Book : This covers optimization. This should come 2nd in your optimization reading

Why Momentum Really Works : Reading this will help you understand the popular ADAM optimizer better.

On the Difficulties of Training Recurrent Networks : A 2013 paper that explains vanishing and exploding gradients

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . This is the most common approaches to normalization.

AutoClip: Adaptive Gradient Clipping for Source Separation Networks is a recent paper out of Pardo’s lab that helps deal with unruly gradients. There’s also a video for this one.

Convolutional Networks (3rd reading topic)

Generalization and Network Design Strategies : The original 1989 paper where LeCun describes Convolutional networks. Start here.

Chapter 9 of Deep Learning: Convolutional Networks .

Regularization and overfitting (4th reading topic)

Chapter 7 of the Deep Learning Book : Covers regularization.

Dropout: A Simple Way to Prevent Neural Networks from Overfitting : Explains a widely-used regularizer

Understanding deep learning requires rethinking generalization : Thinks about the question “why aren’t deep nets overfitting even more than they seem to be”?

The Implicit Bias of Gradient Descent on Separable Data : A study of bias that is actually based on the algorithm, rather than the dataset.

Experimental Design

• The Extent and Consequences of P-Hacking in Science

Visualizing and understanding network representations

Visualizing and Understanding Convolutional Networks : How do you see what the net is thinking? Here’s one way.

Local Interpretable Model-Agnostic Explanations (LIME): An Introduction A technique to explain the predictions of any machine learning classifier.

Popular Architectures for Convolutional Networks

If you already understand what convolutional networks are, then here are some populare architectures you can find out about.

Deep Residual Learning for Image Recognition : The 2016 paper that introduces the popular ResNet architecture that can get 100 layers deep

Very Deep Convolutional Networks for Large-Scale Image Recognition : The 2015 paper introducing the popular VGG architecture

Going Deeper with Convolutions :The 2015 paper describing the Inception network architecture.

Adversarial examples

Explaining and Harnessing Adversarial Examples : This paper got the ball rolling by pointing out how to make images that look good but are consistently misclassified by trained deepnets.

Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images : This paper shows just how screwy you can make an image and still have it misclsasified by a “well trained, highly accurate” image recognition deep net.

Effective and Inconspicuous Over-the-air Adversarial Examples with Adaptive Filtering : Cutting edge research from our very own Patrick O.

Creating GANs

Generative Adversarial Nets : The paper that introduced GANs. If you read only one GAN paper, make it this one.

2016 Tutorial on Generative Adversarial Networks by one of the creators of the GAN. This one’s long, but good.

DCGAN: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks : This is an end-to-end model. Many papers build on this. The homework uses the discriminator approach from this paper

Generative Adversarial Text to Image Synthesis This paper describes generating images conditioned on text descriptions. Pretty interesting…

Recurrent Networks

Chapter 10 of Deep Learning : A decent starting point

The Recurrent Neural Networks Tutorial : This is a 4-part tutorial that starts with an overview and then gets deep into coding up an RNN using Theano (not PyTorch) and has links to GitHub repositories with all the examples. If you just read this for the points, read Part 1. But go deep, if you’re interested, and read all the parts. NOTE the links to the code repositories work. Many of the other hyperlinks don’t.

* Extensions of recurrent neural network language model : This covers the RNN language model discussed in class.

Backpropagation through time: what it does and how to do it

Long Term Short Term Memory : The original 1997 paper introducing the LSTM

Understanding LSTMs : A simple (maybe too simple?) walk-through of LSTMs

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling : Compares a simplified LSTM (the GRU) to the original LSTM and also simple RNN units.

Attention networks (read these before looking at Transformers)

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) ** This is a good starting point on attention models. **

Sequence to Sequence Learning with Neural Networks : This is the paper that the link above was trying to explain.

* Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation : This paper introduces encoder-decoder networks for translation. Attention models were first built on this framework. Covered in class.

* Neural Machine Translation by Jointly Learning to Align and Translate : This paper introduces additive attention to an encoder-decoder. Covered in class.

* Effective Approaches to Attention-based Neural Machine Translation : Introduced multiplicative attention. Covered in class.

Massive Exploration of Neural Machine Translation Architectures : A 2017 paper that settles the questions about which architecture is best for doing translation….except that the Transformer model came out that same year and upended everything. Still, a good overview of the pre-transformer state-of-the-art.

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention : Attention started with text, but is now applied to images. Here’s an example.

Listen, Attend and Spell : Attention is also applied to speech, as per this example.

A Tutorial in TensorFlow : Ths walks through how to use Tensorflow 1.X to build a neural machine translation network with attention.

Transformer networks (Don’t read until you understand attention models)

The Illustrated Transformer : A good walkthrough that helps a lot with understanding transformers ** I’d start with this one to learn about transformers.**

The Annotated Transformer : An annotated walk-through of the “Attention is All You Need” paper, complete with detailed python implementation of a transformer.

Attention is All You Need : The paper that introduced transformers, which are a popular and more complicated kind of attention network.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding : A widely-used language model based on Transformer encoder blocks.

The Illustrated GPT-2 : A good overview of GPT-2 and its relation to Transformer decoder blocks.

Reinforcement Learning

Reinforcement Learning: An Introduction, Chapters 3 and 6 : This gives you the basics of what reinforcement learning (RL) is about.

Playing Atari with Deep Reinforcement Learning : A key paper that showed how reinforcement learning can be used with deep nets.

Mastering the game of Go with deep neural networks and tree search : A famous paper that showed how RL + Deepnets = the best Go player in existence at the time.

A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play : This is the AlphaZero paper. AlphaZero is the best go player…and a great chess player.