Over the past six months, GPT chat has become widely used, with the number of GPT chat users reaching 13 million per day according to OpenAI data for February. So how does GPT chat work?

GPT chat takes its name from the acronym Generative Pre-training Transformer, which refers to a generative language model based on a "transformer" architecture. The architecture of transformers is based on the use of attention, a technique that allows the model to pay attention to different parts of the input sequence at different times during sequence processing. This allows transformers to process information more efficiently and perform natural language processing tasks more accurately.

These models are capable of processing large amounts of text and learning to perform natural language processing tasks very efficiently. In particular, the GPT-3 model has 175 billion parameters, making it the largest language model ever trained. To work, GPT needs to be trained on a large amount of text. For example, the GPT-3 model was trained on a text set that included more than 8 million documents and over 10 billion words. With this text, the model learns to perform natural language processing tasks and generate coherent, well-written text. Once the model is well trained, GPT can be used to perform a wide range of tasks.

Generative Pre-training Transformer (GPT) models were first launched by OpenAI in 2018 as GPT-1. The models continued to evolve throughout 2019 with GPT-2, 2020 with GPT-3, and most recently in 2022 with InstructGPT and ChatGPT. InstructGPT introduced a new approach to incorporating human feedback into the learning process to better align model outputs with user intent. Prior to the integration of human feedback into the system, the greatest progress in the evolution of the GPT model was driven by advances in computational efficiency, which allowed GPT-3 to learn from significantly more data than GPT-2, giving it a more diverse knowledge base and the ability to perform a wider range of tasks.

The project "Understanding Artificial Intelligence", implemented by the Institute for Innovative Governance, is supported by the UK Government. The views expressed in this publication are those of the author and do not necessarily reflect the official position of the UK Government.