Whilst a whole book in itself artificial intelligence (AI) and machine learning (ML) have been around since the late 1940’s / early 1950’s. Rapid improvement in computer processing power and architecture ultimately led to the emergence of neural networks (NN) with OpenAI’s first LLM (large language model) emergence in 2018. The articulation and experimentation around the concept of neural networks goes back to the 1800’s.
Large Language Models (LLMs) work by leveraging deep neural network architectures trained on vast amounts of text data. The training process involves predicting the next word in a sequence based on context, which helps the model learn patterns and relationships in language. LLMs use attention mechanisms to focus on relevant parts of the input and generate responses based on learned associations. During inference, given a prompt, the model generates output by sampling or decoding the most likely next tokens. However, it’s important to note that LLMs primarily rely on statistical patterns and lack true understanding or consciousness.
In some ways the process, as we now know it, is similar to search algorithms which allow one to search for a specific result, as in a Google or Bing search. The work on algorithms over 40 years has allowed for a significant leap in natural language processing (NLP) ultimately leading to GPT-3 (Generative Pre-trained Transformer 3). Generative artificial intelligence or generative AI or GenAI.
The important word here is Generative, the models are trained on large data sets and in response to Prompts can generate text, images, or other output. The pace of development is impressive. We are now seeing art, music and creative writing capabilities emerging from various models and various companies.
In this context it may be useful to think of science fiction writer Arthur C. Clarke and his three laws:
The individuals and organizations at the forefront of LLM development have created something that works and it is magical.
A large language model, such as GPT-3.5, is a type of artificial intelligence (AI) model that can generate human-like text based on the input it receives. These models are trained on vast amounts of text data to learn patterns, relationships, and contextual information.
In a language model, text is broken down into smaller units called tokens. A token can be as short as one character or as long as one word, depending on the language and the specific tokenizer used. For example, in English, a token can represent a single character like “a” or a whole word like “apple.” Tokens allow the model to process and understand text at a granular level.
The tokenization process generally follows these steps:
The number of records or data points that a language model holds depends on the training data it was trained on. GPT-3, for instance, was trained on a massive corpus of diverse texts, including books, articles, websites, and more. The precise contents of the training infrastructure and model implementation are often considered proprietary, although OpenAI periodically releases information via https://github.com/openai
During the training process, the language model learns to predict the next token in a sequence of text. By analyzing the context provided by the preceding tokens, the model generates the most likely token to follow. This predictive capability enables the model to generate coherent and contextually appropriate responses.
When interacting with a language model, you provide a prompt or a starting sentence to guide the model’s output. The model then processes the tokens in the prompt and generates a continuation based on its learned knowledge and patterns from the training data. The length of the output can vary depending on the model’s settings or the specific instructions given.
It’s important to note that while LLMs can generate impressive and contextually relevant text, they don’t possess genuine understanding or consciousness. They operate based on statistical patterns learned from training data and may occasionally produce incorrect or nonsensical responses. This is known as AI hallucination. (this is something to watch for since the AI just ‘fabricates’ the information needed to answer your question.)
The training process involves identifying statistical regularities and patterns in the data, allowing the model to make predictions or generate responses based on probabilities.
Lack of Consciousness: While large language models can generate impressive responses, they do not possess consciousness. Consciousness refers to subjective awareness, self-reflection, and the ability to have subjective experiences. Language model do not have a subjective perspective and do not possess consciousness in the way humans do. They do not have beliefs, desires, or emotions.
In the context of machine learning Parameters refer to the variables incorporated within a trained model that enable the generation of new content through inference. Parameters are not words or tokens they are the variables that encapsulate the learned knowledge from the training data. They include weights and biases in neural networks, which are adjusted during training to optimize the model’s performance and enable accurate predictions or text generation.
The specific nature of these variables varies depending on the type of machine learning model. In the case of neural networks, which are the basis for many language models including GPT-3.5, the parameters consist of weights and biases.
Weights are the numerical values assigned to the connections between neurons in the neural network. They determine the strength of the connections and govern the impact of one neuron’s output on another. Adjusting the weights allows the model to learn and adapt to the patterns in the data during the training process.
Biases are additional parameters in a neural network that provide an offset or baseline activation for each neuron. They help the model account for variations and trends in the data that may not be captured by the weights alone.
Here is an overview of the sequential process that takes place from your prompt to the AI answer:
This is a general overview of the process, the specific implementation details can vary depending on the model architecture and the system configuration.
The pace of development is astounding, looking back at the first GPT launched by OpenAI in 2018 – 117 million parameters.
OpenAI’s GPT 2 release in 2019:
https://openai.com/research/gpt-2-1-5b-release 1.5B parameters – roughly 10 Billion tokens (with the average token size is 4 characters)
By 2020 this parameter range had extended to 175 billion parameters.
By 2023 the May release of GPT4 the parameter count is estimated to be in the order of 100 trillion. OpenAI have not confirmed the actual parameter count.
This is a somewhat simplistic explanation of how an LLM works, the technical details are often discussed / analysed in detail via Artificial Intelligence papers published in Cornell University’s https://arxiv.org/list/cs.AI/recent