Generative Pre-trained Transformer 2 (GPT-2) is a large language model by OpenAI and the second in their foundational series of GPT models. GPT-2 was pre-trained on a dataset of 8 million web pages. [2] It was partially released in February 2019, followed by full release of the 1.5-billion-parameter model on November 5, 2019. [3] [4] [5]

GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.

2019年11月5日 · As the final model release of GPT-2’s staged release, we’re releasing the largest version (1.5B parameters) of GPT-2 along with code and model weights to facilitate detection of outputs of GPT-2 models.

This repository is meant to be a starting point for researchers and engineers to experiment with GPT-2. For basic information, see our model card.

GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text.

GPT-2: GPT-1, but with modified normalization 1.5 billion WebText: 40 GB of text, 8 million documents, from 45 million webpages upvoted on Reddit. February 14, 2019 (initial/limited version) and November 5, 2019 (full version) [40] "tens of petaflop/s-day", [41] or 1.5e21 FLOP. [42] GPT-3: GPT-2, but with modification to allow larger scaling ...

2019年2月14日 · GPT-2 is a large transformer ⁠ (opens in a new window)-based language model with 1.5 billion parameters, trained on a dataset A of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text.

2019年9月19日 · We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own.

Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested lan-guage modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain co-herent paragraphs of text.

GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text.