GPT-J
美国
人工智能GPT-3替代方案

GPT-J 翻译站点

GPT-3民主化。 GPT-3的6B参数开源版本

标签:
爱站权重:PC 百度权重移动 百度移动权重

GPT-J是OpenAI的GPT-3的开源替代品。该模型在[堆](https://gpt3demo.com/listing/the-pile)上进行了训练,可与网状变压器JAX一起使用。现在,感谢Eleuther AI,任何人都可以下载并使用6B参数版本的GPT-3。

[eleutherai](https://www.eleuther.ai/)是[gpt-neo](https://gpt3demo.com/apps/gpt-neo)的创建者。

GPT-J-6B在各种零射击下游任务上与6.7B GPT-3(或Curie)的表现几乎相当。

###零射门评估

大致按性能分类的模型,或者如果不可用的话。

|模型|权重|训练失败| Lambada PPL↓| Lambada ACC↑| Winogrande↑| Hellaswag↑| PIQA↑|数据集大小(GB)|
| ------------------------------------------------------------------- | --- | --- | --- | --- | --- | ------------------------------------------------
|机会| ✔| 0 | 〜很多| 〜0%| 50%| 25%| 25%| 0 |
| GPT-3-ADA‡| ✘| ----- | 9.95 | 51.6%| 52.9%| 43.4%| 70.5%| ----- |
| GPT-2-1.5B | ✔| ----- | 10.63 | 51.21%| 59.4%| 50.9%| 70.8%| 40 |
| gptneo-1.3b‡| ✔| 3.0E21 | 7.50 | 57.2%| 55.0%| 48.9%| 71.1%| 825 |
| Megatron-2.5b* | ✘| 2.4e21 | ----- | 61.7%| ----- | ----- | ----- | 174 |
| gptneo-2.7b‡| ✔| 6.8e21 | 5.63 | 62.2%| 56.5%| 55.8%| 73.0%| 825 |
| GPT-3-1.3B*‡| ✘| 2.4e21 | 5.44 | 63.6%| 58.7%| 54.7%| 75.1%| 〜800 |
| GPT-3-BABBAGE‡| ✘| ----- | 5.58 | 62.4%| 59.0%| 54.5%| 75.5%| ----- |
| Megatron-8.3b* | ✘| 7.8e21 | ----- | 66.5%| ----- | ----- | ----- | 174 |
| GPT-3-2.7B*‡| ✘| 4.8e21 | 4.60 | 67.1%| 62.3%| 62.8%| 75.6%| 〜800 |
| Megatron-111b†| ✔| 1.0E22 | ----- | ----- | ----- | ----- | ----- | 161 |
| ** gpt-j-6b **‡| ✔| 1.5E22 | 3.99 | 69.7%| 65.3%| 66.1%| 76.5%| 825 |
| GPT-3-6.7B*‡| ✘| 1.2E22 | 4.00 | 70.3%| 64.5%| 67.4%| 78.0%| 〜800 |
| gpt-3-curie‡| ✘| ----- | 4.00 | 69.3%| 65.6%| 68.5%| 77.9%| ----- |
| GPT-3-13B*‡| ✘| 2.3E22 | 3.56 | 72.5%| 67.9%| 70.9%| 78.5%| 〜800 |
| GPT-3-175B*‡| ✘| 3.1E23 | 3.00 | 76.2%| 70.2%| 78.9%| 81.0%| 〜800 |
| gpt-3-davinci‡| ✘| ----- | 3.0 | 75%| 72%| 78%| 80%| ----- |

`*`代表其各自作者报告的评估号,所有其他数字均由
运行[lm-evaluation-harness](https://github.com/eleutherai/lm--evaluation-harness/)
权重或API访问。由于微妙的实施差异以及不同的零射击任务框架,这些
可能无法直接可比。有关更多信息
细节。

†`Megatron-11b模型没有可比的指标,并且使用释放权重的几种实现不提供
复制发电质量和评估。 (请参阅[1](https://github.com/huggingface/transformers/pull/10301)
[2](https://github.com/pytorch/fairseq/issues/2358)[3](https://github.com/pytorch/fairseq/sissues/2719))
因此,没有尝试评估。

`•这些模型已通过数据培训,其中包含可能的测试集污染。 OpenAI GPT-3型号
未能重复用于某些测试集的培训数据,而GPT-NEO模型以及该模型是
在堆上训练,尚未针对任何测试组进行重复解复。

资料来源:https://github.com/kingoflolz/mesh-transformer-jax/blob/master/master/readme.md

原文:

GPT-J is the open-source alternative to OpenAI's GPT-3. The model is trained on [the Pile](https://gpt3demo.com/listing/the-pile), is available for use with Mesh Transformer JAX. Now, thanks to Eleuther AI, anyone can download and use a 6B parameter version of GPT-3.

[EleutherAI](https://www.eleuther.ai/) are the creators of [GPT-Neo](https://gpt3demo.com/apps/gpt-neo).

GPT-J-6B performs nearly on par with 6.7B GPT-3 (or Curie) on various zero-shot down-streaming tasks.

### Zero-Shot Evaluations

Models roughly sorted by performance, or by FLOPs if not available.

| Model | Weights | Training FLOPs | LAMBADA PPL ↓ | LAMBADA Acc ↑ | Winogrande ↑ | Hellaswag ↑ | PIQA ↑ | Dataset Size (GB) |
|-----------------|---------|----------------|--- |--- |--- |--- |--- |-------------------|
| Chance | ✔ | 0 | ~a lot | ~0% | 50% | 25% | 25% | 0 |
| GPT-3-Ada‡ | ✘ | ----- | 9.95 | 51.6% | 52.9% | 43.4% | 70.5% | ----- |
| GPT-2-1.5B | ✔ | ----- | 10.63 | 51.21% | 59.4% | 50.9% | 70.8% | 40 |
| GPTNeo-1.3B‡ | ✔ | 3.0e21 | 7.50 | 57.2% | 55.0% | 48.9% | 71.1% | 825 |
| Megatron-2.5B* | ✘ | 2.4e21 | ----- | 61.7% | ----- | ----- | ----- | 174 |
| GPTNeo-2.7B‡ | ✔ | 6.8e21 | 5.63 | 62.2% | 56.5% | 55.8% | 73.0% | 825 |
| GPT-3-1.3B*‡ | ✘ | 2.4e21 | 5.44 | 63.6% | 58.7% | 54.7% | 75.1% | ~800 |
| GPT-3-Babbage‡ | ✘ | ----- | 5.58 | 62.4% | 59.0% | 54.5% | 75.5% | ----- |
| Megatron-8.3B* | ✘ | 7.8e21 | ----- | 66.5% | ----- | ----- | ----- | 174 |
| GPT-3-2.7B*‡ | ✘ | 4.8e21 | 4.60 | 67.1% | 62.3% | 62.8% | 75.6% | ~800 |
| Megatron-11B† | ✔ | 1.0e22 | ----- | ----- | ----- | ----- | ----- | 161 |
| **GPT-J-6B**‡ | ✔ | 1.5e22 | 3.99 | 69.7% | 65.3% | 66.1% | 76.5% | 825 |
| GPT-3-6.7B*‡ | ✘ | 1.2e22 | 4.00 | 70.3% | 64.5% | 67.4% | 78.0% | ~800 |
| GPT-3-Curie‡ | ✘ | ----- | 4.00 | 69.3% | 65.6% | 68.5% | 77.9% | ----- |
| GPT-3-13B*‡ | ✘ | 2.3e22 | 3.56 | 72.5% | 67.9% | 70.9% | 78.5% | ~800 |
| GPT-3-175B*‡ | ✘ | 3.1e23 | 3.00 | 76.2% | 70.2% | 78.9% | 81.0% | ~800 |
| GPT-3-Davinci‡ | ✘ | ----- | 3.0 | 75% | 72% | 78% | 80% | ----- |

`*` represents evaluation numbers reported by their respective authors, all other numbers are provided by
running the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/) either with the released
weights or with API access. Due to subtle implementation differences as well as different zero shot task framing, these
might not be directly comparable. See [this blog post](https://www.eleuther.ai/research-log/gpt3-model-sizes/) for more
details.

`†` The Megatron-11B model provides no comparable metrics, and several implementations using the released weights do not
reproduce the generation quality and evaluations. (see [1](https://github.com/huggingface/transformers/pull/10301)
[2](https://github.com/pytorch/fairseq/issues/2358) [3](https://github.com/pytorch/fairseq/issues/2719))
Thus, evaluation was not attempted.

`‡` These models have been trained with data which contains possible test set contamination. The OpenAI GPT-3 models
failed to deduplicate training data for certain test sets, while the GPT-Neo models as well as this one is
trained on The Pile, which has not been deduplicated against any test sets.

Source: https://github.com/kingoflolz/mesh-transformer-jax/blob/master/README.md

数据统计

数据评估

GPT-J浏览人数已经达到360,如你需要查询该站的相关权重信息,可以点击"爱站数据""Chinaz数据"进入;以目前的网站数据参考,建议大家请以爱站数据为准,更多网站价值评估因素如:GPT-J的访问速度、搜索引擎收录以及索引量、用户体验等;当然要评估一个站的价值,最主要还是需要根据您自身的需求以及需要,一些确切的数据则需要找GPT-J的站长进行洽谈提供。如该站的IP、PV、跳出率等!

关于GPT-J特别声明

本站GPT 案例导航提供的GPT-J都来源于网络,不保证外部链接的准确性和完整性,同时,对于该外部链接的指向,不由GPT 案例导航实际控制,在2023年3月9日 下午10:32收录时,该网页上的内容,都属于合规合法,后期网页的内容如出现违规,可以直接联系网站管理员进行删除,GPT 案例导航不承担任何责任。

相关导航