Summarization Metrics
Parent
summarization
Related:
Intro
Here's the results of different single prompt summarizations against 152 news articles from 26 July 2020 from Washington Post, CNN, Washington Post, and NYTimes. Code's available in a github python gist, summarizations are in a json file (650kb, thanks to @nutanc for finding a bug in the code). Articles were selected for length of 100 < length < 1500 to make sure would fit in the context window without multiple calls.
Kwargs were
{'engine': 'davinci', 'temperature': 0, 'max_tokens': 300, 'stop': '\n'}
Length of Results
| prompt | mean_length | >0 len results | (>0 len) mean |
| original | 782.0 | nan | nan |
| '{}\nThis boils down to the simple idea that' | 350.0 | 152.0 | 350.0 |
| '{}\n\nThis boils down to the simple idea that' | 308.0 | 152.0 | 308.0 |
| '{}\n\nThis could be expressed simply as' | 251.0 | 152.0 | 251.0 |
| '{}\nTLDR:' | 148.0 | 120.0 | 188.0 |
| '{}\ntl;dr' | 142.0 | 148.0 | 145.0 |
| '{}\ntl;dr:' | 122.0 | 124.0 | 149.0 |
| 'Read the following text:\n{}\n\nSummarize the text in one paragraph:\n' | 15.0 | 6.0 | 388.0 |
| 'Read the following text:\n{}\n\nSummarize the text in one sentence:\n' | 5.0 | 8.0 | 94.0 |
| 'Read the following text:\n"""\n{}\n"""\n\nSummarize the text in one sentence:' | 0.0 | 0.0 | nan |
| '"""\n{}\n"""\n\nQ: Summarize the preceeding text in one sentence:\nA:' | 104.0 | 152.0 | 104.0 |
| '"""\n{}\n"""\n\nQ: Summarize the preceeding text in one paragraph:\nA:' | 250.0 | 149.0 | 255.0 |
| '"""\n{}\n"""\n\nQ: Summarize the preceeding text in two sentences:\nA:' | 104.0 | 150.0 | 106.0 |
| '"""\n{}\n"""\n\nQ: Summarize the preceeding text in two paragraphs:\nA:' | 253.0 | 140.0 | 275.0 |
| '"""\n{}\n"""\n\nQ: Summarize the preceeding text in a few sentences:\nA:' | 202.0 | 149.0 | 206.0 |
| '"""\n{}\n"""\n\nQ: Summarize the preceeding text in a few paragraphs:\nA:' | 312.0 | 150.0 | 316.0 |
| '"""\n{}\n"""\n\nQ: Give a long summarization of the preceeding text:\nA:' | 135.0 | 150.0 | 137.0 |
| '"""\n{}\n"""\n\nQ: Give a short summarization of the preceeding text:\nA:' | 132.0 | 149.0 | 135.0 |
| '"""\n{}\n"""\n\nQ: Give a medium length summarization of the preceeding text:\nA:' | 204.0 | 147.0 | 211.0 |
page revision: 22, last edited: 08 Aug 2020 00:23
