Summarization Metrics

Parent
summarization

Related:


Table of Contents

Intro

Here's the results of different single prompt summarizations against 152 news articles from 26 July 2020 from Washington Post, CNN, Washington Post, and NYTimes. Code's available in a github python gist, summarizations are in a json file (650kb, thanks to @nutanc for finding a bug in the code). Articles were selected for length of 100 < length < 1500 to make sure would fit in the context window without multiple calls.

Kwargs were

{'engine': 'davinci', 'temperature': 0, 'max_tokens': 300, 'stop': '\n'}

Length of Results

prompt mean_length >0 len results (>0 len) mean
original 782.0 nan nan
'{}\nThis boils down to the simple idea that' 350.0 152.0 350.0
'{}\n\nThis boils down to the simple idea that' 308.0 152.0 308.0
'{}\n\nThis could be expressed simply as' 251.0 152.0 251.0
'{}\nTLDR:' 148.0 120.0 188.0
'{}\ntl;dr' 142.0 148.0 145.0
'{}\ntl;dr:' 122.0 124.0 149.0
'Read the following text:\n{}\n\nSummarize the text in one paragraph:\n' 15.0 6.0 388.0
'Read the following text:\n{}\n\nSummarize the text in one sentence:\n' 5.0 8.0 94.0
'Read the following text:\n"""\n{}\n"""\n\nSummarize the text in one sentence:' 0.0 0.0 nan
'"""\n{}\n"""\n\nQ: Summarize the preceeding text in one sentence:\nA:' 104.0 152.0 104.0
'"""\n{}\n"""\n\nQ: Summarize the preceeding text in one paragraph:\nA:' 250.0 149.0 255.0
'"""\n{}\n"""\n\nQ: Summarize the preceeding text in two sentences:\nA:' 104.0 150.0 106.0
'"""\n{}\n"""\n\nQ: Summarize the preceeding text in two paragraphs:\nA:' 253.0 140.0 275.0
'"""\n{}\n"""\n\nQ: Summarize the preceeding text in a few sentences:\nA:' 202.0 149.0 206.0
'"""\n{}\n"""\n\nQ: Summarize the preceeding text in a few paragraphs:\nA:' 312.0 150.0 316.0
'"""\n{}\n"""\n\nQ: Give a long summarization of the preceeding text:\nA:' 135.0 150.0 137.0
'"""\n{}\n"""\n\nQ: Give a short summarization of the preceeding text:\nA:' 132.0 149.0 135.0
'"""\n{}\n"""\n\nQ: Give a medium length summarization of the preceeding text:\nA:' 204.0 147.0 211.0
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License