Counting

parent
summarization

Related:

Summarization Metrics Counting Classification Labels

---

Overview

The API has a hard time counting. For instance, let's say we just want to count up how many 1's appear in a sequence of 0's and 1's.

Making sure the API knows what the target is can help. E.g. repeating the target sequence helps for some reason;

The simple prompt would giving some examples and then asking the number of 1's, e.g.

1 1 0 0 0 1 1 1 1 0
q:how many 1s?
a:6

0 1 1 0 0 0 1 0 1 0
q:how many 1s?
a:4

A repeated prompt would repeat the sequence so the API has a better idea of what it wants;

1 1 0 0 0 1 1 1 1 0
q:what was the series?
a:1 1 0 0 0 1 1 1 1 0
q:how many 1s?
a:6

0 1 1 0 0 0 1 0 1 0
q:what was the series?
a:0 1 1 0 0 0 1 0 1 0
q:how many 1s?
a:4

As the graph below shows, running on 500 random sequences at each length, doing the repeated prompt can improve performance.

(Code at https://gist.github.com/brockmanmatt/40f23a3101684a857567524401c4650f)