Math

parent:
logic

Related:

Compare and Contrast Math

---

OpenAI API can do some math out of the gate, but most math it seems it has to learn. Many times, the numbers that it spits out are just random. However, including different priming prompts can result in decent results.

Prompts are displayed followed by [EOP] after which the API results are shown.

Addition

Zero-Shot

1-2 digit Ints

Addition with 1-2 digits is pretty accurate

1+2=[EOP] 2.

40 plus 50 equals[EOP] 90

However, addition breaks once hitting 3 digits; e.g.

120+200=[EOP] 420 **WRONG, actual answer 320**

Few Shot

Addition

For three digits, it can sometimes improve with a few examples. I'll put a chart of performance up.

120+209=329
980+40=1020
591+325=926
123+456=[EOP]579

But at larger numbers it starts failing again

7400357 + 2709779 = 10110136
9359517 + 8280301 = 17639818
6885075 + 9531680 = 16416755
3466242 + 8344302 = 11810544
4334300 + 6629847 =[EOP] 10996547 **WRONG, actual answer 10964147; 3 digits off**

Commas

Commas may improves Performance on Large Numbers (code at https://gist.github.com/brockmanmatt/8ca7cd40c3f79d31edb7fdfde0c782d1)

Adding commas can improve performance; this example can arguable be worse than without commas but so far testing indicates it improves percent error on average too. The API requires prompts to include examples with a similar number of digits or it starts failing quickly.

7,400,357 + 2,709,779 = 10,110,136
9,359,517 + 8,280,301 = 17,639,818
6,885,075 + 9,531,680 = 16,416,755
3,466,242 + 8,344,302 = 11,810,544
4,334,300 + 6,629,847 =[EOP] 11,964,147 **WRONG, actual answer 10,964,147; off by 1 digit**

Testing the performance of the same set of randomly generated ints at different maximum lengths results in commas way outperforming unformatted ints to test on both mean absolute percent error and number entirely correct (see below). Tests were given examples of similar length.

Davinci

Davinci 2b

Davinci 2b has mixed results but generally improvements on raw davinci addition output.

Mean Error (less error good):
Change in Mean Error (No Comma)	-.3%
Change in Mean Error (Comma)	+.01%

Exact Match (more match good)
Change in Exact Match (No Comma)	+.22%
Change in Exact Match (Comma)	+ .55%

Dollar Signs

Prefixing a $ to ints along with commas improves performance until around 6 digits, at which point it begins to degrade performance.

Unknown

Subtraction

Zero shot subtraction doesn't work, one-shot with 3 digits does

541 - 125 = 416
623 - 532 =[EOP] 91

Multiplication

Zero shot multilpication with 'x' or '*' doesn't appear to work, one-shot with 1 digit can;

5 x 5 = 25
5 x 8 =[EOP]  40

Counting

0 shot

OpenAI will continue patterns

1
2
3
4[EOP]
5

5
10
15
25
30[EOP]
35