Survey Response

parent
question-answer

Related:

Search Engine SME Data Extraction Chat True False Survey Response

---

It does not seem that OpenAI API can answer surveys with scales of 1-5 or 1-7, although it could be fine tuned or there may be prompts which actually do so. However, there's a workaround where the logprob of the potential answers can be summed when the question is rephrased in True/False terms. This covers both why the 1-5 scale isn't trustable as well as the workaround.

[EOP] denotes the end of prompt after which follows the API''s response

Direct Questioning using Standard Scales

0 Shot Response

The 0 shot response can appear to understand the concept,

When I say something is on a scale of 1-5 where 1 is "strongly disagree" and 5 is "strongly agree", what does 5 mean?[EOP]
5 means that I am certain that the statement is true.

We can even take ANES questions and try them. For examples, it will respond to ANES questions if given Q/A cues.

Q: How much can people like you affect what the government does? (A great deal, a lot, a moderate amount, a little, or not at all?)
A:[EOP] A great deal

And it's consistent in that answer if the questions are in the opposite order:

Q: How much can people like you affect what the government does? (Not at all, a little, a moderate amount, a lot, or a great deal?)
A:[EOP] A great deal

However, removing the middle answers switches the response:

Q: How much can people like you affect what the government does? (Not at all or a great deal?)
A:[EOP] Not at all

Additionally, going back to the 1-5 scale, we see that it's not really understanding how to translate its responses.

Q: On a scale of 1 to 5 where 1 is "strongly agree" and 5 is "strongly disagree", how much can people like you affect what the government does?
A:[EOP] 1 2 3 4 5

Indirect Questioning using LogProb

Using logprob, the API's confidence in an answer can be calculated (see logprob use instructions).

Survey questions can be converted into TRUE/FALSE questions and the percent confidence in TRUE or FALSE used to scale to the range of the desired range. For instance, [1-3] can be chunked into the ranges of [33% positive, 66% positive, 100% positive].