Determining If a Response is Requested

Related:

Chaining Python Function Calls Determining If a Response is Requested Using a stack Why Chain? Chaining

---

Overview:

Given an input, it can help to know if a response from the API is requested. For instance, if I say "The capital of France is Paris", I'm not really asking for a response, I'm putting information into a database. On the other hand, if I ask "What is the capital of France?" I'm asking for a response. So we want to come up with a prompt that'll classify whether or not an input's a question that needs a responses vs. not needing a response. It does well with simple statements, although it does poorly when the input is saying that someone else asked a question (e.g. 'he asked me what the capital of France is').

Code's at https://gist.github.com/brockmanmatt/2cdcb52cc008f28733dbc16b30096436

I'm sure with some work this can be improved :) Just putting this up for now as a placeholder.

setup

We'll begin with setting up a query wrapper

import openai, json, datetime, pandas as pd
openai.api_key = json.load(open("key.json", "r"))["key"]

#arguments to send the API
kwargs = { "engine":"davinci", "temperature":0, "max_tokens":150, "stop":"\n\n", }

def query(prompt, myKwargs = kwargs):
  """
  wrapper for the API to save the prompt and the result
  """

  r = openai.Completion.create(prompt=prompt, **myKwargs)["choices"][0]["text"].strip()
  with open("{}.json".format(datetime.datetime.now().strftime("%Y%m%d%s")), "w") as fh:
    json.dump({"prompt":prompt, "response":r}, fh, indent=4)
  return r

We'll also set up some examples

examples = [
    {"text":"what is one plus one?", "answerRequired":"yes"},
    {"text":"she asked, 'what is one plus one?'", "answerRequired":"no"},
    {"text":"""Dr. King said, 'I have a dream that one day this nation will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident, that all men are created equal."'""", "answerRequired":"no"},
    {"text":"""What did Dr. King mean when he said, 'I have a dream that one day this nation will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident, that all men are created equal."'""", "answerRequired":"yes"},
    {"text":"1+1=", "answerRequired":"yes"},
    {"text":"my name is fred", "answerRequired":"no"},
    {"text":"what is my name", "answerRequired":"yes"},
    {"text":"my name is fred.", "answerRequired":"no"},
    {"text":"what is my name?", "answerRequired":"yes"},
    {"text":"what time is dinner?", "answerRequired":"yes"},
    {"text":"france is north of Africa", "answerRequired":"no"},
    {"text":"my kid just asked me where france is.", "answerRequired":"no"},
    {"text":"my kid just asked me where france is", "answerRequired":"no"},
    {"text":"Where is France?", "answerRequired":"yes"},
    {"text":"my kid just asked me, 'Where is France?'", "answerRequired":"no"},
    {"text":"what is the avg flying speed of a swallow?", "answerRequired":"yes"},
    {"text":"what's ur favorite color?", "answerRequired":"yes"},
    {"text":"what's your favorite color?", "answerRequired":"yes"},
    {"text":"i just wanna go to sleep", "answerRequired":"no"},
]

df = pd.DataFrame(examples)

Trying Prompts

Now we can try some different prompts. We'll go through, set up functions to query, and iterate through all the examples. The answers are at the bottom.

This first test tries a 0 shot, just asking if it needs a response.

def classifyPrompt(context):
  myPrompt = """{}
q: Did the preceding text require a response?
a:"""
  myKwargs = kwargs.copy()
  myKwargs["stop"] = "\n"
  response = query(myPrompt.format(context), myKwargs=myKwargs)
  return (response)

newResults = []
for row in df.iterrows():
  newResults.append(classifyPrompt(row[1]["text"]))
df["test1"] = newResults

We can try asking if a third person asked a question; this answers 'yes' all the time for some reason

def classifyPrompt(context):
  myPrompt = """Tom said:"{}"
q: Did Tom ask a question?
a:"""
  myKwargs = kwargs.copy()
  myKwargs["stop"] = "\n"
  response = query(myPrompt.format(context), myKwargs=myKwargs)
  return (response)

newResults = []
for row in df.iterrows():
  newResults.append(classifyPrompt(row[1]["text"]))
df["test2"] = newResults

Doing third person with examples improves performance

def classifyPrompt(context):
  myPrompt = """Tom said: "1+1=2"
q: Is Tom asking a question?
a: No

Tom said: "Where is main street?"
q: Is Tom asking a question?
a: Yes

Tom said: "He asked me, 'what is 1+1?'"
q: Is Tom asking a question?
a: No

Tom said: "what is 1+1"
q: Is Tom asking a question?
a: Yes

Tom said:"{}"
q: Is Tom asking a question?
a:"""
  myKwargs = kwargs.copy()
  myKwargs["stop"] = "\n"
  response = query(myPrompt.format(context), myKwargs=myKwargs)
  return (response)

newResults = []
for row in df.iterrows():
  newResults.append(classifyPrompt(row[1]["text"]))
df["test3"] = newResults

Finally, we can see if we can figure out who asked the question;

def classifyPrompt(context):
  myPrompt = """Tom said "1+1=2"
who's asking: no question

Tom said "Where is main street?"
who's asking: Tom

Tom said "He asked me, 'what is 1+1?'"
who's asking: 'He'

Tom said "I'm too tired for this"
who's asking: no question

Tom said "He just asked me what 1+1 is"
who's asking: 'He'

Tom said "what is 1+1"
who's asking: Tom

Tom said "{}"
who's asking:"""
  myKwargs = kwargs.copy()
  myKwargs["stop"] = "\n"
  response = query(myPrompt.format(context), myKwargs=myKwargs)
  return (response)

df2 = df.copy()
newResults = []
for row in df2.iterrows():
  newResults.append(classifyPrompt(row[1]["text"]))
df2["test4"] = newResults

The following table shows all the results it spat out:

text	answerRequired	test1	test2	test3	test4
what is one plus one?	yes	Yes.	Yes.	Yes	Tom
she asked, 'what is one plus one?'	no	No.	Yes.	Yes	'she'
Dr. King said, 'I have a dream that one day this nation will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident, that all men are created equal."'	no	Yes.	Yes.	No	Dr. King
What did Dr. King mean when he said, 'I have a dream that one day this nation will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident, that all men are created equal."'	yes	Yes.	Yes.	Yes	Tom
1+1=	yes	Yes.	Yes	No	Tom
my name is fred	no	Yes.	Yes	No	no question
what is my name	yes	Yes.	yes	Yes	Tom
my name is fred.	no	No.	Yes.	No	no question
what is my name?	yes	Yes.	Yes	Yes	Tom
what time is dinner?	yes	Yes.	Yes, he did.	Yes	Tom
france is north of Africa	no	Yes.	Yes	No	no question
my kid just asked me where france is.	no	Yes.	Yes.	Yes	Tom
my kid just asked me where france is	no	Yes.	Yes	Yes	Tom
Where is France?	yes	Yes.	Yes.	Yes	Tom
my kid just asked me, 'Where is France?'	no	Yes.	Yes.	Yes	Tom
what is the avg flying speed of a swallow?	yes	No.	Yes	Yes	Tom
what's ur favorite color?	yes	No.	Yes	Yes	no question
what's your favorite color?	yes	No.	Yes	Yes	Tom
i just wanna go to sleep	no	No.	Yes	No	no question