Confidence

Every Choice and Score answer carries a confidence between 0 and 1. It tells you how sure Sonoe is, separately from what it chose. Use it to decide whether to act on an answer or hand it to someone else.

What it measures

Confidence is 1 − the normalized entropy of the answer's probabilities:

• 1 when all the probability is on one option.
• 0 when it's spread evenly over every option.

Because it's normalized, it means the same thing with 2 options or with 200. A Choice that puts 0.99 on one option of three gets about 0.95; one that splits 0.41 / 0.11 / 0.49 gets about 0.13.

A Noul answer has no confidence: its probability already says how sure it is. A Noul near 0.5 is undecided; near 0 or 1 it's sure.

Probability and confidence are different

The probability says what; the confidence says whether to trust it. A Choice can pick billing with a probability of 0.49 and a confidence of 0.13: billing is the best guess, but not by much. Decide with both:

team = answers["team"]
if team["confidence"] >= 0.6:
    route_to(team["choice"])
else:
    send_to_a_person()

The right threshold depends on how costly a wrong answer is in your app. Pick it on your own labeled data: look at the accuracy of the answers above each threshold and choose the lowest one that's accurate enough.

Abstaining (native API)

The native /v1/decide endpoint does this for you. Every answer there carries abstain: true when the confidence is below a threshold, meaning "don't trust this one".

The threshold, in order of precedence:

1. min_confidence on the question.
2. min_confidence on the request (applies to every question).
3. The model's default for that type, calibrated so answered questions reach a target accuracy on held-out data:

{
  "state": "the thing",
  "questions": {
    "department": {
      "type": "choice",
      "ask": "Which department?",
      "options": { "billing": null, "shipping": null, "sales": null }
    }
  },
  "min_confidence": 0.5
}

{
  "answers": {
    "department": {
      "type": "choice",
      "answer": "sales",
      "probabilities": { "billing": 0.4075, "shipping": 0.1066, "sales": 0.4859 },
      "confidence": 0.1306,
      "abstain": true
    }
  }
}

Abstaining is an answer, not an error: Sonoe still tells you its best guess and the probabilities. Your code decides what to do: ask the person a follow-up, use a rule, or send it to a human.

The Jev-compatible /v1/systemone doesn't return abstain (Jev has no such field). Compare confidence with your own threshold instead.