Selected work · 01
Pramana: Open-source decision models with calibrated confidence
Small open models that answer a structured question with a probability for every option, and say how much to trust that answer.
What it is
Pramana takes one piece of unstructured state and a typed question, and returns a probability over every option rather than a single label. Each answer carries two outputs from the same logits: a calibrated probability, and a separate confidence score that tracks whether the top answer is actually right.
Two models ship: Pramana-2B and Pramana-0.8B, both trained from Qwen bases and published on Hugging Face with their calibration files and predictions.
How it works
- Soft gold training. Fine-tuned with LoRA against soft labels and a Brier term, so the probabilities come out calibrated without a post-hoc temperature. The fitted temperature lands between 1.00 and 1.05.
- Decoupled calibration. One set of logits, two temperatures: one fitted to the soft gold for KL and Brier, one fitted to the hard label for expected calibration error and deferral.
- A 0.8B to 2B cascade. The small model answers whenever its top probability clears 0.68, chosen on holdout; the rest go to the 2B. One decision in three never reaches the larger model and accuracy holds.
What it measured
| Model | Params | Accuracy | KL | Brier | Confidence ECE |
|---|---|---|---|---|---|
| Pramana-2B | 2.3B | 0.8005 | 0.0755 | 0.0415 | 0.0379 |
| Pramana-0.8B | 0.8B | 0.7970 | 0.0819 | 0.0446 | 0.0249 |
| Board, 3-4B entries | 3-4B | 0.808-0.8205 | 0.077-0.301 | 0.042-0.132 | - |
Typed Decisions benchmark: 400 test cases, 2,000 decisions, four workflows. Pramana is trained on the train split, so it is compared with other fine-tuned entries, not zero-shot rows.
What it does not claim
The 3-4B board entries still edge out Pramana on raw accuracy. The claim here is calibration and cost, not a state-of-the-art score: the lowest KL and Brier of any fine-tuned entry listed, with a cascade that keeps accuracy while leaving the larger model idle for a third of decisions.