Skip to content

alivirgo/Major-AI-Skills / plugins/mas-common-sense-pack/skills/ask-ai-confidence-rating

ask-ai-confidence-rating

Verified

Identify uncertainty, assumptions, and claims needing verification when reviewing an AI answer.

Ask AI for Confidence & Uncertainty Ratings (AI Skill)

Overview

Standard LLMs produce responses with an unwavering, authoritative tone regardless of whether they are stating a verified mathematical axiom or guessing an obscure regulatory nuance.

This skill provides an Epistemic Calibration Framework that instructs AI models to evaluate their own certainty, explicitly flag assumptions, and provide numeric confidence ratings for each component of an analysis.


The Confidence Calibration Matrix

┌─────────────────────────────────────────────────────────────┐
│                 Certainty Breakdown Scale                   │
│                                                             │
│  [90-100%] HIGH CERTAINTY ──► Established facts, syntax,   │
│                               mathematical derivations      │
│  [60-89%]  MODERATE       ──► Best practices, industry      │
│                               heuristics, common patterns   │
│  [0-59%]   LOW / SPECULATIVE► Rapidly changing data, niche  │
│                               laws, subjective predictions  │
└─────────────────────────────────────────────────────────────┘

The Calibrated System Prompt Template

Use this prompt modifier for research, technical architecture, legal/tax guidelines, and high-stakes problem solving:

Answer the following query: [INSERT QUESTION]

Formatting Requirements:
1. Provide the direct, actionable answer first.
2. Underneath your answer, include a "Certainty & Risk Breakdown" with:
  - **Confidence Score (1-100%)**: State your overall confidence in the factual accuracy of this response.
  - **High-Certainty Elements**: List the specific points you are 100% sure about.
  - **Uncertainty & Assumptions**: Explicitly list any assumptions, ambiguities in my prompt, or edge cases where this answer might fail.
  - **Verification Step**: Provide the single most effective action I can take to verify this independently (e.g., official docs page, CLI test, calculation).

Real-World Examples

Example 1: Software Infrastructure Migration

Naive Prompt

"Can I migrate Postgres 12 directly to Postgres 16 using pg_upgrade in AWS RDS?"

Uncalibrated AI Response: "Yes, AWS RDS supports direct major version upgrades from PostgreSQL 12 to 16 using pg_upgrade." (Authoritative, but misses crucial downtime, parameter group, and extension deprecation caveats).

Calibrated Confidence Prompt

User: "Answer the upgrade question, but include a Confidence & Edge Case Audit."

AI Calibrated Response: Answer: Yes, AWS RDS supports multi-major-version automated upgrades from PostgreSQL 12 directly to 16 without intermediate hops.

Certainty & Risk Breakdown:

  • Confidence Score: 95% (AWS RDS documentation officially supports multi-version in-place upgrades).
  • High-Certainty Elements: The RDS upgrade engine handles binary conversion automatically; downtime will be incurred during the catalog update.
  • Uncertain / Risk Areas (5%):
    1. PostGIS or custom extensions installed on v12 may have breaking schema changes in v16.
    2. Custom parameter groups must be manually recreated for PostgreSQL 16.
  • Verification Action: Run aws rds modify-db-instance --db-instance-identifier <id> --engine-version 16.2 --dry-run or test on a restored snapshot first.

Example 2: Contractual & Tax Clause Analysis

Calibrated Assessment Example

User Prompt: "Is home office rent 100% tax deductible for an LLC in California if the room is also used for occasional guest lodging? Rate your confidence."

AI Calibrated Response: Answer: No. Under IRS guidelines (IRC § 280A) and California Franchise Tax Board rules, home office deduction requires regular and exclusive use. Using the space for guest lodging disqualifies it from the simplified or actual expense deduction.

Certainty Breakdown:

  • Confidence Score: 99% (The "exclusive use" requirement is a strict statutory standard with near-zero ambiguity in US federal and California tax code).
  • Key Caveat: Daycare facilities and inventory storage have minor statutory exceptions, none of which apply to guest lodging.
  • Verification Action: Review IRS Publication 587 (Business Use of Your Home), page 3 ("Exclusive Use Rule").

Guarding Against "False Modesty" & "Overconfidence"

Model TendencyWhat HappensHow to Fix It in the Prompt
Overconfidence BiasGives 100% certainty to hallucinated API endpointsAdd: "Penalize your score if the library is newer than 2023 or frequently changes APIs."
SycophancyDrops confidence simply because user asked "Are you sure?"Add: "Do not lower your rating just because I question it. Defend factually grounded points."
Generic DisclaimersDumps boilerplate ("I am an AI, consult a doctor/lawyer")Add: "Skip generic boilerplate; evaluate only the epistemic certainty of the factual statements."

Tactical Summary Checklist

  • When confidence is $\ge 90%$: Safe to proceed with standard testing.
  • When confidence is $60 - 89%$: Always verify against official documentation or run a sandbox trial.
  • When confidence is $< 60%$: Treat the AI output merely as a brainstorming prompt, not an actionable recommendation.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall