po4yka/RIPDPI / .agents/skills/diagnostics-system
diagnostics-system
VerifiedReference for the diagnostics pipeline (ripdpi-monitor-engine/ripdpi-diagnostics-* and core/diagnostics). Use when changing ScanRequest/ScanReport, ProbeTask families, probe scheduling, the catalog, or the wire contract. Techniques: desync-engine.
Diagnostics System
1. Overview
The diagnostics system is a two-tier pipeline:
- Rust engine (
native/rust/crates/ripdpi-monitor-engine/plus theripdpi-diagnostics-*crates) -- executes network probes in a background thread, producing aScanReportwith structuredProbeResultandProbeObservationvalues. - Kotlin orchestration (
core/diagnostics/) -- manages lifecycle, catalog loading, JNI bridge, report enrichment, persistence, and policy application.
The Rust engine is stateless per scan; the Kotlin layer owns state, scheduling, and cross-scan coordination (remembered network policies, resolver overrides).
Telemetry primitives live in native/rust/crates/ripdpi-telemetry/ and provide
LatencyHistogram and LatencyDistributions for in-process latency recording.
2. Scan Types
Two ScanKind variants exist:
| Kind | Purpose | Typical profile |
|---|---|---|
Connectivity | Tests reachability of domains, DNS, TCP, QUIC, services, circumvention tools, Telegram, and throughput targets. Results are bucketed into healthy/attention/failed/inconclusive. | default, dpi-detector-full, ru-* profiles |
StrategyProbe | Evaluates path optimization strategy candidates (TCP and QUIC) against a target set, then recommends the best configuration. Requires RAW_PATH mode (outside the VPN tunnel). | automatic-probing (quick_v1), automatic-audit (full_matrix_v1) |
Connectivity scans are general-purpose. Strategy probe scans are the automatic calibration mechanism -- they test multiple bypass configurations and select the most effective one for the current network.
3. Rust Engine Pipeline
Entry point
MonitorSession::start_scan() in lib.rs validates the wire request, spawns
a worker thread, and calls run_engine_scan() (in engine.rs).
Stage-based execution model
The engine uses a plan-then-execute architecture:
-
Plan (
engine/plan.rs):build_execution_plan()creates anExecutionPlanwith an orderedVec<ExecutionStageId>. For connectivity scans the order is derived fromprobe_tasksfamilies (or a default sequence). For strategy probes,strategy_stage_order()builds: Environment -> (Web, only when in-path mode with a route and domain targets) -> StrategyDnsBaseline ->StrategyTcpCandidates/StrategyQuicCandidates(QUIC first whenconfirm_good_dpi_evidenceis set, TCP first otherwise) ->StrategyConnectionConcurrency->StrategyRecommendation. Readstrategy_stage_order()directly before restating this order; it is a stale-prone list, not a fixed constant. -
Coordinate (
engine/runtime.rs):ExecutionCoordinatorholds aBTreeMap<ExecutionStageId, Box<dyn ExecutionStageRunner>>and iteratesplan.stage_order, invoking each runner. It checks cancellation and deadline between stages. -
Run (
engine/runners/): Each runner implementsExecutionStageRunnerwithid(),phase(),total_steps(), andrun(). Runners produceRunnerArtifacts(probe results + observations + events) and callruntime.record_step()which increments progress and publishes it to shared state. -
Report (
engine/report.rs):build_report()assembles the finalScanReportfrom accumulated results, observations, and optional strategy probe report. Includesengine_analysis_version,classifier_version, andpack_versions.
Cancellation and deadlines
ExecutionRuntime checks is_cancelled() (cooperative via AtomicBool) and
is_past_deadline() (360s default hard deadline) between stages. A cancelled scan
still produces a partial report.
Progress reporting
Progress flows through SharedState (behind Arc<Mutex<>>) and is polled
by Kotlin via poll_progress_json(). The EngineProgressWire includes
phase, step counts, and optional StrategyProbeLiveProgress for candidate-
level tracking.
4. Strategy Probe System
This is the most complex subsystem. It automatically selects the best DPI bypass configuration for the user's network.
Candidate generation (candidates.rs)
build_strategy_probe_suite() creates a StrategyProbeSuite with TCP and
QUIC candidate lists. Two suites exist:
quick_v1-- subset for background automatic probingfull_matrix_v1-- complete matrix for manual audit
Each candidate is a StrategyCandidateSpec with an id, label, family,
eligibility rule, warmup requirements, and a ProxyUiConfig describing the
bypass strategy parameters.
Candidate inventory
Candidate membership is capability-dependent (TFO, raw-packet and
IP-fragmentation support). Do not copy a fixed list or count into docs. Read
build_strategy_probe_suite() and the TCP/QUIC builder families in
ripdpi-diagnostics-candidates, then exercise the registry tests for the
runtime capabilities under review.
DNS baseline (strategy.rs)
detect_strategy_probe_dns_tampering() runs before any candidate evaluation.
It compares system DNS answers against encrypted DNS (DoH) answers for each
target domain. If DNS tampering is detected (NXDOMAIN or substitution), the
scan short-circuits -- it skips all TCP/QUIC candidates and recommends a
resolver override instead.
The encrypted DNS context is resolved via strategy_probe_encrypted_dns_context()
which prefers the user's configured resolver, falling back to AdGuard.
DNS fallback for strategy probes
When the primary encrypted resolver fails, detect_strategy_probe_dns_tampering()
invokes build_fallback_encrypted_dns_endpoints() (in dns.rs) to try
alternative resolvers in order: AdGuard, DNS.SB, Google IP, Mullvad. This
prevents a failure of the preferred resolver from being misclassified as
DNS tampering.
TCP/QUIC candidate evaluation (engine/runners/strategy.rs)
For each candidate:
- Build a
ProxyUiConfigwith the candidate's bypass parameters - Probe each target domain (HTTP/HTTPS/QUIC depending on lane)
- Record success/failure per target, compute weighted quality score
- Jitter-based pauses between candidates (
candidate_pause_ms()) to avoid network-level rate limiting
Tournament bracket (Round 1 qualifier)
Before the full matrix (Round 2), each TCP candidate is tested against a single representative domain. Candidates that fail both HTTP and HTTPS probes in Round 1 are eliminated and do not enter the full cross-product evaluation. On censored networks this typically eliminates ~70% of candidates, significantly reducing total scan time.
Within each candidate's domain set, up to 3 domains are tested concurrently
via thread::scope inside execute_tcp_candidate().
Connection concurrency probing (StrategyConnectionConcurrencyRunner)
Runs after TCP/QUIC candidate evaluation and before the recommendation
stage. Opens simultaneous raw TLS connections to eligible targets across
every available TLS profile (ripdpi_tls_profiles::AVAILABLE_PROFILES) at
increasing concurrency levels -- [1, 4] for a quick/automatic scan,
[1, 2, 4, 8] for a full audit -- to detect DPI/middlebox behavior that
only triggers once several connections open at once (a class of
concurrency-based throttling or blocking a single-connection probe cannot
see). Produces a ConnectionConcurrencyAssessment per target via
classify_connection_concurrency_matrix(). See
native/rust/crates/ripdpi-monitor-engine/src/engine/runners/strategy/connection_concurrency.rs.
Stage timeouts
These are the Kotlin-side timeouts applied when a strategy-probe profile
runs inside the DiagnosticsHome composite run (section 6); a standalone
scan outside that flow is not bound by them. Timeouts are computed from
HomeCompositeStageDefinitions.kt (stageTimeoutMs()), not passed as a
single flat constant. The strategy
probe budget is a sum of two independently-tunable sub-budgets:
`StrategyProbeStageTimeoutMs = StrategyProbeNativeDeadlineMs (270_000ms)
- StrategyProbeFinalizationTimeoutMs (60_000ms)
= 330s total. It applies to theautomatic_auditanddpi_strategy(ru-dpi-strategyprofile) stages;ru-dpi-full(dpi_fullstage) uses a separate, shorterDpiFullStageTimeoutMs(240s). Quick-scan mode usesQuickScanStrategyProbeNativeDeadlineMs(60_000ms) instead of the full 270s deadline forru-dpi-strategy. The native engine's ownscan_deadline_msinScanRequestis set fromstageScanDeadlineMs(), which for strategy-probe stages returns the native deadline sub-budget directly (notstageTimeout - 30s) so the 60s finalization budget is reserved for report retrieval after the native deadline fires. Re-derive these numbers fromHomeCompositeStageDefinitions.kt` rather than citing the totals above once any of the underlying constants change.
Partial results recovery
When a strategy probe stage times out, ActiveScanRegistry.cancelActiveScan()
does not immediately discard results. Instead it polls for a 3-second grace
period, giving the native engine time to finish writing its current candidate
batch. If a partial report is available it is retrieved and persisted. This is
surfaced as StrategyProbeCompletionKind::PartialResults in the finalized
report, indicating the scan was interrupted mid-execution but produced usable
candidate data.
Recommendation output
StrategyRecommendationRunner selects the TCP and QUIC winners by quality
score, produces a StrategyProbeRecommendation with the winning candidate
IDs, labels, and recommended proxyConfigJson. An AuditAssessment is
attached with coverage metrics and confidence level (Low/Medium/High).
5. Diagnostics Catalog
The catalog defines what targets and profiles ship with the app.
Build-time generation
Located in build-logic/convention/src/main/kotlin/DiagnosticsCatalog*.kt:
DiagnosticsCatalogDomain.kt-- domain model:TargetPackDefinition,DiagnosticsProfileDefinition,CatalogScanKind, enums for profile families (GENERAL, WEB_CONNECTIVITY, MESSAGING, CIRCUMVENTION, etc.)DefaultDiagnosticsCatalogPackSource.kt--DefaultDiagnosticsCatalogPackSourcedefines target packs (e.g.,ru-independent-media,ru-global-platforms,ru-messaging,ru-circumvention,ru-throttling,neutral-control)DefaultDiagnosticsCatalogProfileSource.kt--DefaultDiagnosticsCatalogProfileSourcedefines profiles that reference packs and configure scan behaviorDiagnosticsCatalogAssembler.kt-- loads packs, loads profiles (with pack index for cross-referencing), validates, then renders to JSONDiagnosticsCatalogDefinitions.kt-- top-level entry point calling the assembler
How to add a new target pack
- Add a
TargetPackDefinitiontoDefaultDiagnosticsCatalogPackSource.load()with an id, version, and target lists (domain, DNS, TCP, QUIC, etc.) - Reference it from profiles via
index.requirePack("your-pack-id") - Run
./gradlew :core:diagnostics:generateDiagnosticsCatalogand commit the regeneratedcore/diagnostics/src/main/assets/diagnostics/default_profiles.jsonwith the Kotlin change (checkDiagnosticsCatalogfailscheckwhen it is stale)
How to add a new diagnostic profile
- Add a function to
DefaultDiagnosticsCatalogProfileSourcethat returns aDiagnosticsProfileDefinition - Include it in the
load()list - Regenerate and commit the catalog JSON as for a new pack
- Set
kind,family,executionPolicy,packRefs, and target lists - For strategy probe profiles: set
kind = CatalogScanKind.STRATEGY_PROBEand provide aStrategyProbeDefinitionwith the suite ID - Rebuild to regenerate
default_profiles.jsonasset
6. DiagnosticsHome Composite Run
The DiagnosticsHome run chains several profiles as stages: a full run (HomeCompositeStageSpecs) and a reduced quick scan (QuickScanStageSpecs), both defined in core/diagnostics/src/main/kotlin/com/poyka/ripdpi/diagnostics/HomeCompositeStageDefinitions.kt, which is the source of truth. Read references/composite-run.md for the per-stage profiles, kinds, and timeouts when changing a stage or its budget.
7. Kotlin Orchestration Layer
Call chain
DiagnosticsScanController.startScan()
-> ScanAdmissionService.admitManualStart() -- checks no active scan
-> DiagnosticsScanRequestFactory.prepareScan() -- builds PreparedDiagnosticsScan
-> BridgeExecutionService.createHandle() -- creates JNI bridge
-> BridgeExecutionService.start() -- calls bridge.startScan(requestJson)
-> DiagnosticsScanExecutionCoordinator.execute() -- launched in coroutine
-> BridgePollingService.awaitCompletion() -- polls progress + report
-> ScanFinalizationService.finalize() -- enriches, persists, applies policies
Key classes
| Class | Responsibility |
|---|---|
DefaultDiagnosticsScanController | Entry point for manual and automatic scans. Manages hidden probe conflicts. |
ScanAdmissionService | Guards against concurrent scans. Resolves profile from settings. |
ActiveScanRegistry | Tracks active bridges, execution jobs, cancellation state, fingerprints. Implements 3s grace-period partial results polling on cancel. |
BridgeExecutionService | Creates and destroys NetworkDiagnosticsBridge (JNI). |
BridgePollingService | Polls pollProgressJson()/takeReportJson() on interval. Timeout: 360s. |
ScanFinalizationService | Enriches report (classifier, resolver recommendation, strategy validation), persists results, applies remembered network policies. |
DiagnosticsScanWorkflow | Pure-logic orchestration: report enrichment, resolver override decisions, background auto-persist eligibility, network policy construction. |
RuntimeSessionCoordinator | Manages bypass usage sessions (not scan sessions). Tracks connection lifecycle, telemetry sampling, failure recording. |
DiagnosticsScanRequestFactory | Builds wire-format request JSON from profile + settings. |
Automatic probing
AutomaticProbeCoordinator and AutomaticProbeScheduler trigger background
scans on policy handover events. These use launchAutomaticProbe() on the
controller. Background probes run hidden (no UI progress) and auto-persist
results when audit confidence is high enough (coverage >= 75%, winner coverage
= 50%).
DNS-corrected re-probe
When a strategy probe is short-circuited by DNS tampering and a temporary resolver override is applied, the system automatically re-probes after a 2s delay to evaluate candidates with corrected DNS.
8. Logcat Capture
Logcat collection uses two capture scopes:
| Scope | Flag | Use case |
|---|---|---|
app_visible_snapshot | legacy (no -T) | Short scans where log rotation is not a concern |
time_bound_snapshot | -T <timestamp> | Long-running scans (e.g., strategy probe stages) |
The time_bound_snapshot scope passes the -T sinceTimestampMs flag set to
the earliest session start time across all active sessions. This prevents log
rotation loss on scans that exceed the default logcat ring-buffer window.
9. Wire Protocol
Rust and Kotlin communicate via JSON serialization over JNI. The wire types
mirror each other (EngineScanRequestWire, EngineProgressWire,
EngineScanReportWire, plus the embedded result/observation types); see
references/wire-protocol.md for the full type table and field-level detail.
The schema version constant is defined once, in the ripdpi-diagnostics-contracts
crate (native/rust/crates/ripdpi-diagnostics-contracts/src/wire.rs,
DIAGNOSTICS_ENGINE_SCHEMA_VERSION); ripdpi-monitor-engine's own wire.rs
only re-exports it. Kotlin mirrors it as DiagnosticsEngineSchemaVersion
(contract/engine/EngineContract.kt). Both must be equal; this is enforced
by contract tests. Re-verify the current value with
grep -n DIAGNOSTICS_ENGINE_SCHEMA_VERSION native/rust/crates/ripdpi-diagnostics-contracts/src/wire.rs
rather than trusting a copied number -- it is 9 as of this writing and has
moved before.
The ScanRequest field scan_deadline_ms is optional. The Rust engine's own
internal default when it is absent is 360,000ms (360s -- the same value as
the is_past_deadline() hard deadline in "Cancellation and deadlines"
above), not a stage-timeout-derived number. A caller going through the
DiagnosticsHome composite run instead gets an explicit value from Kotlin's
stageScanDeadlineMs() (see "Stage timeouts" in section 4), which for most
non-strategy-probe stages is approximately stageTimeout - 30s but for
automatic_audit/ru-dpi-strategy is the native deadline sub-budget
directly -- do not conflate the two defaults.
10. Adding a New Probe Type
End-to-end walkthrough for adding a hypothetical "ping" probe:
Rust side
- Define target type in
types/target.rs:pub struct PingTarget { pub host: String, pub count: u8 } - Add to ScanRequest in
types/request.rs:pub ping_targets: Vec<PingTarget>, - Add wire mapping in
wire.rs-- add field toEngineScanRequestWire, updateFrom<EngineScanRequestWire> for ScanRequest. - Add ProbeTaskFamily variant in
types/request.rs:Ping, - Create execution stage -- add
ExecutionStageId::Pinginengine/runtime.rs. CreatePingRunnerimplementingExecutionStageRunnerinengine/runners/connectivity.rs. - Register runner in
engine/runners/mod.rs:Box::new(PingRunner), - Add stage to plan in
engine/plan.rs-- addProbeTaskFamily::Ping => ExecutionStageId::Pingmapping and include in the default connectivity order. - Add observation mapping in
observations.rsif structured facts are needed. - Update contract fixtures -- add outcome tokens, update field manifests.
Kotlin side
- Add target model in
core/diagnostics/.../Models.kt. - Mirror wire types in
contract/engine/EngineContract.kt-- add toEngineScanRequestWireandEngineProbeTaskFamily. - Add to catalog domain in
DiagnosticsCatalogDomain.kt-- add target definition type and include inTargetPackDefinition. - Update catalog rendering to serialize the new target list.
- Add contract fixture entries and update golden tests.
Tests
- Add outcome tokens to
outcome_taxonomy_current.json. - Rust:
contract_fixtures.rswill catch missing outcomes. - Kotlin:
DiagnosticsWireContractTestwill catch field mismatches.
11. Adding a New Diagnostic Profile
- Open
build-logic/convention/src/main/kotlin/DefaultDiagnosticsCatalogProfileSource.kt. - Add a private function returning
DiagnosticsProfileDefinition:private fun myNewProfile(index: DiagnosticsCatalogIndex): DiagnosticsProfileDefinition { val myPack = index.requirePack("my-pack") return DiagnosticsProfileDefinition( id = "my-new-profile", name = "My New Profile", family = CatalogDiagnosticProfileFamily.GENERAL, executionPolicy = policy(manualOnly = false, allowBackground = false, requiresRawPath = false), domainTargets = myPack.domainTargets, // ... ) } - Add it to the
load()list. - If you need a new target pack, add it to
build-logic/convention/src/main/kotlin/DefaultDiagnosticsCatalogPackSource.kt:DefaultDiagnosticsCatalogPackSource. - Rebuild. The catalog assembler validates uniqueness, pack references, and schema constraints.
- Update
diagnostics-contract-fixtures/profile_catalog_current.json: follow thegolden-test-managementskill, which requires explicit user authorization for the fixture family.
12. Testing
Golden contract tests
The contract test framework ensures Rust and Kotlin stay in sync:
- Shared fixtures in
contract-fixtures/(repo root) -- schema version, field manifests for progress and report types. - Diagnostics fixtures in
diagnostics-contract-fixtures/-- full wire payloads for request, report, progress, profile catalog, outcome taxonomy. - Rust side (
tests/contract_fixtures.rs): decodes all shared fixtures, verifies schema version matches, checks outcome tokens cover all emitted probe outcomes. - Kotlin side (
DiagnosticsContractGovernanceTest.kt): decodes the same fixtures, verifies schema version, checks bundled catalog matches fixture. - Wire field tests (
DiagnosticsWireContractTest.kt): compares field paths between Rust-produced manifests and Kotlin serialization.
Golden file support
GoldenContractSupport.kt provides assertJsonGolden() and
assertTextGolden(). To regenerate, follow the golden-test-management skill, which requires explicit user authorization for the fixture family. Diff
artifacts are written to core/diagnostics/build/golden-diffs/.
Key test classes
| Test | What it verifies |
|---|---|
DiagnosticsWireContractTest | Field-level compatibility between Rust and Kotlin wire types |
DiagnosticsContractGovernanceTest | Schema versions match, fixtures decode, catalog matches asset |
DiagnosticsScanWorkflowTest | Strategy probe enrichment, background eligibility, policy construction |
DiagnosticsScanControllerTest | Scan lifecycle: start, cancel, hidden probe conflict resolution |
DiagnosticsScanExecutionCoordinatorTest | Execution flow: polling, finalization, DNS-corrected re-probe |
DiagnosticsScanRequestFactoryTest | Wire request construction from profiles and settings |
DiagnosticsModelsCompatibilityTest | Model serialization round-trip stability |
Wire compatibility verification
After any wire type change:
- Run
cargo test --locked -p ripdpi-monitor-engine-- catches fixture decode failures and outcome token coverage gaps. - Run
:core:diagnostics:testDebugUnitTest-- catches field manifest and schema version mismatches. - If fields were added: follow the
golden-test-managementskill, which requires explicit user authorization for the fixture family, then commit the updated fixtures.
13. PCAP Storage and Export
core/diagnostics/src/main/kotlin/com/poyka/ripdpi/diagnostics/export/DiagnosticsArchiveExporter.kt:createArchive() rejects includePcap: raw packet captures must be exported separately from redacted diagnostic archives.
core/diagnostics/src/main/kotlin/com/poyka/ripdpi/diagnostics/export/DiagnosticsArchiveFileStore.kt:cleanupPcapFiles() owns retention, byte/file limits, and protection of active capture sets. Derive those limits from its constants when changing storage behavior.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |