jedafe/japa / .claude/skills/jc-backend-develop
jc-backend-develop
UnverifiedUse when building new features, endpoints, services, or entities in the JapaConnect Go backend (github.com/jedafe/japa). Covers the layered architecture (Handlers→Services→Stores), the Spec-Readiness and Contract-Delivery Gates, full RBAC CRUD delivery per entity, sqlc/pgx patterns, and catalog maintenance.
JapaConnect Backend — Development Skill
Go monolith serving all five frontend apps at /api/v1. Module: github.com/jedafe/japa.
⛔ GATE 1 — SPEC-READINESS (Run Before Any Implementation)
Before writing code for any feature:
- Locate the governing JEOS section doc and/or ADR for the feature.
- If no spec exists, or the spec conflicts with what's being asked: STOP and output:
⚠️ MISSING/CONFLICTING SPEC — implementation paused
Feature: [name]
Issue: [no JEOS section covers this | conflicts with ADR-0XX | conflicts with section XX]
Action needed: create/update the JEOS spec (and ADR if architectural) before backend work continues.
Architectural decisions made mid-implementation without an ADR are spec drift — the exact failure mode JEOS audits exist to catch.
⛔ GATE 2 — CONTRACT-DELIVERY (The Mirror of the Frontend Gates)
The frontend agents STOP when routes are missing. Your job is to make sure they never stop. Every new entity ships with the complete contract or it doesn't ship:
- Full RBAC CRUD route set (non-negotiable — super-admin must be able to manage every entity):
GET /api/v1/admin/[entity]— list with?page=&limit=&sort=&q=&filter=(offset pagination + meta for admin tables)GET /api/v1/admin/[entity]/:id·POSTcreate ·PATCH /:idupdate ·DELETE /:id- Both directions of every state transition:
POST /:id/suspendAND/:id/unsuspend,/banAND/unban,/approveAND/reject, etc. - Bulk endpoints where the DataTable spec needs them:
POST /api/v1/admin/[entity]/bulkwith{action, ids} - Readonly/log entities still get
DELETE /:idandPOST /clear(all / older-than)
- RBAC permission nodes seeded — one node per action (
[entity].list,.view,.create,.update,.delete,.suspend,.ban, …) added to the seed migration AND registered in the RBAC middleware. Regression anchor BE-005: 174 of 249 nodes were once missing from seed. Never partially seed. - Catalogs updated in the same PR —
api-routes-catalog.md(new routes),valkey-keys.md(newjc:keys),system-settings.md(new keys),asynq-tasks.md(new tasks). An uncatalogued route is invisible to the frontend gates and therefore doesn't exist.
If any part of the contract can't be delivered (e.g. spec ambiguity on a state transition): STOP + output:
⛔ INCOMPLETE CONTRACT — implementation paused
Entity: [entity]
Undeliverable: [e.g. unclear whether orgs support suspend; bulk action semantics undefined]
Action needed: [decision required]
Architecture (Layered — Never Skip Layers)
Handler (Chi) → Service (business logic) → Store (sqlc-generated) → PostgreSQL 18
↓ ↓
Middleware chain Asynq enqueue (side effects)
- Handlers: parse/validate input, call ONE service method, render response. No business logic, no direct store calls.
- Services: business logic, transaction boundaries, orchestration across stores, enqueue Asynq tasks.
- Stores: sqlc-generated only. Hand-written SQL goes in
.sqlquery files →sqlc generate. Never string-concatenate SQL. - Middleware order: RequestID → RealIP → Logger → Recoverer → Tenant → GeoIP → VisitorContext → Auth → RBAC → RateLimit → Handler.
- GeoIP (Section 37): MaxMind GeoLite2 in-process, attaches
GeoContext{CountryCode, City, ...}to ctx. - VisitorContext (ADR-064): runs immediately after GeoIP; synthesizes
GeoContext+user_profiles.field_values(japa_status,destination_country) + resolved language intoVisitorContext{OriginCountry, CurrentCountry, DestinationCountry, JapaStage, Archetype, ContentCountry, Confidence}. Cached atjc:visitor:ctx:{session_id}(30-min TTL). Gated bysystem_settings.geo_personalization.enabled.
- GeoIP (Section 37): MaxMind GeoLite2 in-process, attaches
Stack (Non-Negotiable)
Go (latest stable) · Chi router · sqlc + pgx/v5 (pool via pgxpool) · Asynq (Valkey-backed, three tiers: critical/default/low) · PostgreSQL 18 · Valkey (keys namespaced jc:) · Meilisearch 1.45 · SeaweedFS (S3 API) · imgproxy · Honcho (AI memory, separate logical DB, same PG instance) · GlitchTip via sentry-go · golang-migrate · slog structured logging · env vars JC_ prefixed, typed config, missing-required exits at boot.
Response Contract (What the Frontends Consume)
- Success:
RenderJSON(w, status, data)· Lists:RenderPaginated(w, data, meta{total,page,limit})or cursor meta for feeds - Errors:
RenderError(w, status, code, message)— machine code + human message; the frontend surfacesmessage - Validation errors: 422 with per-field map · AuthN 401 · AuthZ 403 · not found 404 — never 500 for expected conditions
- All timestamps RFC 3339 UTC · IDs as strings in JSON
Mandatory Patterns
- Wallet/credits: append-only immutable ledger, balance = SUM; every spend inside a tx holding
pg_advisory_xact_lock(user_id). Never a mutable balance column. (Regression anchor: double-spend defect.) - Revenue splits (ADR-063):
revenue_splitsis an immutable audit record — append-only, never UPDATE. Each paid enrollment or Pro-stream accrual writes one row with the exact percentages/amounts applied at the time.academy:process_enrollmentis idempotent on payment_id (guard against double-credit, cf. AUDIT-BE-005). Payout execution (academy:send_instructor_payout) mirrors the wallet double-spend guard: advisory lock on instructor_id inside the tx, stamp splits withpayout_idonly after transfer succeeds. - Render-time resolution (ADR-052): coupon embeds/affiliate links store entity IDs only; resolve codes/URLs/expiry at render time; fail closed on inactive.
- Idempotency: webhook handlers and payment callbacks guard with Valkey
SET NXon an idempotency key. - Geo-aware personalization (ADR-064):
VisitorContext.ContentCountryis the default for all?geo=surfaces (trending §53, search §13, feed §32, discovery). Explicit?geo=always wins; absent param →ContentCountry. Feed score gets a bounded geo-relevance multiplier (cap so it re-weights, never dominates). Search passesContentCountryas a boost/optional-filter on therelevant_countriesMeilisearch attribute. Never hard-filter by geo — re-weight only. User override:user_profiles.content_country_preference(nullable ISO code) wins over resolved. Killswitch:system_settings.geo_personalization.enabled. - University space type (Section 39): universities are
space_type = 'university'spaces with theuniversity_fieldscustom field group. Auto-seeded on first deploy fromdb/seed/universities.json; logo fetch viauniversity:fetch_logo(low tier). Admin CRUD + CSV bulk import. Discovery routes at/api/v1/universities(public) and/api/v1/admin/universities(admin). Thelogofield doubles as the university page's profile image (Section 34 media + imgproxy). - Uploads: MIME sniffed from content (not extension), size-capped, stored to SeaweedFS, served via imgproxy URLs.
- Tenancy: tenant middleware resolves host → tenant_id into context; stores filter by it.
- GDPR/NDPR erasure: user deletion cascades through owned data AND calls the Honcho deletion API — required, not best-effort. Japa Academy:
academy_instructor_profiles,revenue_splits, andacademy_instructor_payoutsare user-owned — erasure must cascade through them.revenue_splitsare financial audit records: anonymize the instructor reference, never delete the split row (regulatory retention). - Search: writes that affect searchable entities enqueue a Meilisearch sync task (default queue) — never sync inline. Content documents gain a
relevant_countriesfilterable attribute (array of ISO codes from geo spaces/tags) for ADR-064 search weighting. - Errors: wrap with
fmt.Errorf("...: %w", err); unexpected errors captured to GlitchTip with request context. (Regression anchor BE-002: GlitchTip must actually be wired.) - TipTap content: stored as JSON, rendered to HTML server-side, cached in Valkey.
Testing Expectations
Table-driven tests for services · httptest for handlers (auth + RBAC denial cases included) · a failing-permission test per new permission node · store tests against a real PG (dockertest or test DB) for tx-sensitive logic (wallet, claims).
Japa Academy Entities (ADR-063 — Section 36 extension)
When building Japa Academy creator-economy features, the entity set is:
academy_instructor_profiles— one per user per tenant.status ∈ pending|approved|rejected|suspended(gated application).tier ∈ standard|verified|partner. Optionalcommission_pct_override.payout_method(jsonb, encrypted at app layer).kyc_status. Full RBAC CRUD + both-direction state transitions (approve/reject, suspend/unsuspend).revenue_splits— immutable per-event record (creator_pct/platform_pct, amounts,source ∈ purchase|pro_stream). Links toenrollment/paymentin andpayoutout. Append-only — never UPDATE.academy_instructor_payouts— one per instructor per period; gross/commission/net;status ∈ pending|processing|paid|failed. State transitions:pending→processing→paidor→failed.academy_trackscolumns:instructor_id(null = platform course),price_kobo(null = free/Pro-only),currency,is_published_to_academy(marketplace visibility, distinct fromis_published),category,language,total_enrolled(cached counter).enrollmentscolumn:payment_id(nullable FK →payments; null for free/Pro/pro_stream).- Course reviews: reuse polymorphic
reviewstable (Section 29) withentity_type = 'academy_track'. Nocourse_reviewstable. - Migration:
000166(Japa Academy tables + columns),000167(user_profiles.content_country_preferencefor geo). - 5 new Asynq tasks:
academy:process_enrollment(critical),academy:calculate_period_payouts(scheduled, low),academy:send_instructor_payout(critical),academy:notify_instructor_approved(default),academy:notify_course_approved(default). Plusuniversity:fetch_logo(low) for university space seeding. - 12 new RBAC nodes:
university.instructor.*,university.course.*,university.payout.*— seed completely (BE-005). - 7 new settings:
university.default_commission_pct,university.verified_tier_commission_pct,university.partner_tier_commission_pct,academy.pro_stream_rate_kobo,university.payout_min_threshold_kobo,university.payout_period_days,university.surface_domain. - 4 new Valkey keys:
jc:visitor:ctx:{session_id}(30-min TTL, shared with geo),jc:uni:instructor:{user_id}(instructor profile cache),jc:uni:payout:period:{instructor_id}:{period}(payout lock),jc:uni:enrollment:processed:{payment_id}(idempotency guard).
Deliverables Per Task
- Gate 1 + Gate 2 output · 2. migration pair if schema changes (invoke
jc-backend-migrate) · 3. sqlc queries +sqlc generate· 4. store/service/handler code · 5. routes registered + RBAC nodes seeded · 6. Asynq tasks (invokejc-backend-worker) · 7. all catalogs updated · 8. tests · 9.go build ./... && go vet ./...clean (run locally via Claude Code — no Go toolchain in chat sessions).
V2 Mandatory Patterns (Phases 41–50)
AI Personas (§68, ADR-068)
- Persona provisioning is transactional: create
users(is_bot=true) +professional_profiles+ai_personasin one tx. Partial provision is a data integrity failure. disclosure_modecontrols public rendering only.is_bot=trueis permanent — never flip it, never expose it publicly.- Every persona action writes
platform_events.is_synthetic = true. This is non-negotiable and cannot be disabled by any setting.
RAG Layer (§69, ADR-069)
content:generate_embeddingUPSERTS intocontent_embeddings(never raw INSERT — re-indexing a document must replace its chunks, not add duplicates).- Text extracted from TipTap JSON before embedding — strip block structure to plain text first (utility function, not inline in task).
- Retrieval at
/ai/chatis gated byrag.enabledsystem setting — fall back to non-RAG LLM call if false. - Persona-authored content is indexed exactly like human-authored content (it passed HITL review).
Content Studio (§75, ADR-076)
content.generate_draftwithstudio_job_idmust be idempotent on that ID — guard against re-running an already-generated job.- AI-populated custom fields: if required fields are missing from model output, set
field_validation_warning = trueon the job. Never block publish — the human reviewer has final say. studio.auto_publish_typesis checked at routing time. Guide categoriesvisa,banking,grants,health_insurance,admissionare hardcoded to always require review — this check must be in code, not configurable.
Participation Orchestration (§76, ADR-077)
- Orchestration actions are dispatched as the persona's backing account through existing action endpoints — no new action routes. The backing account must have the correct RBAC permission nodes to perform each action type.
- Daily action counters: Valkey keys
jc:persona:daily:{persona_id}:{date}:{action_type}with TTL until midnight UTC. Counter overflow (exceeding budget) = skip the action entirely, do not error. - Run-lock
jc:persona:lock:{persona_id}(5-min TTL SET NX) prevents concurrent orchestration runs for the same persona. - High-stakes gate is in task code: check parent content's guide_category or system tags BEFORE generating action content — don't waste AI gateway tokens on content that will route to review anyway.
Remittance Tracker (§74, ADR-075)
api_configJSONB onremittance_providerscontains API credentials — encrypt at application layer (same pattern as payment provider secrets, §17). Never log or expose in responses.- Rate history is append-only (new row per fetch).
remittance:prune_old_ratesis the cleanup mechanism — never delete rows inline in the fetch task. - Provider adapter interface (
RateProvider) must have a timeout and return an error on rate-fetch failure — the main task catches errors per-provider and logs to GlitchTip, never fails the whole batch.
Newsletter (§62, ADR-080)
- Unsubscribe is token-based (
newsletter_unsubscribe_tokens) — no auth required. Token must be single-use (delete after use) and short-circuit on invalid/expired token (404, not error). - Bounce and complaint webhooks from the email provider MUST update
newsletter_subscribers.status— undeliverable addresses that keep receiving email are a deliverability risk.
V2 Community Features & Functional Tools (Phases 48–50, §§79–102)
Geo-Aware Tool Pattern (§§79–101, .jeos/skills/geo-aware-tool-pattern.md)
- Destination country is the primary data axis — every v2 functional tool (§§90–101) filters by destination country. No hardcoded nationality anywhere in query/store/service code.
?dest=filter on all list endpoints for §§79–101 — public and admin. The filter accepts an ISO 3166-1 alpha-2 country code. Absent?dest=returns global/cross-destination results where sensible, but the filter must always be honored when present.destination_country_idin schema — every v2 functional tool table includes adestination_country_idFK tocountries. This is the join axis for geo-filtering, not a nullable attribute.- Destination selector is the first UI input — the frontend renders a destination country picker before any other filter on v2 tool surfaces. The backend must accept
destination_country_idas the primary query parameter and return 400 if it's required but absent (tool-specific — some tools allow browsing without a destination, others require it). - Feature-flag middleware returns 404 (not 403) when disabled (ADR-042): when a v2 feature flag is off, the middleware returns 404 Not Found, not 403 Forbidden. This prevents leaking the existence of disabled features to unauthorized users. Verify the feature-flag middleware path returns 404 for all §§79–102 routes.
- RBAC seed migration naming:
000NNN_seed_{feature_domain}_rbac_nodes.sql— each v2 feature domain gets its own dedicated RBAC seed migration with this naming convention (e.g.000180_seed_story_rbac_nodes.sql,000181_seed_wiki_rbac_nodes.sql). Never batch multiple feature domains into one seed migration. - Admin list endpoints for all v2 entities support
?filter=&sort=&paginate=(filter + sort + paginate) — same DataTable contract as v1 admin endpoints. No v2 admin list endpoint ships without all three.
ADR-088 Universal AI Content Generation
- Standard columns on all admin-managed content tables for §§79–101:
ai_generated(boolean, default false),ai_model(text, nullable — model identifier when AI-generated),hitl_reviewed_by(uuid, nullable FK → users — the admin who reviewed),ai_generated_at(timestamptz, nullable). These four columns are mandatory on any table where content can be AI-drafted. - AI generation via existing §20 Gateway — no new AI client or provider wiring. All AI content generation (story drafts, wiki article drafts, scam alert summaries, digest text, etc.) goes through the existing AI Gateway abstraction. Never call an LLM provider directly from a service or task.
- HITL review is mandatory before publish — AI-generated content cannot be published without a human reviewer. The
hitl_reviewed_bycolumn must be non-null andhitl_reviewed_atset beforeis_published = true. The publish endpoint must reject ifai_generated = true AND hitl_reviewed_by IS NULL. is_japabot_indexedon RAG-retrievable content — content that should appear in Japabot search results (wiki pages, guides, stories) getsis_japabot_indexed = trueafter HITL approval. This flag gates thecontent:generate_embeddingtask — unindexed content is never embedded.
Community Features Reusing contents Table (§§79–89, §98)
- Community features (Community Story Wall §79, Achievement Feed §88, Service Directory §98) reuse the existing
contentstable withcontent_typediscriminators (story,achievement,service_listing) — do not create separate top-level tables for these content types. - Extension tables (
story_meta,achievement_meta,service_listing_meta) hold type-specific columns and link back tocontents.id. The discriminator column oncontentsdrives which extension table to join. - This pattern keeps the content pipeline (search indexing, embedding, geo-tagging, media) unified — no parallel infrastructure for community content.
Mentorship Matching (§102, ADR-110)
active_mentee_countis maintained by a PostgreSQL trigger (trg_mentee_count), not computed at query time. The trigger fires on INSERT/UPDATE/DELETE ofmentorship_connectionswherestatus = 'active'and adjusts the mentor'sactive_mentee_countaccordingly. Never compute this count with a subquery orCOUNT(*)in a read path — read the cached column.- Match scoring: 60% keyword overlap + 40% pgvector cosine similarity. The matching algorithm combines keyword overlap (on mentor/mentee profile tags and bio tokens) with pgvector cosine similarity (on profile embeddings). The weights are 60/40 — verify the scoring function applies these exact weights, not configurable ones.
- Asynq tasks:
mentor:generate_embedding(low queue — generates pgvector embedding from mentor/mentee profile text) andmentor:update_meilisearch(low queue — syncs mentor profiles to Meilisearch for search). Both must be registered and wired. - DM auto-open on accept (§48 integration): when a mentorship connection transitions to
status = 'accepted', a DM conversation is auto-created and auto-opened for both parties. This is the §48 DM system — no new messaging infrastructure.
New Asynq Tasks for §§79–102
| Task | Queue | Schedule | Notes |
|---|---|---|---|
visa_timeline:refresh_stats | default | nightly | Recomputes visa timeline aggregate stats |
visa_timeline:prompt_update | default | weekly | Prompts users with stale timelines to update |
digest:generate | default | weekly | Generates community digest draft content |
digest:send_email | default | on publish | Sends published digest to subscribers |
mutual_aid:expire | default | nightly | Expires stale mutual aid posts |
calendar:staleness_check | default | nightly | Flags calendar entries not updated within threshold |
calendar:send_reminders | default | daily | Sends upcoming event reminders |
planning_group:expire | default | as needed | Expires inactive planning groups |
planning_group:checkin_reminder | default | as needed | Sends check-in reminders to group members |
visa_routes:check_links | default | weekly | Validates external links on visa route pages |
embassy:poll_{slug} | critical | per embassy | Polls a specific embassy's appointment availability |
embassy:dispatch_alerts | critical | as needed | Sends appointment availability alerts to subscribers |
credentials:check_links | default | weekly | Validates external links on credential checker pages |
cv_format:process | default | on demand | Processes a CV formatting job |
cv_format_sessions:purge | low | nightly | Purges expired CV format sessions |
mentor:generate_embedding | low | on profile change | Generates pgvector embedding from profile text |
mentor:update_meilisearch | low | on profile change | Syncs mentor profiles to Meilisearch |
New Valkey Keys for §§79–102
| Key | TTL | Purpose |
|---|---|---|
jc:cv_quota | rolling window | CV formatting quota per user (rate limit) |
jc:embassy_avail | short (per poll cycle) | Cached embassy appointment availability |
jc:digest_draft | draft lifecycle | Community digest draft-in-progress lock |
jc:wiki_edit_lock | edit session timeout | Prevents concurrent edits on a wiki page |
jc:ama_active | AMA session lifetime | Marks an AMA session as active/live |
New RBAC Nodes for §§79–102
48 v2 RBAC nodes total across the following domains: story.*, wiki.*, scam_alert.*, planning_group.*, visa_timeline.*, digest.manage, mutual_aid.*, calendar.*, profile.japa_stage.edit, forum.ama.*, achievement.*, visa_routes.manage, appointment_alert.*, embassy_monitor.manage, arrival_checklist.manage, credential_checker.manage, cv_formatter.*, housing_guide.manage, attestation_guide.manage, service.*, language_test_hub.manage, tax_calculator.manage, tax_content.review, community_locator.*, mentorship.*. Total RBAC node count: 297 (249 v1 + 48 v2). Seed completely per feature domain (BE-005).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 117k | Updated | Claude CodeCodex | Unverified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 117k | Updated | Claude CodeCodex | Unverified | Install |