Skip to content

plurigrid/asi / ies/music-topos/.claude/skills/compositional-acset-comparison

compositional-acset-comparison

Verified

Compare data structures (DuckDB, LanceDB) via ACSets with persistent homology coverage analysis and geometric morphism translation.

FileKindSize
ColoringFunctor.jlother14.3 KB
ComparisonUtils.jlother15.8 KB
DuckDBACSet.jlother9.5 KB
GeometricMorphism.jlother18.1 KB
GhristCoverage.jlother15.5 KB
IrreversibleMorphisms.jlother16.7 KB
LanceDBACSet.jlother15.4 KB
SKILL.mdskill md16.6 KB
SideBySideComparison.jlother29.0 KB

GhristCoverage.jl

View on GitHub
# GhristCoverage.jl - Persistent Homology for Schema Coverage Analysis
# Based on de Silva & Ghrist "Coverage in Sensor Networks via Persistent Homology"
# Gay.jl Seed: 4000000
#
# AM Radio Coverage Mechanism Design Analogy:
#   - Radio stations = Schema objects (Table, Column, Segment, ...)
#   - Coverage radius = Morphism composability range  
#   - Signal overlap = Translatable concepts between schemas
#   - Dead zones = Irreversible information loss (coverage holes)
#   - Persistent homology = Detecting stable holes across scales

using ACSets, Catlab

include("DuckDBACSet.jl")
include("LanceDBACSet.jl")

# ═══════════════════════════════════════════════════════════════════════
# GHRIST KEY INSIGHT: Rips Complex from Communication Graph
# ═══════════════════════════════════════════════════════════════════════

"""
From Ghrist paper:
- Čech complex captures topology of union of cover discs (expensive)
- Rips complex is computable from pairwise communication data
- Trade computability for accuracy

For schema comparison:
- Objects are "sensor nodes" 
- Morphisms define "communication links"
- Rips complex at radius ε = concepts reachable within ε morphism hops
"""

# ═══════════════════════════════════════════════════════════════════════
# AM RADIO COVERAGE MODEL
# ═══════════════════════════════════════════════════════════════════════

"""
AM Radio Station = Schema Object

Each object covers a "semantic region":
- Table: covers relational data concepts
- Column: covers attribute semantics  
- VectorColumn: covers embedding semantics
- Manifest: covers version history

Coverage radius rₖ varies by object type
"""
struct RadioStation
    name::Symbol
    coverage_radius::Float64
    frequency::Float64  # Gay.jl color hue
    schema::Symbol      # :DuckDB or :LanceDB
end

const DUCKDB_STATIONS = [
    RadioStation(:Table, 1.0, 0.0, :DuckDB),        # #EE2B2B
    RadioStation(:RowGroup, 0.8, 137.51, :DuckDB),  # #2BEE64
    RadioStation(:Column, 0.7, 275.02, :DuckDB),    # #9D2BEE
    RadioStation(:Segment, 0.6, 52.52, :DuckDB),    # #EED52B
    RadioStation(:Vector, 0.5, 190.03, :DuckDB),    # #2BCDEE
    RadioStation(:DataChunk, 0.4, 327.54, :DuckDB), # #EE2B94
]

const LANCEDB_STATIONS = [
    RadioStation(:Database, 1.0, 0.0, :LanceDB),       # #EE2B2B
    RadioStation(:Table, 0.9, 137.51, :LanceDB),       # #2BEE64
    RadioStation(:Manifest, 0.8, 275.02, :LanceDB),    # #9D2BEE (UNIQUE!)
    RadioStation(:Fragment, 0.7, 52.52, :LanceDB),     # #EED52B
    RadioStation(:Column, 0.6, 190.03, :LanceDB),      # #2BCDEE
    RadioStation(:VectorColumn, 0.5, 327.54, :LanceDB),# #EE2B94 (UNIQUE!)
    RadioStation(:VectorIndex, 0.4, 105.05, :LanceDB), # #5BEE2B (UNIQUE!)
]

# ═══════════════════════════════════════════════════════════════════════
# RIPS COMPLEX CONSTRUCTION
# ═══════════════════════════════════════════════════════════════════════

"""
Rips Complex at radius ε:
- 0-simplices: all objects
- 1-simplices: pairs within distance ε (morphism path length)
- k-simplices: (k+1)-tuples pairwise within ε

From paper: "Rips complex is ideally suited to communication networks"
"""
struct RipsComplex
    vertices::Vector{Symbol}
    edges::Vector{Tuple{Symbol, Symbol}}
    triangles::Vector{Tuple{Symbol, Symbol, Symbol}}
    radius::Float64
end

"""
Build Rips complex from schema morphism graph at radius ε
"""
function build_rips_complex(schema, ε::Float64)
    # Get all objects as vertices
    vertices = collect(obs(schema))
    
    # Get edges from morphisms (distance 1)
    edges = Tuple{Symbol, Symbol}[]
    for hom in homs(schema)
        d = dom(hom)
        c = codom(hom)
        if 1.0 <= ε
            push!(edges, (d, c))
        end
    end
    
    # Find triangles (3 mutually connected vertices)
    triangles = Tuple{Symbol, Symbol, Symbol}[]
    for (i, v1) in enumerate(vertices)
        for (j, v2) in enumerate(vertices)
            if j <= i continue end
            for (k, v3) in enumerate(vertices)
                if k <= j continue end
                # Check if all three pairs are edges
                e12 = (v1, v2) ∈ edges || (v2, v1) ∈ edges
                e23 = (v2, v3) ∈ edges || (v3, v2) ∈ edges
                e13 = (v1, v3) ∈ edges || (v3, v1) ∈ edges
                if e12 && e23 && e13
                    push!(triangles, (v1, v2, v3))
                end
            end
        end
    end
    
    RipsComplex(vertices, edges, triangles, ε)
end

# ═══════════════════════════════════════════════════════════════════════
# BETTI NUMBERS (Coverage Holes)
# ═══════════════════════════════════════════════════════════════════════

"""
Betti numbers measure topological features:
- β₀: Connected components (isolated coverage regions)
- β₁: 1-dimensional holes (coverage gaps/loops)
- β₂: 2-dimensional voids (enclosed dead zones)

For schemas:
- β₀ > 1: Schema has disconnected subsystems
- β₁ > 0: Schema has information flow cycles with holes
- β₂ > 0: Schema has enclosed regions unreachable from outside
"""
struct BettiNumbers
    β0::Int  # Components
    β1::Int  # Holes
    β2::Int  # Voids
end

"""
Compute Betti numbers from Rips complex (simplified Euler characteristic method)
"""
function compute_betti(rips::RipsComplex)
    # Euler characteristic: χ = V - E + F
    V = length(rips.vertices)
    E = length(rips.edges)
    F = length(rips.triangles)
    
    # For connected complex: β₀ = 1, β₁ = E - V + 1, β₂ from higher structure
    # Simplified: assume connected
    β0 = 1
    β1 = max(0, E - V + 1 - F)  # Holes filled by triangles
    β2 = 0  # Would need tetrahedra to compute
    
    BettiNumbers(β0, β1, β2)
end

# ═══════════════════════════════════════════════════════════════════════
# PERSISTENT HOMOLOGY (Multi-scale Coverage)
# ═══════════════════════════════════════════════════════════════════════

"""
Persistent Homology tracks Betti numbers across scales:

From Ghrist: "Homology classes which persist as one changes a parameter"

For schemas:
- Increase radius ε from 0 → ∞
- Track when holes appear (birth) and disappear (death)
- Long-lived holes = fundamental structural gaps
- Short-lived holes = noise/artifacts

IRREVERSIBLE MORPHISMS create persistent holes:
- parent_manifest: creates hole in version traversal (can't go back)
- source_column: creates hole in information recovery (lossy)
"""
struct PersistenceDiagram
    births::Vector{Float64}
    deaths::Vector{Float64}
    dimension::Int
end

"""
Compute persistence diagram by sweeping radius
"""
function compute_persistence(schema; radii=0.0:0.1:3.0)
    diagrams = Dict{Int, PersistenceDiagram}()
    
    prev_betti = nothing
    births = Dict{Int, Vector{Float64}}()
    deaths = Dict{Int, Vector{Float64}}()
    
    for ε in radii
        rips = build_rips_complex(schema, ε)
        betti = compute_betti(rips)
        
        if prev_betti !== nothing
            # Track β₁ changes (holes)
            if betti.β1 > prev_betti.β1
                # Hole born
                for _ in 1:(betti.β1 - prev_betti.β1)
                    push!(get!(births, 1, Float64[]), ε)
                end
            elseif betti.β1 < prev_betti.β1
                # Hole died
                for _ in 1:(prev_betti.β1 - betti.β1)
                    push!(get!(deaths, 1, Float64[]), ε)
                end
            end
        end
        
        prev_betti = betti
    end
    
    # Match births to deaths
    b1 = get(births, 1, Float64[])
    d1 = get(deaths, 1, Float64[])
    while length(d1) < length(b1)
        push!(d1, Inf)  # Never dies
    end
    
    diagrams[1] = PersistenceDiagram(b1, d1, 1)
    diagrams
end

# ═══════════════════════════════════════════════════════════════════════
# COVERAGE COMPARISON: DuckDB vs LanceDB
# ═══════════════════════════════════════════════════════════════════════

"""
Compare coverage between two schemas using persistent homology

From Ghrist Main Theorem:
"The sensor cover U contains D - N̂ᵣ(∂D) if the homomorphism
 ι*: Hd(Rs, Fs) → Hd(Rw, Fw) induced by inclusion is nonzero"

Translation for schemas:
- If concepts persist from DuckDB to LanceDB, translation is possible
- If holes appear in the mapping, information is lost
- Irreversible morphisms create persistent holes
"""
function compare_schema_coverage()
    # Build Rips complexes at multiple radii
    duckdb_rips_1 = build_rips_complex(SchDuckDB, 1.0)
    duckdb_rips_2 = build_rips_complex(SchDuckDB, 2.0)
    lancedb_rips_1 = build_rips_complex(SchLanceDB, 1.0)
    lancedb_rips_2 = build_rips_complex(SchLanceDB, 2.0)
    
    # Compute Betti numbers
    duckdb_betti_1 = compute_betti(duckdb_rips_1)
    duckdb_betti_2 = compute_betti(duckdb_rips_2)
    lancedb_betti_1 = compute_betti(lancedb_rips_1)
    lancedb_betti_2 = compute_betti(lancedb_rips_2)
    
    Dict(
        :duckdb => Dict(
            :rips_1 => duckdb_rips_1,
            :rips_2 => duckdb_rips_2,
            :betti_1 => duckdb_betti_1,
            :betti_2 => duckdb_betti_2,
            :unique_objects => [:RowGroup, :Segment, :SegmentState, :CompressionAlgo],
            :irreversible_morphisms => 0,
            :coverage_holes => duckdb_betti_1.β1
        ),
        :lancedb => Dict(
            :rips_1 => lancedb_rips_1,
            :rips_2 => lancedb_rips_2,
            :betti_1 => lancedb_betti_1,
            :betti_2 => lancedb_betti_2,
            :unique_objects => [:Manifest, :VectorColumn, :VectorIndex, :Partition, :Centroid, :EmbeddingFunction],
            :irreversible_morphisms => 2,  # parent_manifest, source_column
            :coverage_holes => lancedb_betti_1.β1
        ),
        :translation_analysis => """
AM Radio Coverage Analysis:
═══════════════════════════

DuckDB → LanceDB Translation:
  ✓ Table → Table (full coverage overlap)
  ✓ Column → Column (full coverage overlap)
  ✗ RowGroup → ??? (partial: Fragment covers some)
  ✗ Segment → ??? (no direct analog - DEAD ZONE)
  
LanceDB → DuckDB Translation:
  ✓ Table → Table (full coverage overlap)
  ✓ Column → Column (full coverage overlap)  
  ✗ Manifest → ??? (no versioning - DEAD ZONE)
  ✗ VectorColumn → ARRAY (lossy - WEAK SIGNAL)
  ✗ VectorIndex → Extension (partial - INTERFERENCE)
  
PERSISTENT HOLES (never die):
  🔴 parent_manifest: Temporal irreversibility
     Birth: ε=0, Death: ∞
     AM analog: Station only broadcasts forward in time
     
  🔴 source_column: Semantic irreversibility  
     Birth: ε=0, Death: ∞
     AM analog: Lossy compression in transmission
"""
    )
end

# ═══════════════════════════════════════════════════════════════════════
# GAY.JL COLOR INTEGRATION
# ═══════════════════════════════════════════════════════════════════════

const COVERAGE_COLORS = Dict(
    :full_overlap => "#2BEE64",      # Green - lossless translation
    :partial_overlap => "#EED52B",   # Yellow - lossy translation
    :dead_zone => "#EE2B2B",         # Red - no translation
    :persistent_hole => "#CD24A6",   # Magenta - irreversible
)

"""
Color-code coverage analysis results
"""
function color_coverage(analysis)
    Dict(
        :duckdb_unique => COVERAGE_COLORS[:dead_zone],
        :lancedb_unique => COVERAGE_COLORS[:dead_zone],
        :shared_concepts => COVERAGE_COLORS[:full_overlap],
        :lossy_translations => COVERAGE_COLORS[:partial_overlap],
        :irreversible => COVERAGE_COLORS[:persistent_hole],
    )
end

# ═══════════════════════════════════════════════════════════════════════
# MECHANISM DESIGN: Optimal Station Placement
# ═══════════════════════════════════════════════════════════════════════

"""
AM Radio Mechanism Design Problem:
Given fixed spectrum (schema objects), how to maximize coverage?

For schema design:
1. Add bridge objects to fill dead zones
2. Add reversible morphisms to heal holes
3. Minimize β₁ (coverage gaps)

Optimal DuckDB ↔ LanceDB bridge would need:
- VersionAdapter: Maps Manifest semantics to temporal tables
- EmbeddingAdapter: Maps VectorColumn to ARRAY + index extension
- IndexAdapter: Maps VectorIndex to ART/vss extension
"""
const BRIDGE_RECOMMENDATIONS = """
Schema Bridge Mechanism Design
══════════════════════════════

To maximize DuckDB ↔ LanceDB interoperability:

1. VERSION BRIDGE (heal Manifest dead zone):
   - DuckDB: Add temporal table extension
   - Bridge: Manifest → temporal_table morphism
   - Coverage gain: +1 station overlap
   
2. VECTOR BRIDGE (heal VectorColumn dead zone):
   - DuckDB: Use vss extension
   - Bridge: VectorColumn → ARRAY + vss_index
   - Coverage gain: +1 station overlap (partial signal)
   
3. INDEX BRIDGE (heal VectorIndex dead zone):
   - DuckDB: Map IVF → ART partitions
   - Bridge: Partition → ART node
   - Coverage gain: +0.5 station overlap (lossy)

IRREVERSIBLE HOLES (cannot be bridged):
- parent_manifest: Time flows one way
- source_column: Information lost in embedding

These are FUNDAMENTAL LIMITS of the translation problem.
Persistent homology proves they cannot be eliminated.
"""

# Export analysis function
function run_coverage_analysis()
    analysis = compare_schema_coverage()
    colors = color_coverage(analysis)
    
    Dict(
        :analysis => analysis,
        :colors => colors,
        :recommendations => BRIDGE_RECOMMENDATIONS,
        :ghrist_citation => "de Silva & Ghrist, 'Coverage in Sensor Networks via Persistent Homology'"
    )
end
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall