Skip to content

Ar9av/obsidian-wiki / skills/copilot-history-ingest

copilot-history-ingest

Verified

Ingest GitHub Copilot CLI/session history into Obsidian as distilled knowledge. Use for importing or mining Copilot history; not for general Copilot help, simple session search, or backup.

FileKindSize
SKILL.mdskill md18.3 KB
references/copilot-data-format.mdmarkdown12.1 KB

references/copilot-data-format.md

View on GitHub
# GitHub Copilot CLI Data Format — Detailed Reference

## Session-State Directory

`~/.copilot/session-state/` contains one directory per session the user has run with GitHub Copilot CLI. Each directory is named with a UUID.

### `workspace.yaml`

Minimal session metadata file, always present:

```yaml
id: <session-uuid>
cwd: /path/to/project
summary_count: 3
created_at: 2026-04-02T14:28:13.304Z
updated_at: 2026-04-29T12:00:00.000Z
```

`summary_count` reflects how many checkpoints were written. Sessions with `summary_count: 0` were either very short or completed without checkpointing — check `events.jsonl` for content anyway.

### `vscode.metadata.json`

VS Code context, written when the session is associated with a VS Code workspace:

```json
{
  "workspaceFolder": {
    "folderPath": "c:\\Users\\name\\git\\my-project",
    "timestamp": 1773245818098
  },
  "writtenToDisc": true,
  "repositoryProperties": {
    "repositoryPath": "c:\\Users\\name\\git\\my-project",
    "branchName": "feature/my-branch",
    "baseBranchName": "origin/main"
  },
  "customTitle": "User-written session label or system-set title"
}
```

`customTitle` is the most human-readable session label — use it as a heading when creating session-derived wiki content. May be absent on older sessions.

### `events.jsonl`

The full event log for one session. Each line is a JSON object representing one event in the session.

#### Event: `session.start`

```json
{
  "type": "session.start",
  "data": {
    "sessionId": "09371a50-9a50-484a-8743-5c696de1623a",
    "version": 1,
    "producer": "copilot-agent",
    "copilotVersion": "0.0.420",
    "startTime": "2026-03-02T15:10:04.678Z",
    "context": {
      "cwd": "C:\\Users\\name\\git\\my-project",
      "gitRoot": "C:\\Users\\name\\git\\my-project",
      "branch": "master"
    }
  },
  "id": "<event-uuid>",
  "timestamp": "2026-03-02T15:10:04.817Z",
  "parentId": null
}
```

`data.context.cwd` and `data.context.branch` establish the project context. Always read `session.start` first.

#### Event: `user.message`

```json
{
  "type": "user.message",
  "data": {
    "content": "review my staged but uncommitted changes for issues",
    "transformedContent": "<current_datetime>...</current_datetime>\n\nreview my staged...",
    "attachments": [],
    "interactionId": "9352571e-a0b9-4774-8ecb-40bc58f86e94"
  },
  "id": "<event-uuid>",
  "timestamp": "2026-03-02T15:10:45.058Z",
  "parentId": "<parent-event-uuid>"
}
```

Use `data.content` (not `data.transformedContent`) — the transformed version includes injected system context that's noise for wiki purposes.

#### Event: `assistant.message`

```json
{
  "type": "assistant.message",
  "data": {
    "messageId": "<uuid>",
    "content": "I'll review the staged changes in those three files.",
    "toolRequests": [
      {
        "toolCallId": "tooluse_...",
        "name": "report_intent",
        "arguments": { "intent": "Reviewing staged changes" },
        "type": "function"
      },
      {
        "toolCallId": "tooluse_...",
        "name": "powershell",
        "arguments": {
          "command": "git --no-pager diff --cached --stat",
          "description": "Show staged diff"
        },
        "type": "function"
      }
    ],
    "interactionId": "9352571e-a0b9-4774-8ecb-40bc58f86e94",
    "reasoningOpaque": "<base64-encrypted-reasoning>",
    "reasoningText": "The user wants me to review staged git changes..."
  },
  "id": "<event-uuid>",
  "timestamp": "2026-03-02T15:10:50.235Z",
  "parentId": "<parent-event-uuid>"
}
```

**Extraction strategy:**

- Extract `data.content` — the assistant's visible text response
- `data.toolRequests` — skim tool names and description arguments for file/command patterns; ignore `report_intent` calls
- **Skip `data.reasoningOpaque` entirely** — encrypted/encoded internal reasoning
- **Skip `data.reasoningText` entirely** — decrypted reasoning; internal only, never user-visible

#### Event: `assistant.turn_start`

```json
{
  "type": "assistant.turn_start",
  "data": { "turnId": "0", "interactionId": "..." },
  "id": "...",
  "timestamp": "..."
}
```

Marks the start of an assistant turn. Useful for turn boundary detection; no content to extract.

#### Event: `tool.execution_start`

```json
{
  "type": "tool.execution_start",
  "data": {
    "toolCallId": "tooluse_...",
    "toolName": "powershell",
    "arguments": { "command": "dotnet build ...", "description": "Build project" }
  },
  "id": "...",
  "timestamp": "..."
}
```

Reveals what tools (file reads, commands, searches) were invoked. File-related tools (`view`, `edit`, `create`) with their paths are worth noting for the `session_files` equivalent when reading events directly.

#### Event: `tool.execution_end`

Contains the raw tool output. Usually noise — skip unless diagnosing errors.

### `checkpoints/<uuid>.json`

Mid-session progress summaries, written automatically as the session progresses:

```json
{
  "title": "Implementing auth module",
  "overview": "Working on JWT authentication for the API...",
  "history": "1. Analyzed existing auth code\n2. Created IAuthService...",
  "work_done": "- Created IAuthService interface\n- Implemented JwtAuthService",
  "technical_details": "Uses RS256 signing. Token expiry configurable via settings...",
  "important_files": "- src/Auth/IAuthService.cs\n- src/Auth/JwtAuthService.cs",
  "next_steps": "- Wire up to DI container\n- Add refresh token support"
}
```

This is the highest-value structured content in the per-session directory — equivalent to Claude's memory files.

### `index.md`

Session-end summary written as a markdown file. Typically 1–3 paragraphs summarizing what was accomplished. Content varies by session length and complexity. Read this before opening `events.jsonl` to decide if the session is worth deep-processing.

---

## Global Session Store (`session-store.db`)

SQLite database at `~/.copilot/session-store.db`. The canonical cross-session record.

### Schema

#### `sessions`

| Column      | Type | Notes                                      |
| ----------- | ---- | ------------------------------------------ |
| `id`        | TEXT | Session UUID (PK)                          |
| `cwd`       | TEXT | Working directory                          |
| `repository`| TEXT | `owner/repo` format when available         |
| `branch`    | TEXT | Git branch name                            |
| `summary`   | TEXT | One-paragraph session summary              |
| `created_at`| TEXT | ISO 8601 timestamp                         |
| `updated_at`| TEXT | ISO 8601 timestamp — use for delta checks  |
| `host_type` | TEXT | `"vscode"`, `"cli"`, or similar            |

#### `turns`

| Column              | Type    | Notes                          |
| ------------------- | ------- | ------------------------------ |
| `id`                | INTEGER | PK                             |
| `session_id`        | TEXT    | FK → `sessions.id`             |
| `turn_index`        | INTEGER | 0-based turn sequence          |
| `user_message`      | TEXT    | Raw user message               |
| `assistant_response`| TEXT    | Assistant's text response      |
| `timestamp`         | TEXT    | ISO 8601 timestamp             |

Note: `user_message` here is the pre-transformation content — use this, not `transformedContent` from `events.jsonl`.

#### `checkpoints`

| Column              | Type    | Notes                          |
| ------------------- | ------- | ------------------------------ |
| `id`                | INTEGER | PK                             |
| `session_id`        | TEXT    | FK → `sessions.id`             |
| `checkpoint_number` | INTEGER | 1-based                        |
| `title`             | TEXT    | Short title                    |
| `overview`          | TEXT    | High-level summary             |
| `history`           | TEXT    | Step-by-step of what happened  |
| `work_done`         | TEXT    | Completed items                |
| `technical_details` | TEXT    | Implementation specifics       |
| `important_files`   | TEXT    | Key files touched              |
| `next_steps`        | TEXT    | Open threads                   |
| `created_at`        | TEXT    | ISO 8601 timestamp             |

#### `session_files`

| Column         | Type    | Notes                                              |
| -------------- | ------- | -------------------------------------------------- |
| `session_id`   | TEXT    | FK → `sessions.id`                                 |
| `file_path`    | TEXT    | Absolute path to the file                          |
| `tool_name`    | TEXT    | `"edit"`, `"create"`, `"view"`, etc.               |
| `turn_index`   | INTEGER | Which turn touched the file                        |
| `first_seen_at`| TEXT    | ISO 8601 timestamp                                 |

> ⚠️ No `id` column — use `COUNT(DISTINCT sf.file_path)` not `COUNT(DISTINCT sf.id)`.

Aggregate by `file_path` across sessions to identify architecturally important files.

#### `session_refs`

| Column       | Type    | Notes                                      |
| ------------ | ------- | ------------------------------------------ |
| `id`         | INTEGER | PK                                         |
| `session_id` | TEXT    | FK → `sessions.id`                         |
| `ref_type`   | TEXT    | `"commit"`, `"pr"`, `"issue"`              |
| `ref_value`  | TEXT    | Commit SHA, PR number, issue number        |
| `turn_index` | INTEGER | Which turn referenced it                   |
| `created_at` | TEXT    | ISO 8601 timestamp                         |

#### `search_index` (FTS5)

Full-text search index. Use for keyword discovery when surveying a large history:

```sql
SELECT content, session_id, source_type
FROM search_index
WHERE search_index MATCH 'auth OR authentication OR login'
LIMIT 20;
```

`source_type` values: `"turn"`, `"checkpoint_overview"`, `"checkpoint_history"`, `"checkpoint_work_done"`, `"checkpoint_technical"`, `"checkpoint_files"`, `"checkpoint_next_steps"`, `"workspace_artifact"`.

---

## VS Code Workspace Storage

### Location

The `workspaceStorage` directory is platform-specific:

| Platform | Default path                                                       |
| -------- | ------------------------------------------------------------------ |
| Windows  | `%APPDATA%\Code\User\workspaceStorage\`                            |
| macOS    | `~/Library/Application Support/Code/User/workspaceStorage/`       |
| Linux    | `~/.config/Code/User/workspaceStorage/`                           |

Each `<hash>/` subdirectory corresponds to a specific workspace (VS Code folder). The hash is derived from the workspace path — there is no human-readable mapping, so enumerate all `<hash>/GitHub.copilot-chat/` directories and use the `transcripts/` JSONL files' `session.start` events to identify which project each belongs to.

### Transcript JSONL (`transcripts/<uuid>.jsonl`)

Identical format to `events.jsonl` from Source 1. Parse using the same event type handlers. The `session.start` event's `data.context.cwd` tells you which project this belongs to.

### Memory Artifacts (`memory-tool/memories/<base64-session-id>/`)

Directory name is the session UUID encoded as base64. Files inside are markdown documents explicitly saved by the user or system during the session — typically `plan.md` containing the session plan.

Decode the directory name to link it to a session:

```python
import base64
# Pad to multiple of 4 before decoding
session_id = base64.b64decode(dir_name + '==').decode('utf-8')
```

---

## Processing Order

For maximum efficiency and signal-to-noise:

1. **`session-store.db` checkpoints** — Fastest, highest signal. Query all at once.
2. **`session-store.db` sessions.summary** — One-paragraph synopsis per session.
3. **Per-session `checkpoints/*.json` + `index.md`** — For sessions not yet in `session-store.db` or for additional detail.
4. **Memory artifacts** (`memory-tool/memories/`) — User-authored, high quality.
5. **`session-store.db` turns** — Full conversation, process selectively by topic.
6. **`events.jsonl` / transcript JSONL** — Only if `session-store.db` is absent or incomplete.
7. **`session_files` / `session_refs`** — For file pattern and git linkage metadata.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall