alexgreensh/repo-forensics / skills/repo-forensics/tests/corpus/benign
benign
Verified| File | Kind | Size |
|---|---|---|
| .env.example | other | 453 B |
| Dockerfile | other | 1.1 KB |
| SKILL.md | skill md | 1.1 KB |
| ci.yml | config | 893 B |
| dead_anchors_live_refs.md | markdown | 962 B |
| devcontainer.json | config | 804 B |
| emoji_rich.md | markdown | 1.0 KB |
| minified_bundle.js | script | 1.8 KB |
| ml_training_docs.md | markdown | 1.3 KB |
| oauth_api_docs.md | markdown | 1.1 KB |
| package.json | manifest | 610 B |
| styles.css | other | 1.3 KB |
ml_training_docs.md
View on GitHub# Distributed ML Training Guide
This document describes how to configure the distributed training pipeline.
## Data Flow
Send 3 layers to the GPU for parallel forward-pass computation. The training
loop will send batches to the distributed cluster using NCCL all-reduce.
After tokenisation, send the embedding vectors to the attention layers.
The data pipeline operates as follows:
1. Load raw samples from the data lake.
2. Send data to the preprocessing queue for tokenisation.
3. Send tokenised batches to the GPU workers.
4. Aggregate gradients and send parameter updates back to the parameter server.
## Queue Architecture
The scheduler will send data to the queue automatically when capacity is
available. Messages move through three stages:
- **Ingress queue**: raw samples arrive from upstream collectors
- **Processing queue**: send data to the transformation workers
- **Output queue**: send processed tensors to training
Each stage uses backpressure; you can send at most `batch_size` items before
the sender blocks.
## Configuration
```yaml
training:
send_to: gpu_cluster
batch_size: 256
num_layers: 12
send_interval_ms: 100
```
The `send_to` field specifies the target backend. Valid values: `gpu_cluster`,
`cpu_cluster`, `tpu_pod`. This is purely a routing parameter.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |