metalbear-co/mirrord / skills/skills/mirrord-operator
mirrord-operator
VerifiedHelp users install and configure the mirrord Operator for team/enterprise environments. Use when users ask about operator setup, Helm installation, cloud API key or license configuration, air-gapped/offline licensing, enabling features (queue splitting, DB branching, preview environments, multi-cluster), internal registries, OpenShift/GKE Autopilot, RBAC, or multi-user mirrord deployments. Also use when an AI agent hits a Team or Enterprise feature on a cluster with no license and needs to offer the user a trial (agent-started trial; the trial is Enterprise tier).
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 2.9 KB |
| SKILL.md | skill md | 28.0 KB |
| references/helm-values.md | markdown | 15.4 KB |
| references/troubleshooting.md | markdown | 5.9 KB |
| references/values.yaml | config | 76.7 KB |
references/troubleshooting.md
View on GitHub# Common Issues
> Sources:
> - https://metalbear.com/mirrord/docs/reference/operator
> - https://metalbear.com/mirrord/docs/faq/troubleshooting
## "Operator not found" when running mirrord
The mirrord client cannot find or connect to the operator in the cluster.
**Solution:** Confirm the operator is up and reachable:
```bash
mirrord operator status # preferred — checks the client can reach the operator
kubectl get deployment -n mirrord mirrord-operator
kubectl get pods -n mirrord
```
If the pod isn't running, inspect it:
```bash
kubectl describe deployment -n mirrord mirrord-operator
kubectl logs -n mirrord deployment/mirrord-operator
```
## Licensing / authentication errors
The operator can't obtain or validate a license.
**How the operator authenticates (know which path you're on):**
- **Cloud API key (default):** the operator exchanges a cloud API key for a license over the API. Provide it via `cloud.apiKey.keyRef` (Secret), `cloud.apiKey.gsmRef` (Google Secret Manager), or `cloud.apiKey.key` (inline dev/test). A revoked/rotated key or no cloud connectivity causes failures — regenerate in the dashboard (Settings, app.metalbear.com).
- **License key (deprecated for cloud):** `license.key` / `license.keyRef` (Secret data key `OPERATOR_LICENSE_KEY`). Still required as the shared secret when using a self-hosted license server.
- **Air-gapped:** offline PEM (`license.file.secret.data.license.pem` or `license.pemRef`) or a `license.licenseServer` URL.
**Checks:**
```bash
# If using a Secret ref, confirm it exists and is populated
kubectl get secret -n mirrord <your-secret-name>
# Look for auth/license errors in the logs
kubectl logs -n mirrord deployment/mirrord-operator | grep -iE 'licen|api key|cloud|auth'
```
To rotate a **cloud API key**, update it in the dashboard, recreate the Secret (have the user run it — never paste the key into the agent), and `helm upgrade`. Never pass key material via `--set`.
## "Permission denied" / users can't create sessions (RBAC)
Users lack permission on mirrord's CRDs.
**Solution:** Use the **roles the chart creates** — don't hand-write a role for mirrord's CRDs (they span several API groups: `operator.metalbear.co`, `mirrord.metalbear.co`, `queues.mirrord.metalbear.co`, `preview.mirrord.metalbear.co`, and change over time).
1. For each namespace where developers run mirrord, add it to `roleNamespaces` so the chart creates a namespaced role there:
```yaml helm-values
roleNamespaces:
- staging
- development
```
2. `helm upgrade`, then bind users/groups/service accounts to the chart's roles with your own RoleBinding/ClusterRoleBinding. Available cluster roles: `mirrord-operator-user-basic`, `mirrord-operator-user`, and `mirrord-operator-ci` (scoped for CI / preview environments). Example:
```yaml k8s
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: alice-mirrord
subjects:
- kind: User
name: alice
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: mirrord-operator-user
apiGroup: rbac.authorization.k8s.io
```
`roleNamespaces: []` (empty) creates no namespaced roles.
## Operator pod keeps crashing or restarting
```bash
kubectl logs -n mirrord deployment/mirrord-operator --previous
kubectl get events -n mirrord --sort-by='.lastTimestamp'
```
Raise resources if the cluster is large (defaults suit ~200 concurrent sessions):
```yaml helm-values
operator:
requests: { cpu: 100m, memory: 100Mi }
limits: { cpu: 500m, memory: 200Mi } # increase for more sessions
```
## Port 443 conflict or unavailable
The operator can't bind port 443.
```yaml helm-values
operator:
port: 8443 # or 3000
```
Ensure nodes can reach the chosen port, then `helm upgrade`.
## `mirrord operator status` fails with `503 Service Unavailable` on GKE
With private networking, firewall rules may block the operator's API service.
**Solution:** Add a firewall rule allowing your cluster's master nodes to reach the operator port (default 443) on the cluster's pods.
## TLS / API-service certificate errors
The API server can't validate the operator's aggregated API service.
**Solution:**
- If you use a verified certificate, set `tls.apiService.insecureSkipTLSVerify: false`.
- To provision the cert out-of-band (ExternalSecret, cert-manager), set `tls.useExistingSecret: true` (then `tls.data`/`tls.certManager` are ignored), or enable `tls.certManager.enabled: true`.
- Restart to regenerate a self-signed cert if neither cert-manager nor `tls.data` is set:
```bash
kubectl rollout restart deployment -n mirrord mirrord-operator
```
## Sessions fail silently with a service mesh
Meshes (Istio, Linkerd) can interfere with operator↔agent traffic.
**Solution:** Exclude the mirrord namespace from the mesh, or pin a static agent port and exclude it:
```yaml helm-values
agent:
port: 50000
```
Istio target-pod annotation:
```
traffic.sidecar.istio.io/excludeInboundPorts: '50000'
```
## Feature enabled in config but rejected by the operator
A feature (queue splitting, DB branching, preview environments) is used client-side but its Helm flag isn't enabled, or the operator/CLI version is too old.
**Solution:** Enable the matching `operator.*` flag (e.g. `operator.kafkaSplitting`, `operator.pgBranching`, `operator.previewEnv`) and `helm upgrade`. Check each feature's minimum operator/CLI/chart versions in the corresponding feature skill (`mirrord-kafka`, `mirrord-db-branching`, `mirrord-prev-env`). Note **generic DB branching** (`operator.genericBranching`) lets branch creators run arbitrary images — gate it and restrict `genericBranchConfig.dbPod.allowedImages`.
## Upgrading the operator breaks in-flight sessions
Running sessions can fail during an upgrade.
**Solution:** Upgrade when the cluster is quiet. Check activity with `mirrord operator status`, then:
```bash
helm repo update
helm upgrade --install -f values.yaml mirrord-operator metalbear/mirrord-operator
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |