Hello-QM/catgo-LRG / .claude/skills/troubleshoot-workflow
workflow_errors
VerifiedDiagnose and fix CatGo workflow engine errors including HPC connection failures, missing POTCARs, stuck tasks, and REMOTE_ERROR states.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.8 KB |
SKILL.md
View on GitHub---
name: workflow_errors
description: Diagnose and fix CatGo workflow engine errors including HPC connection failures, missing POTCARs, stuck tasks, and REMOTE_ERROR states.
---
# CatGo Workflow Engine Error Troubleshooting Skill
## When to Use
Use this skill when the problem is NOT with the DFT code itself, but with
CatGo's workflow engine, HPC connectivity, or job management. Route here when:
- Tasks are stuck in READY or SUBMITTED state
- HPC connection fails (SSH, SFTP)
- POTCAR or pseudopotential files not found
- REMOTE_ERROR status on a task
- Workflow does not start after submission
- Job disappears from queue unexpectedly
## Diagnostic Steps
### Step 1: Check system status
```json
catgo_system(action: "status")
```
This reports:
- Backend health (FastAPI server running)
- HPC connection status (SSH connected, SFTP available)
- Engine status (scanner running, poller active)
### Step 2: Check workflow and task status
```json
catgo_workflow_engine(action: "status", params: { workflow_id: "<wf_id>" })
```
### Step 3: Get recent error logs
```json
catgo_system(action: "errors")
```
---
## Error Reference
### No HPC Connection
**Symptoms:**
- Tasks stay in READY state indefinitely
- `catgo_system(action: "status")` shows no active HPC connections
- Error: "No HPC connection available"
**Diagnosis:**
```json
catgo_system(action: "status")
```
**Fixes:**
1. Verify HPC credentials are configured in the CatGo settings
2. Check that the HPC host is reachable (not under maintenance)
3. If using SSH key + OTP, ensure the OTP token is current
4. Reconnect via the CatGo HPC settings panel in the frontend
**Note:** CatGo uses asyncssh with keyboard-interactive authentication.
Some HPC systems have specific prompt formats for OTP that may need special
handling. See `server/CLAUDE.md` for the KbdintSSHClient OTP issue.
### POTCAR Not Found
**Symptoms:**
- VASP task fails immediately after submission
- Error message contains "POTCAR" or "pseudopotential"
- Error: "Could not find POTCAR for element X"
**Diagnosis:**
```json
catgo_workflow_engine(action: "get_result", params: {
workflow_id: "<wf_id>",
task_id: "<task_id>"
})
```
**Fixes:**
1. Check that `VASP_PP_PATH` is set correctly on the HPC:
- Should point to the directory containing `potpaw_PBE/`, `potpaw_LDA/`
- Each element dir must contain a `POTCAR` file
2. For CP2K, check that the basis set and pseudopotential files are in the
CP2K data directory (usually `$CP2K_DATA_DIR`).
3. If the element was recently added to the structure (e.g., a dopant),
the POTCAR generation may not include it. Verify the structure has
correct elements:
```json
catgo_view(action: "get_state")
```
### Task Stuck in READY
**Symptoms:**
- Task shows `state: "READY"` for a long time
- No HPC job is submitted
- No error message
**Diagnosis:**
```json
catgo_workflow_engine(action: "status", params: { workflow_id: "<wf_id>" })
```
**Possible causes and fixes:**
1. **Upstream task not complete:** The task depends on another task that has
not finished. Check the DAG:
```json
catgo_workflow_engine(action: "get_dag", params: { workflow_id: "<wf_id>" })
```
2. **Engine scanner not running:** The workflow engine scanner picks up READY
tasks periodically. Check system status:
```json
catgo_system(action: "status")
```
3. **Workflow not submitted:** The workflow was created but never submitted:
```json
catgo_workflow_engine(action: "submit", params: { workflow_id: "<wf_id>" })
```
4. **Manual reset:** Force the task back to READY and re-trigger:
```json
catgo_workflow_engine(action: "reset", params: {
workflow_id: "<wf_id>",
task_id: "<task_id>"
})
```
### REMOTE_ERROR
**Symptoms:**
- Task status shows `state: "REMOTE_ERROR"`
- The job was submitted to HPC but something failed during execution
**Diagnosis:**
```json
catgo_workflow_engine(action: "get_result", params: {
workflow_id: "<wf_id>",
task_id: "<task_id>"
})
```
**Common causes:**
1. **Job killed by scheduler:** Exceeded walltime or memory limit. Increase
resources in run_config.
2. **File system error:** HPC scratch space full or quota exceeded.
3. **Module not loaded:** Required software module not available. Check the
HPC submission script.
4. **SFTP failure during file transfer:** CatGo could not upload input files
or download results. The SFTP fallback (exec-based transfer) should handle
this automatically, but some HPC systems restrict both.
**Fix:** After addressing the root cause:
```json
catgo_workflow_engine(action: "retry", params: {
workflow_id: "<wf_id>",
task_id: "<task_id>"
})
```
### Task Stuck in SUBMITTED / RUNNING
**Symptoms:**
- Task shows SUBMITTED or RUNNING but the HPC job has already finished
- The poller is not picking up the completed job
**Diagnosis:**
```json
catgo_system(action: "status")
```
Check if the engine poller is active. If not, the backend may need a restart.
**Fix:** Reset the task to re-trigger polling:
```json
catgo_workflow_engine(action: "reset", params: {
workflow_id: "<wf_id>",
task_id: "<task_id>"
})
```
### Workflow Paused Unexpectedly
An error handler may pause a workflow after task failure to prevent cascading
failures. Fix the failed task first, then resume:
```json
catgo_workflow_engine(action: "resume", params: { workflow_id: "<wf_id>" })
```
## Recovery Actions Summary
- **Retry**: `catgo_workflow_engine(action: "retry", params: {workflow_id, task_id})`
- **Reset**: `catgo_workflow_engine(action: "reset", params: {workflow_id, task_id})`
- **Modify + retry**: `modify_params` then `retry`
- **Check DAG**: `catgo_workflow_engine(action: "get_dag", params: {workflow_id})`
## When to Escalate
- Backend down: user needs to restart (`pnpm desktop:serve`)
- Repeated HPC connection failures: re-enter credentials or check maintenance
- Same REMOTE_ERROR after multiple retries: likely environmental (HPC config)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| planning-and-task-breakdownaddyosmani/agent-skillsBreaks work into ordered tasks. Use when you have a spec or clear requirements and need to break work into implementable tasks. Use when a task feels too large to start, when you need to estimate scope, or when parallel work is possible. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| screenpipethedotmack/claude-memSearch screen and audio history recorded by Screenpipe and optionally save selected context into claude-mem. Use when the user asks to recall desktop activity, a meeting, or something seen in another app through Screenpipe. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| contact-widgetnexu-io/open-designSelf-contained floating chat widget with welcome screen, social links, meeting button, and message input. Single HTML file, zero dependencies. | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |
| meeting-notesnexu-io/open-designMeeting notes page — title bar with attendees, agenda checklist, decisions block, action items table with owners + dates, and a "next meeting" footer. Use when the brief mentions "meeting notes", "minutes", "1:1 notes", "all-hands recap", or "会议纪要". | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |