Disentinel/grafema / .claude/skills/elixir-escript-daemon-stdio-encoding
elixir-escript-daemon-stdio-encoding
VerifiedFix intermittent `{:no_translation, :unicode, :latin1}` crashes in Elixir escript daemons that use length-prefixed framed IPC on stdin/stdout. Use when: (1) daemon worker crashes only on some input files, usually ones with non-ASCII bytes (kanji, cyrillic, emoji); (2) error surfaces as `Protocol error` from daemon's error branch or as garbled frame-length bytes seen by the orchestrator/client side; (3) standalone one-shot mode works fine on the same input but multi-request daemon mode fails; (4) `IO.binread(:stdio, N)` returns `{:error, {:no_translation, :unicode, :latin1}}` despite the "bin" prefix suggesting it should be encoding-agnostic. Root cause is the escript default `:standard_io` encoding — it's `:unicode`, and `IO.binread` still routes through the io_server, which translates bytes to codepoints and errors when raw binary frames contain invalid UTF-8 sequences.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.9 KB |
SKILL.md
View on GitHub---
name: elixir-escript-daemon-stdio-encoding
description: |
Fix intermittent `{:no_translation, :unicode, :latin1}` crashes in Elixir escript
daemons that use length-prefixed framed IPC on stdin/stdout. Use when: (1) daemon
worker crashes only on some input files, usually ones with non-ASCII bytes (kanji,
cyrillic, emoji); (2) error surfaces as `Protocol error` from daemon's error branch
or as garbled frame-length bytes seen by the orchestrator/client side; (3) standalone
one-shot mode works fine on the same input but multi-request daemon mode fails;
(4) `IO.binread(:stdio, N)` returns `{:error, {:no_translation, :unicode, :latin1}}`
despite the "bin" prefix suggesting it should be encoding-agnostic.
Root cause is the escript default `:standard_io` encoding — it's `:unicode`, and
`IO.binread` still routes through the io_server, which translates bytes to codepoints
and errors when raw binary frames contain invalid UTF-8 sequences.
author: Claude Code
version: 1.0.0
date: 2026-04-19
---
# Elixir Escript Daemon — Stdio Encoding Trap
## Problem
Elixir escripts that implement a length-prefixed framed IPC daemon over stdin/stdout
(e.g. language analyzers, worker pools, plugin runners driven by an external
orchestrator) crash intermittently with:
```
Protocol error: {:no_translation, :unicode, :latin1}
```
The crash appears non-deterministic: same daemon, same code, some files trigger it,
most don't. Files that trigger it tend to contain multi-byte UTF-8 sequences (kanji,
cyrillic, emoji) in comments, atoms, or string literals. Running the daemon in
stand-alone single-request mode works fine — only the long-lived multi-request mode
fails.
## Context / Trigger Conditions
All of these must apply:
1. Elixir escript (`mix escript.build` output) with a `--daemon` mode
2. Length-prefixed frames on stdin/stdout (typical pattern: `<<len::32>> <> payload`)
3. Payloads can contain arbitrary bytes — JSON with UTF-8, raw binaries, etc.
4. The daemon uses `IO.binread(:stdio, N)` / `IO.binwrite(:stdio, data)` for framing
5. External orchestrator feeds requests over a pipe (`Stdio::piped()`), not a terminal
Typical symptom chain:
- Orchestrator logs `beam analyzer failed ... Pool request failed`
- Daemon stderr contains `Protocol error: {:no_translation, :unicode, :latin1}`
- The daemon's own error branch printed that line after `read_frame` returned
`{:error, {:no_translation, :unicode, :latin1}}`
- `IO.binread(:stdio, 4)` produced the tuple-error — despite being "binary" read
- The specific failing file(s) repeat between runs, but *which* daemon worker in the
pool gets the file varies, giving the illusion of flakiness
## Root Cause
Two non-obvious facts collide:
1. **Escript initializes `:standard_io` with `encoding: :unicode`.** This is the default
group leader configuration; it's not documented as "will bite you on binary IPC."
2. **`IO.binread` and `IO.binwrite` still route through the io_server.** The "bin"
prefix means "returns binaries, not lists" — NOT "bypasses encoding." The io_server
validates/translates bytes against the device's current encoding. For a `:unicode`
device, non-UTF-8 byte sequences raise `{:no_translation, :unicode, :latin1}`.
So any frame payload that happens to contain a byte sequence that isn't valid UTF-8
(random length headers with high-bit bytes, binary data, specific multi-byte tails of
truncated reads) kills the io_server interaction, which surfaces on the very next
`IO.binread` as that error tuple.
Secondary contributor: compiler/parser warnings from things like `Code.string_to_quoted`
go through `Logger`, whose default handler also writes to `:standard_io`. Any such
warning from a worker adds extra bytes to the frame stream and desynchronizes the
peer even when the io_server doesn't error.
## Solution
At daemon startup, **before** any stdin read or stdout write:
```elixir
def main(args) do
# Force stdio to flat byte passthrough. latin1 is 1:1 with bytes — any byte
# value is a valid latin1 codepoint, so IO.binread/binwrite never translate.
:io.setopts(:standard_io, encoding: :latin1)
# stderr is human text — keep it :unicode so inspect()/tuples with multi-byte
# binaries still render correctly.
:io.setopts(:standard_error, encoding: :unicode)
# Route Logger away from stdout so parser/compile warnings can't pollute the
# frame stream. Elixir 1.17+ uses the OTP :logger default handler.
_ =
:logger.update_handler_config(:default, :config, %{type: :standard_error})
case args do
["--daemon"] -> daemon_loop()
_ -> one_shot()
end
end
```
That's it. No need to rewrite the Protocol module, no raw `Port.open({:fd, 0, 1}, ...)`
(which causes its own `prim_tty: stealing control of fd=0` warning on stdout).
## Why Not `encoding: :unicode` on stdio?
Intuitive first attempt: "set stdio to `:unicode` so encoding matches actual content."
It makes the Logger/IO.write path for UTF-8 strings work cleanly, and in one-shot mode
(single request per process lifetime) it appears to fix things.
But it makes daemon mode *worse*: after a successful `IO.binwrite` of a large UTF-8
JSON payload, the io_server's state is fine for writes but later `IO.binread` calls
intermittently return the same `:no_translation` error. Tested on Elixir 1.19 / OTP 27.
`:latin1` is the safe choice because every byte is a valid codepoint — no translation
can fail.
## Why Not Raw Port via `{:fd, 0, 1}`?
`Port.open({:fd, 0, 1}, [:binary, {:packet, 4}])` seems elegant — it bypasses the
io_server entirely and the VM handles 4-byte length framing for you. It works
functionally, but on escript startup the `prim_tty` driver already owns fd 0/1 and
prints a warning to stdout when a new port takes over:
```
21:09:30.549 [error] driver_select(...) by fd (0/1) driver #Port<0.2> stealing control
of fd=0 from resource prim_tty:tty
```
Those bytes appear on stdout *before* the first frame, parsed by the orchestrator as
a bogus length header, and the whole conversation desyncs from byte 1. Workarounds
(close `user_drv` first, dup fds, etc.) add significant complexity. The `:io.setopts`
fix above is strictly simpler and sufficient.
## Verification
Reproducer: a daemon worker that ingests 50+ files in sequence, at least a few
containing multi-byte UTF-8 in source:
```python
# harness.py — feed multi-file stress through one daemon
import json, struct, subprocess
files = ["heavy_utf8_1.ex", "ascii_1.ex", "heavy_utf8_2.ex", ...]
proc = subprocess.Popen(["./daemon", "--daemon"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for f in files:
src = open(f).read()
payload = json.dumps({"file": f, "source": src}).encode("utf-8")
proc.stdin.write(struct.pack(">I", len(payload)) + payload)
proc.stdin.flush()
length = struct.unpack(">I", proc.stdout.read(4))[0]
resp = json.loads(proc.stdout.read(length))
assert resp["status"] == "ok", f"daemon crashed on {f}"
```
Without the fix: random crashes after 5-30 files.
With the fix: 0 crashes across 3+ consecutive full runs.
## Example
Real-world case from the grafema repo (`packages/beam-analyzer/lib/beam_analyzer.ex`,
commit 7d0d9f78): the daemon analyzed 60 Elixir source files from a project that
had kanji atoms (`:守り`, `:内観`) and cyrillic comments. Pre-fix: 2–3 files failed
per run, random which ones. Post-fix: 3 consecutive runs, 0 failures, 12166 graph
nodes produced deterministically.
## Notes
- The `:standard_io` device encoding is set at escript VM startup and is stable — one
call at the top of `main/1` is enough; no need to re-set per worker or per request.
- `:io.setopts(:standard_error, encoding: :unicode)` matters for stderr-bound logs
that embed `inspect/1` results with non-ASCII binaries; without it you can get the
same `:no_translation` error on error-path writes.
- Redirecting Logger doesn't silence it — stderr is inherited by the orchestrator,
which typically captures it for diagnostics. That's what you want.
- The `IO.binread` / `IO.binwrite` semantics described here are Elixir-specific; in
pure Erlang, `file:read/2` and `file:write/2` on stdio have the same gotcha because
they share the io_server.
- This fix is strictly safer than the alternatives; there's no scenario where you
*want* escript stdio to do encoding translation on framed binary IPC.
## Related Skills
- `beam-elixir-ast-gotchas` — other non-obvious Elixir/Erlang AST processing issues
in analyzer tools
- `nodejs-child-process-stdio-cleanup` — analogous trap on the Node.js side when
spawning daemons
## References
- Elixir [`IO.binread/2`](https://hexdocs.pm/elixir/IO.html#binread/2) — does not
document the encoding interaction
- Erlang [`io:setopts/2`](https://www.erlang.org/doc/man/io.html#setopts-2) — the
`encoding` option controls io_server translation
- Erlang [`logger` handler config](https://www.erlang.org/doc/man/logger.html) —
`type: :standard_error` is the documented way to redirect the default handler
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |