K-Dense-AI/scientific-agent-skills / skills/vaex
vaex
VerifiedProcesses large tabular scientific datasets with Vaex expressions, filtered views, streamed statistics, binned visualizations, and file conversion. Use for larger-than-RAM HDF5, Arrow, CSV, or Parquet analysis, virtual feature engineering, or Vaex ML preprocessing; distinguishes these operations from estimators and conversions that materialize data.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.5 KB |
| references/core_dataframes.md | markdown | 4.6 KB |
| references/data_processing.md | markdown | 7.1 KB |
| references/io_operations.md | markdown | 7.6 KB |
| references/machine_learning.md | markdown | 9.8 KB |
| references/performance.md | markdown | 5.1 KB |
| references/review.md | markdown | 4.5 KB |
| references/visualization.md | markdown | 5.0 KB |
references/performance.md
View on GitHub# Performance and memory
Targets core 4.19.0. See [caching](https://vaex.io/docs/guides/caching.html),
[configuration](https://vaex.io/docs/conf.html), and released executor/cache source.
## Batch independent reductions
```python
import asyncio
import numpy as np
import vaex
df = vaex.from_arrays(x=np.arange(10.), y=np.arange(10.) ** 2)
df['derived'] = df.x ** 2 + df.y
pending = [df.derived.mean(delay=True), df.x.std(delay=True), df.y.sum(delay=True)]
df.execute()
values = [task.get().item() for task in pending]
assert np.allclose(values, [57., np.std(np.arange(10.)), 285.])
async def compute(frame):
mean = frame.x.mean(delay=True)
count = frame.count(delay=True)
await frame.execute_async()
return mean.get().item(), count.get().item()
assert asyncio.run(compute(df)) == (4.5, 10)
```
In a notebook with a running event loop, use `await compute(df)` instead of
`asyncio.run`. Defining an expression or virtual column defers per-row evaluation;
ordinary reductions execute now. Promises must be executed before `.get()`.
There is no public `vaex.execute([tasks])` collection API. Batching compatible tasks
can share scans, but "one pass" is not universal: limits, means, variance or dependant
tasks can need preliminary work. Measure actual data passes and timing on the target workload.
## Memory boundaries
| Operation | Allocation to plan for |
| --- | --- |
| HDF5/Arrow open | Metadata, handles, mapped/address space and accessed pages |
| Virtual columns | Expression metadata and per-chunk temporary arrays |
| Filters/selections | Boolean masks and selection/cache metadata |
| Reductions/grids | Chunk buffers and grid accumulators; grid size grows with dimensions |
| Groupby/unique/join/sort | Cardinality dictionaries, output tables and/or row index arrays |
| `.values`, `.to_numpy()`, unchunked pandas conversion | Requested full arrays |
| `.materialize()` | Full selected virtual columns in memory |
| sklearn `Predictor.fit` | Full feature matrix and target, plus estimator allocations |
`df.byte_size()` describes data bytes, not resident memory or a process limit.
`df.is_local()` tells whether the frame is local rather than a remote server frame;
it accepts no column argument and does not indicate virtual columns or memory mapping.
Use `df.virtual_columns` to inspect derived definitions. `df.copy()` is shallow.
```python
assert 'derived' in df.virtual_columns
assert df.is_local()
materialized = df.materialize('derived') # small fixture only
assert 'derived' not in materialized.virtual_columns
assert np.allclose(materialized.derived.to_numpy(), 2 * np.arange(10.) ** 2)
rows = 0
for start, stop, chunk in df[['x', 'derived']].to_pandas_df(chunk_size=3):
assert len(chunk) == stop - start
rows += len(chunk)
assert rows == 10
```
Export already evaluates virtual columns in chunks. Materializing first is not a fix
for an out-of-memory export. Reduce selected columns/chunk size, bound group cardinality,
check disk capacity and profile the actual operation instead. Avoid collecting all
pandas chunks into a list, which reconstructs the full allocation.
## Explicit caching and timing
`vaex.cache` starts disabled unless configured. A repeated mean is not guaranteed to
be cached. Optional `cachetools` supplies the bounded in-memory cache below:
```python
import time
import vaex.cache
with vaex.cache.memory(maxsize='8MiB', clear=True):
first = df.x.mean()
second = df.x.mean()
assert first == second
with vaex.cache.off():
start = time.perf_counter()
result = df.y.sum()
elapsed = time.perf_counter() - start
assert result == 285
print({'elapsed_seconds': elapsed, 'column_bytes': df.byte_size()})
```
Cache budget is not a total-RSS cap. Cached results, operating-system file cache and
warm compiled kernels can change timing; report warm/cold conditions, package versions,
row/column counts, dtype, hardware, storage and workload. No universal billion-rows-per-second
claim applies to arbitrary expressions or files. Use Python profiling tools or timing;
`with vaex.profiler():` is not a supported context manager.
Set `VAEX_NUM_THREADS` in the process environment before starting Python for repeatable
thread limits. Avoid oversubscription with BLAS/Numba/estimator threads. More threads
can increase per-worker buffers and do not guarantee faster I/O.
## Other execution engines
Core 4.19.0 does not have `to_dask_dataframe()`. Its documented Dask integration is
`to_dask_array()` for numerical arrays; that alone does not provision a distributed
cluster. Core pins Dask `<2024.9`, so current standalone Dask recommendations cannot
be transplanted into this environment without compatibility work. [Dask guide](https://vaex.io/docs/guides/dask.html).
For custom functions, prefer built-in expression operations. `df.apply(function,
arguments=[...], vectorize=True)` calls the function on array chunks and can allocate
chunk outputs; `vectorize=False` has different, row-function behavior. Verify types,
nulls and independent numerical results. JIT compilation does not create a nonexistent
`Expression.custom_agg` API, and compilation time is separate from steady-state timing.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |