K-Dense-AI/scientific-agent-skills / skills/vaex
vaex
VerifiedProcesses large tabular scientific datasets with Vaex expressions, filtered views, streamed statistics, binned visualizations, and file conversion. Use for larger-than-RAM HDF5, Arrow, CSV, or Parquet analysis, virtual feature engineering, or Vaex ML preprocessing; distinguishes these operations from estimators and conversions that materialize data.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.5 KB |
| references/core_dataframes.md | markdown | 4.6 KB |
| references/data_processing.md | markdown | 7.1 KB |
| references/io_operations.md | markdown | 7.6 KB |
| references/machine_learning.md | markdown | 9.8 KB |
| references/performance.md | markdown | 5.1 KB |
| references/review.md | markdown | 4.5 KB |
| references/visualization.md | markdown | 5.0 KB |
references/machine_learning.md
View on GitHub# Machine learning and state transfer
Targets vaex-ml 0.19.0, core 4.19.0. See [official tutorial](https://vaex.io/docs/tutorial_ml.html)
and [released ML source](https://github.com/vaexio/vaex/tree/v4.19.0/packages/vaex-ml/vaex/ml).
Current package wheels are the runtime authority when the website or repository differs.
## Fit only on training observations
Split by the scientific sampling unit (participant, batch, site, or time) before any
imputation, scaling, feature selection, encoder fitting or resampling. Vaex's
`df.ml.train_test_split` slices an already shuffled frame; it does not guarantee a
random, stratified, grouped or temporal split. Model selection needs validation data
separate from the final test set. Target encoding on the same rows used to estimate
target means leaks labels; use out-of-fold training encodings and training-only
mappings for held-out rows. No Vaex target encoder here automatically cross-fits.
The following is a deterministic mechanics example, not an accuracy benchmark:
```python
from pathlib import Path
from tempfile import TemporaryDirectory
import numpy as np
import vaex
import vaex.ml
from vaex.ml.sklearn import Predictor, IncrementalPredictor
from sklearn.linear_model import LinearRegression, SGDRegressor
x = np.linspace(-1, 1, 60)
df = vaex.from_arrays(x=x, x2=x**2, target=3*x-2,
category=np.array(['A', 'B', 'C'] * 20),
binary=(x > 0).astype(int))
train, test = df[:40], df[40:] # illustrative chronological split
scaler = vaex.ml.StandardScaler(features=['x'], prefix='scaled_')
train_scaled = scaler.fit_transform(train)
test_scaled = scaler.transform(test)
assert np.isclose(train_scaled.scaled_x.mean(), 0, atol=1e-12)
assert np.isclose(scaler.mean_[0], train.x.mean())
assert not np.isclose(test_scaled.scaled_x.mean(), 0)
model = Predictor(model=LinearRegression(), features=['scaled_x'], target='target',
prediction_name='prediction')
model.fit(train_scaled)
scored = model.transform(test_scaled)
assert np.allclose(scored.prediction.to_numpy(), test.target.to_numpy())
```
`Predictor.fit()` materializes the full feature matrix and target; wrapping sklearn
in Vaex does not make its fit out of core. `transform()` attaches a lazy prediction
expression; evaluating all predictions still allocates their full result. For
incremental training, choose an estimator whose `partial_fit` genuinely supports the
problem and configure batches/epochs explicitly:
```python
incremental = IncrementalPredictor(
model=SGDRegressor(random_state=7, learning_rate='constant', eta0=0.01),
features=['scaled_x'], target='target', batch_size=8, num_epochs=2,
prediction_name='online_prediction', shuffle=False,
)
incremental.fit(train_scaled)
assert np.isfinite(incremental.transform(test_scaled).online_prediction.to_numpy()).all()
```
A classifier usually needs `partial_fit_kwargs={'classes': ...}` with the fixed
class set. Vaex's `shuffle=True` shuffles within a batch, not necessarily across the
whole dataset. Batch training can differ from a single full fit; assess convergence,
ordering effects and held-out calibration rather than merely checking finite predictions.
## Transformers: current classes and behavior
Transformations generally return a shallow copy with virtual output columns. Fitting
still computes statistics/mappings, and these may allocate substantial state. Explicit
prefixes make feature references unambiguous; inspect actual output names.
```python
for transformer in [
vaex.ml.MinMaxScaler(features=['x'], feature_range=(-1, 1)),
vaex.ml.MaxAbsScaler(features=['x']),
vaex.ml.RobustScaler(features=['x']),
vaex.ml.LabelEncoder(features=['category']),
vaex.ml.OneHotEncoder(features=['category']),
vaex.ml.FrequencyEncoder(features=['category'], unseen='nan'),
vaex.ml.KBinsDiscretizer(features=['x'], n_bins=3, strategy='uniform'),
]:
transformed_train = transformer.fit_transform(train)
transformed_test = transformer.transform(test)
assert len(transformed_train) == len(train)
assert len(transformed_test) == len(test)
assert len(transformed_test.get_column_names()) > len(test.get_column_names())
# One scalar period per transformer, not a list of periods.
cycle = vaex.ml.CycleTransformer(features=['x'], n=24)
cyclic = cycle.fit_transform(train)
assert np.allclose(cyclic.x_x.to_numpy()**2 + cyclic.x_y.to_numpy()**2, 1)
pca = vaex.ml.PCA(features=['x', 'x2'], n_components=2)
pca.fit(train)
projected = pca.transform(test)
assert projected[['PCA_0', 'PCA_1']].values.shape == (20, 2)
assert np.isclose(sum(pca.explained_variance_ratio_), 1)
random_projection = vaex.ml.RandomProjections(features=['x', 'x2'], n_components=2)
assert len(random_projection.fit_transform(train)) == len(train)
```
- `RandomProjections` is plural; `KBinsDiscretizer` replaces the nonexistent `Discretizer` example.
- `MaxAbsScaler` uses the default `absmax_scaled_` prefix. One-hot outputs depend on
fitted category values; do not infer a fixed naming/schema contract from a toy example.
- Labels encode categories as integers without establishing a scientific order.
One-hot state can be very large for high cardinality; define unseen/missing policy.
- `BayesianTargetEncoder(features=[...], target=..., weight=..., unseen='nan')`
is the current class. Unseen choices are `'nan'` or `'zero'`, not automatically
the global mean. `WeightOfEvidenceEncoder` requires a meaningful binary target
and training-only estimates. Neither repairs label leakage.
- Standard scaling divides by standard deviation, not variance. Check zero/near-zero
dispersion, missingness and nonfinite results before model fitting.
- RobustScaler and quantile discretization use approximate percentile machinery;
their boundaries need independent scientific checks. PCA covariance state is
quadratic in feature count; scale comparable quantities when appropriate and
retain loadings/explained variance. Component signs may flip without changing the subspace.
## K-means
```python
from vaex.ml.cluster import KMeans
clusterer = KMeans(features=['x', 'x2'], n_clusters=2, max_iter=10,
random_state=7, prediction_label='cluster')
clusterer.fit(train)
clusters = clusterer.transform(test)
assert len(clusterer.cluster_centers) == 2
assert set(clusters.cluster.tolist()) <= {0, 1}
```
The class lives in `vaex.ml.cluster`; fitted centers are `cluster_centers`, without
an underscore. Default prediction name is `prediction_kmeans`. It uses Numba kernels
and repeated passes. Empty clusters, scaling, seeds and convergence require checking;
finite labels in a toy test do not demonstrate useful clusters or biological subtypes.
## Boosting adapters
Current classes live in `vaex.ml.xgboost.XGBoostModel`,
`vaex.ml.lightgbm.LightGBMModel`, and `vaex.ml.catboost.CatBoostModel`.
`params` and `num_boost_round` belong in the constructor, not as duplicate keywords
to `.fit`. Tiny native fits were run with XGBoost 3.4.1, LightGBM 4.7.0 and CatBoost
1.2.10; these establish API execution only.
```python
from vaex.ml.xgboost import XGBoostModel
booster = XGBoostModel(
features=['x'], target='target', prediction_name='boost_prediction',
params={'objective': 'reg:squarederror', 'max_depth': 2, 'nthread': 2},
num_boost_round=3,
)
booster.fit(train)
boosted = booster.transform(test)
assert np.isfinite(boosted.boost_prediction.to_numpy()).all()
```
XGBoost and LightGBM adapters construct full in-memory training matrices; their
Vaex wrappers do not imply external-memory learning. LightGBM fit accepts
`valid_sets`/`valid_names` and converts its callback options internally. CatBoost
accepts `evals` and optional `batch_size`; its batch mechanism trains/sums models,
which is not interchangeable with one full-data fit. Set prediction semantics
explicitly: CatBoost defaults to `prediction_type='Probability'`; labels and probabilities
are different outputs. For regression set `prediction_type='RawFormulaVal'`; the
default probability mode can produce two columns even for an RMSE-trained model. For sklearn use `prediction_type='predict_proba'` when needed,
and choose the desired class column using the fitted estimator's `classes_`.
## TensorFlow: source-verified only
There is no `vaex.ml.keras` module in 0.19.0. `vaex.ml.tensorflow.KerasModel` wraps a
**fitted** model for inference; its `.fit()` deliberately raises `NotImplementedError`.
Train with TensorFlow's own API or the optional `df.ml.tensorflow.to_keras_generator`
after checking that integration against the actual TensorFlow/Keras version. TensorFlow
was not installed or executed for this review. Do not promise a `target`/`epochs`
training interface on this wrapper or model serialization portability across Keras versions.
## Save attached predictions and preserve target rows
```python
# Fitting alone does not attach prediction expressions to the frame being saved.
train_with_prediction = model.transform(train_scaled)
with TemporaryDirectory() as directory:
state = Path(directory) / 'pipeline.json'
train_with_prediction.state_write(str(state))
fresh = df[40:].copy()
fresh.state_load(str(state), set_filter=False, trusted=True)
assert len(fresh) == len(test)
assert fresh.x.tolist() == test.x.tolist()
assert np.allclose(fresh.prediction.to_numpy(), scored.prediction.to_numpy())
```
Only load trusted state: models/functions can use pickle-like executable serialization.
State restores transformations, not raw data or experimental provenance. Confirm names,
types, units, row membership, class order and package versions on load; a training
filter must not silently become an inference filter. Evaluate with design-appropriate
held-out metrics and uncertainty; fold standard deviation is not a confidence interval.
Resampling/weights and feature selection belong inside training folds, never on the test
population. Exact row alignment must be preserved between targets and predictions.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |