bioepic-data/ecosim-agent / .agents/skills/ameriflux-era5-to-ecosim
ameriflux-era5-to-ecosim
VerifiedConvert AmeriFlux ERA5 half-hourly meteorological CSV files into EcoSIM hourly climate forcing NetCDF files, audit annual and seasonal precipitation budgets against sourced site climatology, compare generated forcing against overlapping measured tower meteorology, and report biases or unverified targets. Use when preparing or validating EcoSIM climate inputs for AmeriFlux sites, checking ERA5 against in situ records, diagnosing forcing discrepancies, or writing climate-quality reports. For non-AmeriFlux paper sites that need a fresh Copernicus CDS ERA5 point download by longitude/latitude, use era5-cds-point-download first.
AmeriFlux ERA5 to EcoSIM Converter
Use When
- You need to convert AmeriFlux ERA5 CSV forcing data to EcoSIM NetCDF climate forcing.
- You need hourly
TMPH,WINDH,RAINH,DWPTH,SRADH, orPATMvariables. - You need range-aware quality control, comparison against in situ meteorology, or a JSON quality report for climate forcing.
- You already have an AmeriFlux-format ERA5 CSV. If the task is to download global CDS ERA5 point data for a non-AmeriFlux literature site, use
era5-cds-point-downloadbefore any EcoSIM conversion.
Constraints
- NEVER use it extract soil data.
Overview
This skill converts AmeriFlux ERA5 half-hourly climate forcing data into the EcoSIM hourly climate format. When a matching AmeriFlux FULLSET file is available, it also reads the generated NetCDF back and compares the forcing against measured-only tower records.
For ordinary paper-derived sites outside AmeriFlux/FLUXNET, keep ERA5 acquisition separate: use era5-cds-point-download to retrieve Copernicus CDS point time series by longitude/latitude, inspect source units and accumulation conventions, then adapt or convert the data into the AmeriFlux-format columns expected here only after those units are verified.
Input Data Format
The input is a CSV file with the following columns:
TIMESTAMP_START: Start timestamp (YYYYMMDDHHMM)TIMESTAMP_END: End timestamp (YYYYMMDDHHMM)TA_ERA: Air temperature (°C)SW_IN_ERA: Shortwave incoming radiation (W m⁻²)LW_IN_ERA: Longwave incoming radiation (W m⁻²)VPD_ERA: Vapor pressure deficit (hPa in AmeriFlux ERA5 products)PA_ERA: Atmospheric pressure (kPa)P_ERA: Precipitation (mm h⁻¹)WS_ERA: Wind speed (m s⁻¹)
Pay close attention to the source ERA5 time step before conversion. AmeriFlux
archives may provide ERA5_HH half-hourly files or ERA5_HR hourly files, and
precipitation aggregation must match that cadence to avoid doubling or otherwise
distorting the water input.
Output Data Format
Climate NetCDF files are named by the forcing years they contain: <SITE_ID>_ecosim_climate_<first>-<last>.nc (e.g. US-UMB_ecosim_climate_1981-2021.nc). The writer reads the year variable after writing and renames the file, so an --output given without years (or with a stale range) gets the actual range appended; the final path is printed and recorded in the quality report (output_file).
The output is a netCDF file with the following variables:
TMPH: Hourly air temperature (°C)WINDH: Hourly wind speed (m s⁻¹)RAINH: Hourly precipitation (mm m⁻² hr⁻¹)DWPTH: Hourly actual atmospheric vapor pressure (kPa), computed ases(TA_ERA) - VPD_ERA/10with EcoSIM'svapsat0saturation formula. EcoSIM readsDWPTHas vapor pressure (VPK=min(DWPTH,VPS)), never as a deficit.SRADH: Hourly incident solar radiation (W m⁻²)PATM: Hourly Surface atmospheric pressure (kPa)year: Year ADZ0G: Windspeed measurement height (m)IFLGW: Flag for raising Z0G with vegetationZNOONG: Mean solar noon on the forcing clock (hour),12 - lon/15 + utc_offset. AmeriFlux timestamps are local standard time, soZNOONGmust be local too; EcoSIM adds the daily equation of time and caps hourlySRADHby the sun angle fromZNOONG.
Workflow
- Data Reading: Reads half-hourly climate data from the input CSV file
- Timestamp Parsing: Converts timestamp strings to datetime objects
- Data Aggregation: Averages consecutive half-hourly values to create hourly data
- Quality Control: Masks physically invalid values before aggregation and fills those gaps by time interpolation.
- Variable Mapping: Maps ERA5 variables to ECOSIM variable names and units
- NetCDF Creation: Creates a properly formatted netCDF file in ECOSIM format
- In Situ Validation: Reads the generated NetCDF and compares it with complete, measured-only hourly tower records from an overlapping AmeriFlux FULLSET file.
- Warning Report: Adds comparison metrics and large-difference warnings to the JSON quality report and prints warnings to stderr.
Physical Range Checks
The converter must ensure derived climate values are in legitimate ranges before writing EcoSIM NetCDF:
TMPHfromTA_ERA: -90 to 60 degC.WINDHfromWS_ERA: 0 to 75 m s^-1.RAINHfromP_ERA: non-negative source precipitation.DWPTHfromVPD_ERAandTA_ERA: non-negative source vapor pressure deficit; resulting vapor pressure clipped to[0, es(TA)].SRADHfromSW_IN_ERA: 0 to 1400 W m^-2.PATMfromPA_ERA: kPa bounds centered on site elevation whenALTIGis available, otherwise broad physical fallback bounds.
For sentinel values, non-finite values, and values outside these bounds, mask the value and interpolate along the half-hourly time axis. Use nearest-edge filling only at the beginning or end of the available source period. Emit a JSON quality report when --quality-report is supplied.
Time Registration (ERA5 vs Tower Clock)
Some AmeriFlux releases ship an ERA5 file whose clock is offset from the tower record (e.g. US-Ton 5-7, US-Ne1 4-7, and US-Wkg 4-7 are 1 h late; their tower SW_IN_F is on local standard time). Mean bias cannot detect this, so the converter coregisters ERA5 with the tower before QC and hourly aggregation:
- Find the FULLSET file (
--in-situ, or auto-discovery beside the ERA5 CSV) and load measuredSW_IN_F(QC 0). - Cross-correlate
SW_IN_ERAwith towerSW_IN_Fover lags of +/-3 h in source-timestep steps; require at least 2000 paired steps. - Treat a non-zero best lag as real only when it raises the correlation by at least 0.01 over zero lag and agrees with the diurnal-centroid lag within 0.5 h; otherwise report
ambiguousand do not shift. - In
--time-registration auto(default), shift every ERA5 variable onto the tower clock, keep the original record span, and fill the shifted-out edge steps from the nearest valid value.checkreports without shifting;offskips the check.--time-shift-hoursapplies a known lag (positive = ERA5 late) without a tower file. - Infer
ZNOONGafter registration, so it follows the corrected (tower) clock.
The quality report's time_registration block records status (registered, corrected, misregistered in check mode, ambiguous, unverified without a tower file, insufficient_overlap, manual, skipped), the lag, correlations by lag, centroid lag, and applied_shift_hours. After a correction, confirm that the in situ SRADH diurnal_centroid_lag_hours is near zero. A record without a tower file is unverified: its clock is assumed, not checked.
Solar Noon and Forcing Clock
- The UTC offset of the CSV clock is not stored in AmeriFlux files. By default the converter infers it as the whole hour nearest to
(SW_IN_ERA diurnal centroid) - (12 - lon/15); roundinglon/15is wrong for sites far from their zone meridian (US-UMB at -84.7 deg is UTC-5). Pass--utc-offsetto override. - The quality report's
solar_noonblock recordsZNOONG, the offset and its source, and the centroid mismatch; a mismatch above 0.75 h is a warning.
In Situ Comparison
For AmeriFlux sites, compare against observations whenever possible:
- Use
--in-situ <FULLSET_HH_or_HR.csv>when the observation file is known. Otherwise, auto-discover a matchingFULLSET_HHorFULLSET_HRCSV beside the ERA5 input. - Prefer
TA_F,WS_F,P_F,VPD_F,SW_IN_F, andPA_F, but retain only records whose corresponding*_F_QCvalue is0(measured). Do not validate ERA5 against records whose QC value indicates gap filling or ERA substitution. - If consolidated fields are unavailable, use direct measured columns such as
TA,WS,P,VPD,SW_IN, orPAwhen present. - Convert measured
VPD_Fto vapor pressure with measuredTA_F(both QC 0). Average temperature, wind, vapor pressure, shortwave radiation, and pressure to hourly values. Sum precipitation. Require every source interval within an hour to be measured. - Compare the observations against values read back from the generated NetCDF, not merely against the pre-write DataFrame.
- Record paired hours, forcing and observed means, mean bias, MAE, RMSE, Pearson correlation, and relative bias. For precipitation, emphasize total bias over paired intervals.
- Require at least 720 paired measured hours before issuing difference warnings by default. Use
--comparison-min-pairsonly when a shorter validation period is scientifically justified.
Use these broad operational warning screens:
| EcoSIM variable | Large-difference screen |
|---|---|
TMPH | absolute mean bias > 2 degC or RMSE > 5 degC |
WINDH | absolute mean bias > 1.5 m s-1, RMSE > 3 m s-1, or relative mean bias > 50% when absolute bias is at least 0.5 m s-1 |
RAINH | absolute relative total bias > 30% when measured total is at least 10 mm |
DWPTH | absolute mean bias > 0.30 kPa, RMSE > 0.75 kPa, or relative mean bias > 40% when absolute bias is at least 0.15 kPa |
SRADH | absolute mean bias > 40 W m-2, RMSE > 150 W m-2, or relative mean bias > 30% when absolute bias is at least 20 W m-2 |
PATM | absolute mean bias > 2 kPa or RMSE > 4 kPa |
Treat these as failure-detection screens, not universal ERA5 performance criteria. A warning requires diagnosis of units, timestamp convention, accumulation interval, source-product processing, and representativeness. Never apply an automatic multiplicative correction from these metrics. If no suitable observations are found, record not_available in the quality report rather than treating the comparison as passed.
Mandatory precipitation-budget review
Run the budget audit on the written NetCDF even when every hourly value passes
range checks or measured tower precipitation is unavailable. The converter now
uses Tools/precipitation_quality.py to report each year's valid-hour count,
calendar-complete annual total, monthly totals and calendar-year seasonal sums
(DJF uses January/February/December of that year). Exclude day-366 padding in
non-leap years. Partial years remain partial; never extrapolate their sum.
Climate precipitation excludes irrigation, which needs a separate water budget.
Obtain site mean annual precipitation (MAP) and its source from the site page,
local metadata or field literature. New site extraction retains optional
mean_annual_precip_mm and precipitation_reference; older cached metadata may
lack them. Do not infer MAP from the forcing under review. Record the reference
period/uncertainty when known and state when it is unavailable. Supply a sourced
value through --mean-annual-precip-mm and --precip-reference when metadata
lack it. These options also work in Tools/create_ecosim_climate_forcing.py.
The reference is retained in NetCDF global attributes for subsequent preflight.
Broad review screens flag a complete year's total outside 0.1–3 times site MAP, and a mean of at least five complete years outside 0.5–2 times MAP. These are operational warnings, not universal climatic limits or evidence for a correction factor. Missing MAP explicitly leaves climatological plausibility unverified. Do not auto-rescale, interpolate away or clip an anomalous annual budget. Investigate accumulation units/cadence, upstream transformations, location and source representativeness; compare with an independent gauge or weather product. A source-to-output match validates conversion fidelity, not the source climate.
Tower checks use measured QC support. If precipitation has zero/insufficient
measured pairs (for example all P_F_QC=2), retain an explicit unverified
precipitation finding. Successful checks on temperature or radiation do not
make precipitation verified: the overall comparison is partially_compared
when only some targets have sufficient observations. Reanalysis-filled tower
rainfall cannot serve as an independent validation of reanalysis forcing.
The quality report has separate precipitation_budget, per-variable tower
status and validation_status. review_required and missing comparisons must
remain prominent even when conversion exits successfully. The climate-readiness
checker recomputes selected-year budgets from the actual NetCDF and propagates
warnings through case preflight. It verifies the stored rainfall identity before
reusing measured-comparison status. No sourced reference or no measured check
means unverified, not a scientific pass; resolve material water-input conflicts
before interpreting soil moisture or calibrating hydraulic conductivity.
For existing input files, audit read-only without adding attributes or rewriting forcing:
python -m Tools.precipitation_quality /path/to/climate.nc \
--mean-annual-mm 790.37 \
--reference-source https://ameriflux.lbl.gov/sites/siteinfo/US-Ne1 \
--output /path/to/precipitation_budget.json
The value above is a US-Ne1 example, not a default for other sites. On the 2026-11-01 US-Ne1 review, all 44 forcing years had large totals (mean about 3,125 mm/year versus site MAP about 790 mm). Tower rain was unavailable, so previous measured-bias warnings never ran despite all hourly values passing range checks. Retain this failure mode in regression coverage.
Validation:
python -m unittest Tools.test_precipitation_quality
python -m unittest discover -s .agents/skills/ameriflux-era5-to-ecosim -p 'test_converter.py'
python -m unittest discover -s .agents/skills/ecosim-climate-readiness/tests
python -m unittest Tools.test_case_preflight
Usage
To execute the skill, run the following command from the project root. The resulting JSON will be saved to the ./result/ directory:
python .agents/skills/ameriflux-era5-to-ecosim/era5_to_ecosim_converter.py \
--input data/AMF_US-Ha1_FLUXNET_FULLSET_1991-2020_3-5/AMF_US-Ha1_FLUXNET_ERA5_HR_1981-2021_3-5.csv \
--output result/US-Ha1/US-Ha1_ecosim_climate.nc \
--site-id US-Ha1 \
--in-situ data/AMF_US-Ha1_FLUXNET_FULLSET_1991-2020_3-5/AMF_US-Ha1_FLUXNET_FULLSET_HH_1991-2020_3-5.csv \
--quality-report result/US-Ha1/US-Ha1_era5_quality_report.json \
[--utc-offset -5] [--time-registration auto|check|off] [--time-shift-hours 1]
Omit --in-situ to use automatic discovery. Use --skip-in-situ-comparison only when the task explicitly excludes tower validation.
Key Features
- Handles physically invalid source data by interpolation before NetCDF writing
- Properly converts precipitation from half-hourly to hourly values (summing)
- Averages temperature, wind speed, and solar radiation over half-hour periods
- Compares generated values with measured-only AmeriFlux meteorology when available
- Emits variable-specific warnings for large bias or RMSE
- Supports multiple years of data
- Creates valid netCDF files with proper metadata
Limitations
- Calendar padding that is not present in the source data may still use EcoSIM fill values.
- In situ comparison depends on overlapping FULLSET observations and their QC flags.
- Warning thresholds diagnose large discrepancies but do not establish that ERA5 or the tower is universally correct.
- Does not automatically bias-correct climate forcing.
Requirements
- Python 3.6+
- pandas
- numpy
- netCDF4
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |