Written by: Olivier Lam, Physical AI Team, Jua.ai AG
Key Takeaways
- European TSOs and energy traders use six standardized metrics (nRMSE, MAE, CRPS, bias, skill scores, and spatial-aggregation effects) to judge renewable forecast quality across wind and solar portfolios.
- Each metric captures a specific aspect of performance. nRMSE and MAE quantify point accuracy, CRPS assesses probabilistic calibration, bias reveals systematic directional error, skill scores benchmark against references like ECMWF, and spatial aggregation shows how portfolio-level errors shrink.
- Better values on these metrics reduce imbalance penalties and balancing costs for BRPs that operate onshore wind, offshore wind, and solar PV assets in European day-ahead markets.
- Forecast accuracy depends on technology and horizon. Day-ahead nRMSE, MAE, and CRPS values shift with season, geography, and lead time, while portfolio aggregation adds further error reduction.
- Schedule a live benchmark with Jua to compare EPT-2 and EPT-2e against ECMWF models on your own region and variables using all six metrics in seconds.
Executive Summary
Renewable energy forecast evaluation rests on three lenses: model capability, operational usability, and reliability. Model capability describes how accurately the physics model predicts atmospheric variables. Operational usability covers cadence and resolution for the trading horizon. Reliability focuses on whether probabilistic outputs are calibrated and free of systematic bias. No single metric spans all three areas, so practitioners combine several.
nRMSE and MAE measure deterministic point accuracy. CRPS measures probabilistic calibration. Bias captures systematic directional error. Skill scores benchmark performance against a reference model. Spatial-aggregation effects explain why portfolio-level errors are structurally lower than asset-level errors.
Typical day-ahead nRMSE values for European onshore wind, offshore wind, and solar PV vary with season and geography. CRPS values for probabilistic day-ahead wind and solar forecasts in Europe also vary. These metrics depend on technology and forecast horizon, and every improvement in accuracy reduces imbalance exposure. Jua’s EPT-2 model delivers forecasts that outperform ECMWF HRES across all forecast horizons on RMSE, with EPT-2e beating the 50-member ECMWF ENS mean on both RMSE and CRPS at virtually every lead time, as documented in arXiv:2507.09703.
Jua is a foundation model and agent company. Jua for Energy is the first applied product, built on EPT, a general physics foundation model, and Athena, an AI agent. The Jua platform benchmarks more than 25 models live and reports all six metrics on any region, variable, and time window in seconds.
See these metrics on your own portfolio by running a live benchmark of EPT-2 and EPT-2e against your current forecast provider.
nRMSE: Linking Forecast Error to Imbalance Risk
Normalized Root Mean Square Error is the square root of the mean squared forecast error divided by a normalizing factor, typically installed capacity or mean observed generation. This normalization makes nRMSE comparable across sites and technologies with different absolute output levels. A wind farm with 500 MW installed capacity and an nRMSE of 12% carries an average quadratic error of 60 MW on its day-ahead forecast.
In European day-ahead markets, nRMSE values vary by technology and lead time. Values usually increase at longer lead times across all technologies. nRMSE penalizes large errors more heavily than small ones, because squaring magnifies big deviations. A single missed wind ramp can contribute more to nRMSE than several smaller errors of the same total magnitude. TSOs therefore favor nRMSE when assessing balancing reserve requirements, because tail errors drive procurement costs.
For balancing-responsible parties, nRMSE maps directly to imbalance exposure. Under most European imbalance settlement mechanisms, forecast deviations above a threshold trigger penalty payments. A BRP operating a 1 GW wind portfolio that reduces nRMSE by four percentage points saves approximately €1.5 million per year in hedging and imbalance costs under typical European penalty structures.
MAE: Interpreting Average Error in Operational Terms
Mean Absolute Error is the average of the absolute differences between forecast and observed values, normalized by installed capacity or mean generation for cross-site comparison. Unlike nRMSE, MAE weights all errors equally, regardless of magnitude. This property makes MAE more robust to outliers and easier to interpret in daily operations. An MAE of 8% means the forecast is, on average, 8% of capacity away from actual generation.
MAE values for day-ahead forecasts in Europe vary by technology. MAE is always lower than nRMSE for the same forecast, because squaring large errors inflates RMSE. The ratio of nRMSE to MAE, sometimes called the error distribution shape, indicates whether forecast errors concentrate in a few large events or spread more evenly. For wind, this ratio typically lies between 1.2 and 1.6, reflecting the leptokurtic distribution of wind power errors driven by ramp events.
Research on Belgian offshore wind power forecasts shows that aggregate metrics such as MAE can hide deficiencies in capturing rapid power fluctuations that matter for grid operations. Practitioners therefore pair MAE with CRPS and ramp-specific verification frameworks instead of relying on MAE alone.
CRPS: Measuring Probabilistic Forecast Quality
The Continuous Ranked Probability Score measures the accuracy of a full probabilistic forecast, such as an ensemble or predictive distribution, rather than a single deterministic value. CRPS is the integral of the squared difference between the forecast cumulative distribution function and the step function at the observed value. Lower CRPS values indicate sharper and better-calibrated probabilistic forecasts. When the forecast is deterministic, CRPS reduces to MAE, which turns CRPS into a generalization that supports direct comparison between probabilistic and point forecasts.
CRPS values for wind power and solar PV probabilistic forecasts vary across Europe. Studies of wind power ramping event prediction show that a Transformer model can achieve the lowest deterministic MAE and probabilistic CRPS while still producing overly smoothed outputs that reduce skill at detecting actual ramps. This behavior illustrates that CRPS alone does not capture every operationally relevant property.
EPT-2e, Jua’s ensemble variant of EPT-2, beats the 50-member ECMWF ENS mean on CRPS at virtually every lead time, as documented in arXiv:2507.09703. Traders who price weather-derivative positions or manage intraday balancing exposure rely on CRPS-optimized ensemble forecasts to obtain the probabilistic calibration required to size positions correctly.
Bias: Managing Systematic Over- and Underprediction
Bias is the mean signed error, defined as the average of forecast minus observed over a period. A positive bias means the forecast systematically overpredicts generation. A negative bias means it underpredicts. Bias appears in the same units as the forecast variable or as a value normalized by capacity.
Bias has a direct and asymmetric impact on imbalance penalties. A BRP with a persistent positive wind bias will systematically over-schedule generation in the day-ahead market and create a long position that settles at the imbalance price. In markets where the imbalance price usually sits below the day-ahead price during surplus periods, as in many high-renewable European grids, a positive bias becomes structurally costly. A negative bias creates the opposite exposure. TSOs monitor fleet-level bias across BRPs as an indicator of systematic model error. Persistent bias in a single direction can trigger regulatory scrutiny under balancing responsibility frameworks such as those applied by National Energy System Operator in Great Britain.
Bias correction is a standard post-processing step in operational forecasting pipelines. This step works best when the underlying model behavior remains stable and well characterized. Models that shift bias between seasons or regimes, which is common in NWP models with fixed parameterization schemes, are harder to correct reliably.
Skill Scores: Comparing Models Against ECMWF
A skill score measures forecast performance relative to a reference forecast and expresses the result as a fractional improvement. The general form is Skill = 1 − (metric_forecast / metric_reference). A skill score of zero means the forecast performs identically to the reference. A positive score means it outperforms the reference. A negative score means it underperforms. The choice of reference model determines what the skill score represents.
In European energy markets, ECMWF HRES serves as the standard reference for deterministic skill, and ECMWF ENS serves as the reference for probabilistic skill. ECMWF HRES has held the gold standard in numerical weather prediction for more than forty years, so a positive skill score against HRES carries real operational weight. EPT-2 achieves positive skill scores against ECMWF HRES at every lead time from 0 to 240 hours on 10 m wind, 100 m wind, 2 m temperature, and surface solar radiation (see arXiv:2507.09703 for full results). EPT-2e achieves positive skill scores against the ECMWF ENS mean on both RMSE and CRPS at virtually every lead time.
Skill scores matter most for procurement decisions. A meteorologist who evaluates a new forecast provider needs to know not only the absolute error but also whether the new model outperforms the incumbent on the variables and regions that drive the desk’s P&L.
Spatial-Aggregation Effects Across Wind and Solar Portfolios
Forecast errors at individual wind or solar assets are only partially correlated across geography. When a portfolio aggregates errors from spatially distributed assets, the uncorrelated components cancel and reduce portfolio-level nRMSE below the asset-level average. This spatial-aggregation effect is one of the most operationally significant properties of renewable forecast error for large BRPs and TSOs.
The magnitude of the aggregation benefit depends on the spatial extent of the portfolio, the meteorological correlation length scale of the dominant error source, and the technology mix. For European onshore wind portfolios that span multiple bidding zones, portfolio-level nRMSE is typically lower than the average single-asset nRMSE at day-ahead horizons. For solar PV, where cloud-cover errors are more spatially correlated within a synoptic weather system, the aggregation benefit is usually smaller at the same horizon.
Normalization practice also matters. nRMSE normalized by total portfolio capacity will appear lower than nRMSE normalized by individual asset capacity, even when the underlying physics is identical. TSOs and BRPs must specify the normalization basis when they compare forecasts across providers. Beyond normalization, the forecast model’s native resolution determines whether it can represent the spatial error structure that drives aggregation benefits. EPT-2’s native 5 km resolution over Europe, delivered via EPT-2 HRRR, preserves the spatial structure of wind and solar fields at the scale relevant to individual asset dispatch and supports accurate aggregation from the asset level up to the portfolio level.
Typical 2025 Metric Ranges for European Day-Ahead Forecasts
Day-ahead forecast accuracy in Europe varies significantly by technology, season, and geography. Onshore wind nRMSE often falls in the low double digits as a percentage of installed capacity, while offshore wind typically performs slightly better because of smoother wind fields. Solar PV usually shows lower nRMSE and MAE than wind at the same horizon, although cloud regimes can push errors higher in specific regions or seasons.
Probabilistic metrics such as CRPS follow similar patterns. Better deterministic accuracy usually coincides with lower CRPS, but local weather dynamics and model calibration also play strong roles. Bias for well-tuned portfolios tends to cluster near zero, with typical ranges of a few percentage points of capacity. Spatial aggregation usually reduces portfolio-level nRMSE below the average single-asset nRMSE, especially for geographically broad onshore wind portfolios.
Because these values depend strongly on site conditions and portfolio design, Jua recommends evaluating forecast performance through live benchmarking on your specific assets rather than relying on generic industry averages. The Jua platform computes all six metrics on your chosen region, technology mix, and time window in seconds.
From Metrics to Day-Ahead Bidding and Imbalance Costs
Day-ahead electricity markets in Europe close between 12:00 and 15:00 CET for delivery the following day. BRPs submit generation schedules based on their best available forecast at gate closure. Deviations between the submitted schedule and actual generation settle at the imbalance price, which the TSO sets based on the cost of activating balancing reserves.
nRMSE forms the primary link between forecast quality and imbalance cost. A 1% reduction in nRMSE on a 1 GW wind portfolio reduces expected imbalance volume by roughly 10 MW on average, with disproportionate reductions in tail events that carry the highest imbalance prices. Jua’s forecasts carry an estimated $1.5 million P&L impact per gigawatt annually in European energy markets. A 1 GW solar portfolio with the same four-percentage-point accuracy gain saves approximately €3 million per year.
Bias sets the direction of imbalance exposure. A BRP with a +2% capacity bias on a 500 MW solar portfolio will over-schedule by an average of 10 MW every hour of the day-ahead period and create a systematic long position settled at the imbalance price. CRPS determines the quality of probabilistic information available for hedging. A well-calibrated ensemble with low CRPS allows the BRP to size hedging positions in the intraday market in proportion to forecast uncertainty instead of applying a fixed conservative margin. Skill scores against ECMWF HRES provide procurement-level evidence that a new forecast provider adds measurable value over the existing stack.
How Jua for Energy Delivers and Reports These Metrics
The Jua platform includes a live model benchmarking surface that reports all six metrics, including nRMSE, MAE, CRPS, bias, skill scores, and spatial-aggregation-adjusted portfolio errors, across more than 25 models simultaneously. The benchmark runs on any region, any variable, and any time window, and returns results in seconds. Models on the platform include EPT-2, EPT-2e, ECMWF HRES, ECMWF ENS, ECMWF AIFS, NOAA GFS, Microsoft Aurora, GFS GraphCast, DWD ICON Global, ICON-EU, and others, all under a unified schema and a single API.
EPT-2, Jua’s deterministic flagship, achieves the benchmark performance against ECMWF HRES described earlier and documented in arXiv:2507.09703. Building on this deterministic foundation, EPT-2e extends the advantage to probabilistic forecasting and beats the 50-member ECMWF ENS mean on both RMSE and CRPS at virtually every lead time. This probabilistic capability arrives at operationally relevant resolution. EPT-2 HRRR provides forecasts at a native 5 km scale over Europe and preserves the spatial structure required for accurate asset-level and portfolio-level error decomposition, with updates four times daily to support intraday reforecasting workflows.
Athena, Jua’s AI agent instrumented with the Jua for Energy tool surface, turns a natural-language query into a full benchmark report, a backtest, or a custom widget in roughly 90 seconds. A meteorologist who evaluates EPT-2e against ECMWF ENS on German offshore wind CRPS can run that comparison directly on the Jua platform without writing any code. Quant developers who prefer programmatic access use the same metrics through the REST API or via pip install jua.
The three-part evaluation lens maps directly onto the platform’s output. EPT-2 addresses model capability through peer-reviewed benchmark numbers. EPT-2e’s calibrated ensemble and four-times-daily update cadence address operational usability and reliability. The live benchmarking surface makes all six metrics auditable by the customer on their own data.
Explore a live comparison to see EPT-2 and EPT-2e benchmarked against ECMWF HRES and ENS on your region and variables with all six metrics reported in real time.
Frequently Asked Questions
What is the difference between nRMSE and MAE for renewable energy forecasts?
nRMSE and MAE both measure average forecast error normalized by installed capacity or mean generation, but they weight errors differently. nRMSE squares each error before averaging, which means large individual errors, such as a missed wind ramp, contribute disproportionately to the score. MAE averages absolute errors equally and becomes more robust to outliers and easier to interpret operationally. For a given forecast, nRMSE is always equal to or greater than MAE. TSOs typically use nRMSE when they assess balancing reserve requirements, because tail errors drive reserve procurement costs. Traders often use MAE as a more stable day-to-day performance indicator. Reporting both metrics together reveals whether forecast errors concentrate in a few large events or distribute more evenly across the forecast period.
Why does CRPS matter more than MAE for day-ahead bidding?
MAE measures the accuracy of a single deterministic forecast value. CRPS measures the accuracy of a full probabilistic forecast, such as an ensemble or predictive distribution, against the observed outcome. For day-ahead bidding, traders need both the expected generation and the distribution of possible outcomes to size hedges. A well-calibrated probabilistic forecast with low CRPS enables a BRP to quantify imbalance risk at each quantile of the distribution and hedge accordingly in the intraday market. A deterministic forecast with low MAE provides no direct information about uncertainty. Because CRPS reduces to MAE when the forecast is deterministic, a model with lower CRPS than a competitor is at least as good on MAE and usually better on probabilistic calibration.
How do European TSOs use these metrics in imbalance settlement?
European TSOs use forecast accuracy metrics mainly to set balancing reserve requirements and to monitor BRP compliance with balancing responsibility obligations. Under most European imbalance settlement frameworks, BRPs submit day-ahead generation schedules and remain financially responsible for deviations from those schedules at the imbalance price. TSOs monitor fleet-level nRMSE and bias across BRPs to identify systematic model errors that could destabilize the balancing market. Persistent positive bias, which corresponds to systematic over-scheduling, creates a structural long position in the balancing market that TSOs must absorb through downward regulation. Persistent negative bias creates the opposite exposure. Some TSOs publish aggregate forecast accuracy statistics for their balancing zones as part of transparency reporting, which allows BRPs to benchmark their own forecast quality against the fleet average.
What is a good skill score for a wind power forecast against ECMWF HRES?
Any positive skill score against ECMWF HRES is operationally meaningful, because ECMWF HRES has been the gold standard in numerical weather prediction for more than forty years and serves as the reference model that most European energy companies already use. A skill score of 5–10% on nRMSE at 24-hour lead time represents a substantial improvement in day-ahead bidding accuracy. EPT-2 achieves positive skill scores against ECMWF HRES on every lead time from 0 to 240 hours on 10 m wind, 100 m wind, 2 m temperature, and surface solar radiation, as documented in arXiv:2507.09703. For probabilistic forecasts, EPT-2e achieves positive skill scores against the ECMWF ENS mean on both RMSE and CRPS at virtually every lead time, which is a demanding benchmark because the ENS mean aggregates 50 ensemble members.
How much does spatial aggregation reduce forecast error for a European wind portfolio?
Spatial aggregation usually reduces portfolio-level nRMSE relative to the average single-asset nRMSE for European onshore wind portfolios that span multiple bidding zones. Forecast errors at geographically dispersed assets are only partially correlated, so the uncorrelated components cancel when aggregated. For offshore wind, the reduction is typically smaller, and for solar PV the benefit is also usually smaller, reflecting the higher spatial correlation of cloud-cover errors within a synoptic weather system. The magnitude of the benefit depends on the spatial extent of the portfolio, the meteorological correlation length scale of the dominant error source, and the normalization basis used. A BRP that reports portfolio-level nRMSE normalized by total installed capacity will always show lower values than one that reports asset-level nRMSE. Both approaches are correct, but they are not directly comparable without specifying the normalization basis. High-resolution forecast models that preserve spatial structure at the 5 km scale support more accurate aggregation from the asset level up to the portfolio level.
Does Jua for Energy report all six metrics on a single platform?
Yes. The Jua platform’s live benchmarking surface reports nRMSE, MAE, CRPS, bias, skill scores, and spatial-aggregation-adjusted portfolio errors across more than 25 models simultaneously on any region, variable, and time window selected by the user. The benchmark compares EPT-2, EPT-2e, ECMWF HRES, ECMWF ENS, and all other models on the platform under a unified schema, with results available in seconds. Athena, Jua’s AI agent, can generate a full benchmark report, backtest, or custom widget from a natural-language query in roughly 90 seconds. Programmatic access to all metrics is available through the REST API and the Python SDK.
Run a Live Benchmark on Your Region
Every claim in this reference guide is auditable on the Jua platform. You can select your region, such as German onshore wind, North Sea offshore, Iberian solar, or any other European zone, choose your variables, and run a head-to-head comparison of EPT-2 and EPT-2e against ECMWF HRES and ENS on all six metrics. The benchmark returns in seconds and presents a clear, quantitative comparison.
See the comparison on your data and run a live benchmark of EPT-2 against your current forecast provider on your own region and variables across more than 25 models, with all six accuracy metrics reported on the spot.