Ruleset 1.1.0 · d7ae66670a5c

How a forecast gets a score

A rank is published after 90 days scored and is an early read until 180. The headline uses days 1–5. Days 6 and 7 stay off that score.

How we grade is the same rules in plain English.

Forecast Scoreboard compares the National Weather Service and weather models on the same climate day. A meteorologist is scored only after they opt in. Television stations are not graded. These rules are the file the scorer reads. The page and the arithmetic use the same version. A score is a function of confirmed forecasts, final observations, and this file. Nothing commercial is an input.

What is ranked

The published ranks are weather models, against the National Weather Service on the same issue dates. The Weather Service is the baseline, so it is not ranked against itself. Television stations are not graded, and a station page does not show a station score.

A meteorologist is scored only after they opt in, and only on public forecasts they post or share. The rules are the same as for a model, including no rank until 90 days scored (matched days). They can ask to be removed at any time. A staff page, a shift, or a chyron is not an opt-in. rank_individuals in this version is false, so this version does not publish a meteorologist board.

The scoreboard covers every US television market. Each market is checked at one official airport climate site, the station ACIS and the NWS daily climate report use, not a downtown point and not every climate in the market. Names, call letters, and the airport for each market are on the markets page. The committed list is weatherman/markets.json.

Kansas City stays on Kansas City International (KMCI, grid EAX 38,60) and Oklahoma City stays on Will Rogers World (KOKC, grid OUN 94,90). Those two geometries were retrieved from api.weather.gov on September 26, 2026. The other sites were retrieved on September 27, 2026. Ottumwa Regional is an ACIS climate station that is not in the NWS station index, so its coordinates come from ACIS. A market with no climate-report location id is still checked with ACIS.

Which forecast counts

The forecast of record is the last confirmed version captured at or before 23:00 local time. Day 1 is the next calendar day. A capture after the cutoff belongs to the next issue date.

Each stored forecast records a source type: on-air graphic, vendor widget, station-modified values, NWS point forecast, or a local fixture. The National Weather Service document is a structured feed. A model file is a structured grid. A meteorologist’s forecast counts only when they have opted in and the forecast is one they posted or shared in public.

Lead day is recomputed when the page is scored, from the capture time, the valid date a reviewer confirmed, and the cutoff in this file. Forecasts already stored keep their bytes. Changing the cutoff changes which capture is the forecast of record; it does not rewrite the archive.

The climate day

High, low, and precipitation are the official climate-day values: midnight to midnight, local standard time. In summer that window is 1:00 a.m. to 1:00 a.m. daylight time. That is the day the National Weather Service climate report describes, and it is the day ACIS returns for the airport station.

Primary source is ACIS (data.rcc-acis.org, elements maxt, mint, pcpn). The NWS daily climate report on api.weather.gov is the fallback, because that product list only keeps about a week of reports. Every report we fetch is copied into the local archive. A report whose text says “VALID TODAY AS OF” is preliminary. It is stored and it is not scored. A final report, or a complete ACIS day, replaces it.

If ACIS and a final climate report both have values and they disagree by half a degree or more, or they disagree on whether the day was a precipitation event, the row is flagged. The scored values stay with ACIS.

The low temperature

Version 1 scores the low printed for a forecast date against the climate-day minimum of that same date. Television graphics usually mean the overnight low, the one that falls on the night after that day’s high and often after local-standard midnight. That overnight low can belong to the next climate day.

An overnight window, built from hourly observations, is the right sensitivity test and it is not what this version scores. The observation table has a column for it. The column is empty. Absolute low error can include that mismatch. Skill against the NWS forecast is still a comparison of like with like: the baseline low is the NWS overnight period, scored against the same climate-day minimum.

The high is the climate-day maximum of the labeled date. That lines up with what a 7-day graphic means by “high.”

Precipitation

An event is at least 0.01 inch on the climate day. A trace is not an event, which matches the usual “measurable rain” line. The Brier score uses 1 for an event and 0 otherwise.

For the NWS baseline, the daily probability is the daytime period on the gridpoint forecast. The low is the nighttime period that starts on that same local date. Television station probabilities are not collected.

A blank probability on a graphic is scored as 0%, because that is what a viewer sees when the number is absent. An unreadable number is missing, not zero, and it stays out of the score until someone fills it in.

Metrics

Temperature error is forecast minus observed. Mean absolute error is the average of the absolute errors. Bias is the average of the signed errors. Positive bias means the forecast was too warm. “Within 3°F” is reported beside the high and is not part of the composite.

Precipitation uses the Brier score, the average of (probability − outcome) squared, with probability as a fraction from 0 to 1.

Skill against the NWS forecast captured for the same issue date is 1 − (this forecast’s mean loss / the NWS mean loss). Loss is absolute error for temperature and squared error for precipitation. Zero is a tie with the NWS. Positive means a lower loss. If both losses are zero, skill is zero. If the NWS loss is zero and this forecast’s is not, that component is left out rather than turned into an infinite penalty.

The composite

The headline score uses days 1 through 5. Days 6 and 7 are shown beside the National Weather Service baseline and are not in the composite. Extended forecasts have little skill and are easy to market; they should not decide a rank.

Model leaderboards are split so a one-day model is not graded as if it forecast the week. Next day is lead 1 only, with these same variable weights, and every model that has a lead-1 forecast can rank. Full week is leads 1 through 7, equal weight on each lead that both the model and the National Weather Service forecast, then these variable weights. A source whose files do not cover seven days, including NAM and HRRR, is not eligible and is not given that number. Both still use skill against the National Weather Service, and both still wait for 90 matched days.

Lead weights:

• Day 1: 0.30
• Day 2: 0.25
• Day 3: 0.20
• Day 4: 0.15
• Day 5: 0.10

Variable weights:

• High: 0.35
• Low: 0.30
• Precipitation probability: 0.35

The composite is the weighted mean of the component skills. If a component is missing, it is dropped and the remaining weights are scaled so they still sum to 1. An equal-weight mean of the components is computed for the record and is not published as a station grade.

The whole stored record is scored (all_confirmed_days_on_file). A fixed 12-month season is not computed yet. When it is, it should be a separate published number, not a silent replacement of this one.

Matched days

A model is scored on the days it published, against the National Weather Service on those same issue dates. A source with no confirmed forecast yet does not block the others and does not get a score. The rules file still has a station comparison set. That set is not published as a rank.

A day counts as matched for a model lead and a variable when the model and the NWS baseline both have a value, and the final observation exists. The record length used for “provisional” and “established” is the number of issue dates with a matched day-1 high. The composite uses those issue dates. On each of them, a longer lead counts only when that lead is matched too.

Model ranks in different markets do not have to share calendar days. The NWS error on a market page uses every day the baseline forecast and a final observation both exist. Station-versus-station tables are not published.

When a rank is published

Under 90 matched days, a model is still collecting. The arithmetic can be shown. It is not a rank.

At 90 matched days the rank is provisional. At 180 it is established. An official all-season ranking is a later milestone and is not claimed here.

Rows stored as sample data are excluded from every public page. They are not a rank.

Intervals and ties

The 95% interval is a percentile bootstrap of the composite, resampling the matched day-1 issue dates 2,000 times. The random seed is a hash of this ruleset and the station, so the same archive and the same file always produce the same interval.

Models are ordered by the point estimate. If an interval overlaps the model above, the page marks them statistically tied. Overlapping intervals are not a license to treat a small sample as a settled order.

Review

An extracted graphic does not count until a person confirms it or corrects it. The original extractor output is kept. Corrected values are what the scorer reads. Rejected extractions are ignored. The NWS document is confirmed on ingest because it is a structured feed, not an OCR guess.

There is no public corrections form yet. Until there is one, the review queue is the corrections path, and it should sit behind an admin token on any shared host.

Models

A model is a forecast source with role model. Models are scored against the NWS baseline, with the same mean absolute error, Brier score, skill, weights, and sample thresholds. station_comparison_set_excludes_models is true. Station-versus-station tables are not published.

GFS daily values come from the 0.25° NOAA Open Data files. The collector reads the index, fetches only the 2 m maximum temperature, 2 m minimum temperature, and surface precipitation messages, and keeps the nearest grid point. The daily high is the maximum of the four 6-hour TMAX buckets on the local-standard climate day. The low is the minimum of the four TMIN buckets. Precipitation is the sum of the four 6-hour accumulations, not the overlapping run-total. A day missing any bucket is dropped.

GFS does not issue a probability. At ingest, a climate-day total at or above 0.01 inch is stored as 100%, and a smaller total as 0%. The threshold used is written into the archived JSON. Temperature skill does not use that conversion. Lower error wins. Shared issue dates are those on which both the model and the National Weather Service have a day-1 high and low and the observation is final.

Older forecasts, where a public archive exists, are scored with the same rules. Each archived model row is that calendar day's 12Z open-data run, captured as of 12:00 UTC, which is before the 23:00 local cutoff in every US time zone. NAM is the exception: its scored files cover 36 hours, and a 12Z run ends before the next local midnight, so the archive uses the 00Z run early the next UTC date. That instant is still before 23:00 local. The archived National Weather Service row is the National Digital Forecast Database maximum temperature, minimum temperature, and 12-hour precipitation probability at the nearest grid point, from the morning file nearest 12:00 UTC. That mosaic is not the Weather Forecast Office grid the live feed returns. Canadian GEM and ICON open-data servers do not keep this window, so those models are not in the archive. Television station forecasts are not backfilled. Archive rows are labeled on the leaderboard.

ECMWF IFS, ECMWF AIFS, Canadian GEM, ICON, NAM, and HRRR are collected from each provider’s public files and scored on the same airports, the same observations, and the same 90-day threshold as GFS. Each model is graded on its own: a short run does not erase a day another model published. IFS uses 3-hour and 6-hour 2 m extremes. GEM and ICON use 6-hour extremes. AIFS open data and the HRRR surface file do not include a period maximum, so those daily highs and lows are the max and min of the samples they do publish. NAM's hourly extremes run through 36 hours and HRRR through 48, and only the climate days those hours cover are scored. Alaska and Hawaii are outside the NAM and HRRR grids, so those airports have no NAM or HRRR score. GEML is public and not collected yet. UKMET’s global grid and Google’s operational model output are not anonymous public downloads, so they are listed and skipped. Credits are on the model catalog.

Events

Hurricane track error is the great-circle distance in statute miles (Earth radius 3,958.7613 miles) at leads 24, 48, 72, 120 hours. The forecast-track page shows that distance in statute miles and wind in miles per hour, and it can include 96 hours when the deck has a paired fix. That lead stays out of the score table. Intensity error in the file is the absolute difference in knots. A forecast is paired with the NHC operational b-deck fix at the same valid time. Positions are not interpolated. Skill uses the same ratio as the daily board, against the NHC official forecast (OFCL) from that issue time. Public a-deck aids ingested for scoring are GFS (AVNO), GEFS mean (AEMN), Canadian GEM (CMC), UKMET (UKX), HAFS-A, HAFS-B, HWRF, HMON, and NAVGEM. Interpolated aids and ensemble perturbations are not the primary score. Google DeepMind is the Weather Lab operational ensemble mean (ATCF tech FNV3), and WeatherNext 3 is the WeatherNext 3 ensemble mean (WNV3), from deepmind.google.com/science/weatherlab. A missing file is skipped. Wind miss is scored when that file includes a wind. ECMWF AIFS, ECMWF HRES, and the ECMWF ENS mean are read from ECMWF open-data tropical-cyclone BUFR. ECMWF HRES is the IFS high-resolution oper track. The ECMWF ENS mean is the average of the member centers in the IFS ensemble track file at that lead. A member with no center at that lead is left out of the average. AI GFS is the NOAA AIGFS tracker file (ATCF tech AGFS) on NOMADS. That directory keeps about two days, and a cycle that has aged off is skipped rather than filled in. These aids use the same pairing rules as the a-deck aids.

Landfall location and timing are scored only when an explicit landfall row is stored. The track is not scanned for a coastline crossing.

Snow, heavy rain, and local storm impacts may be a range. The absolute error is the distance from the observation to the midpoint. A hit means the observation is inside the inclusive range, so 6 inches hits a 4–8 inch forecast and the midpoint error is 0. A single number is a range whose ends are equal. Verification sources, when collection starts, are NWS Public Information Statements, CoCoRaHS, and ASOS for snow and rain, and SPC storm reports for severe weather. Those collectors are not running, and no sample storm is mixed into the real event list.

After the last date of a severe storm or a flood, the final scorecard grades the afternoon Day 1 outlook for each event date against the stored National Weather Service local storm reports. A severe storm uses the Storm Prediction Center Day 1 categorical outlook issued at 1630 UTC, the archive file day1otlk_YYYYMMDD_1630_cat.lyr.geojson. A flood uses the Weather Prediction Center Day 1 Excessive Rainfall Outlook issued at 16 UTC, the archive shapefile shp_94e_YYYYMMDD16.zip. Other issuances that day are not substituted. A report counts when its time is at or after that issuance and before 12 UTC the next day. A report exactly at 12 UTC is outside the window, and so is a report between 12 UTC and the issuance. A report on the outline counts as inside. A point in more than one outline is counted once, at the higher risk. General thunder on the categorical outlook is not a Marginal, Slight, Enhanced, Moderate, or High risk. The test is the outline itself, not the Storm Prediction Center neighborhood that treats a nearby report as inside. A day whose archive file could not be fetched has no score. A graded day is not replaced on a later collect.

World leaderboard

The world leaderboard scores ECMWF IFS, ECMWF AIFS, GFS, and Canadian GEM (MSC GDPS) at 50 major cities. The check is the city's main airport. IFS, AIFS, and GFS use the nearest 0.25° open-data point. GEM uses the nearest 0.15° point from the MSC Datamart GDPS GRIB2, the same files the US board already downloads. Each forecast is reduced to the local-standard climate day the same way as the US board. IFS uses the 3-hour and 6-hour 2 m extremes. AIFS uses the max and min of the 6-hour instantaneous 2 m temperatures, which is not a true daily extreme. GFS uses the four 6-hour TMAX, TMIN, and APCP buckets. GEM uses the max of the 6-hour 2 m maxima, the min of the 6-hour 2 m minima, and the 6-hour precipitation accumulation. The forecast of record is the last stored cycle at or before 23:00 local.

The observation is a routine METAR from NOAA's Aviation Weather Center, not the National Weather Service climate report. The daily high is the highest temperature in those reports on the local-standard climate day, and the low is the lowest. A 6-hour maximum or minimum in a report is included. A day counts only when the reports cover at least 18 distinct hours, so a partial day is not treated as the extreme. Rain is yes or no. A forecast of at least 0.01 inch is a yes. A METAR day is a yes when any report carries a precipitation type (rain, drizzle, snow, ice pellets, hail, or unknown precipitation) or a measured amount of at least that threshold. A trace does not count. Present weather is the weather at the report time, so a shower between reports can be missed.

There is no National Weather Service point forecast at these airports, so the world rank is not a skill score against the NWS. It is the average absolute error of the high and the low. Days 1 through 5 use the same lead weights as the US board. The high and low weights are the published temperature weights, renormalized, because rain is not on the same scale. Rain is a yes-or-no hit rate, shown beside the rank, and it is not part of the rank. A city-day counts when IFS, AIFS, GFS, and Canadian GEM each have a high and a low and the METAR day is complete. The record length is the number of issue dates with a matched day-1 high. Under 90 matched days the row is still collecting. The miss can be shown. It is not a rank. At 90 days the rank is provisional. At 180 it is established. Scores start from the first day the collect stores. Nothing is backfilled, and no number is invented.

Contains ECMWF data. ECMWF IFS and ECMWF AIFS open data are used under the Creative Commons Attribution 4.0 International licence (CC BY 4.0). GFS and the METARs are U.S. government works. Canadian GEM is MSC GDPS output from the MSC Datamart and contains information licensed under the Open Government Licence – Canada. This board does not replace the US market pages, which use the climate report.

Left out of this version

• Sky cover and precipitation timing on the daily board.
• An overnight-low window from hourly observations.
• A metro-wide precipitation check. The daily event is the single official site.
• Climatology and persistence baselines. The published daily comparison is the NWS point forecast.
• Television station grades.
• A public meteorologist board before an opt-in request is approved.
• HURDAT2 as a replacement for the operational b-deck. The b-deck is what is stored, and it is revised after the season.
• Inferred landfall, and live snow or SPC report collection.