Correcting Extremes
Introduction
As Cumulus processes each reading from your weather station, it checks that value (and any derived from it) against the extremes currently stored in various .ini files, and if necessary updates the extreme records that are affected. The extreme records that are maintained in this way are:
| Period | File | Example web page | Notes |
|---|---|---|---|
| For current day so far | today.ini | today.htm | Many entries in this file get transferred to dayfile.txt at end of day |
| For current month-to-date | month.ini | thismonth.htm | |
| For current year-to-date | year.ini | thisyear.htm | |
| For all readings since a 'start date | alltime.ini | records.htm | See table below for start date |
| For a particular month in all years | monthlyalltime.ini |
Following the links in the second column leads you to more information about the relevant file. Following links in third column leads you to more information about the web tags that you can incorporate in your own templates.
The purpose of this article is to help you to edit incorrect entries in any of the .ini files in the second column, to understand where else to investigate, and where necessary amend other log files.
The article starts with corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).
There is more information in Category:Log_Files and the pages relating to individual files, for all of the extreme holding files.
Why might your extremes need to be corrected?
If there is a rogue value read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.
It is also possible that you have discovered that you made a mistake in setting up or calibrating a sensor, and this leads you to identifying a constant/multiplier adjustment to be applied to adjust all values over the past period.
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
All-time extreme functionality
For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time. Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.
There were bugs introduced sometimes in builds of the original Cumulus (known now as Cumulus 1). Information about a few of the bugs and fixes can be found in File:Changes.zip, although that does not cover any 1.7.x versions, and does not detail bugs created and fixed within the beta builds. More information may be found by searching within Cumulus forum announcements, but it will require a lot of effort as there are a lot of posts to search. (For historic interest only, one example is that what is stored in month.ini and year.ini depends on when they were first created, because they are initiated from the daily summary log for the relevant period, an individual parameter can only be initialised if the corresponding field is present in dayfile.txt for the whole of that period).
Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The updates.txt that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in the relevant support forum will yield more information in return for a lot more effort.
The start date referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has added more extreme records to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
| Parameter | First released | First in Version | First in Build |
|---|---|---|---|
| highest/lowest apparent temperature | 26 Oct 2010 | 1.9.1 beta | 957 |
| highest/lowest feels like temperture | 24 June 2020 | 3.6.10 | 3086 |
| highest Canadian Humidity Index (humidex) | 28 Jul 2020 | 3.7.0 | 3089 |
| highest minimum temperature | 15 April 2004 | 1.2.2 | (lost) |
| highest USA heat index | 29 Aug 2010 | 1.9.0 beta | 955 |
| wettest month | 5 April 2004 | 1.2 | (lost) |
| highest daily windrun | 3 Jul 2011 | 1.9.2 beta | 1001 |
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked (lost), and there is some vagueness in information mentioned elsewhere in this page.
Correction of All Time Extreme records
Cumulus software makes it easy to correct the all-time extremes (held in alltime.ini).
This is because whenever Cumulus makes an update to that file, from version 1.8.9 onwards, it logs the previous and new values to the Alltimelog.txt file. Consequently, if you detect a rogue value, you can look up the latter file to determine what how to revert an entry in the former file.
Alternative ways to find a replacement for a rogue value
Of course, it is possible that the old value in Alltimelog.txt is not appropriate. It might be that after the rogue value was stored in alltime.ini, a new extreme was seen, and this new extreme was different to the previous value stored in Alltimelog.txt, but it did not cause an update in alltime.ini because of the rogue value that was stored there being more extreme.
Looking at graphical representations
One way to check whether the above speculation is correct is by looking at Cumulus 1 select-a-graphs (available from version 1.2 released on 5th April 2004), or MX charts in the admin interface (available in , if you can find a plot that covers the period between when the false extreme was recorded and now you are ready to do a correction, you can look for evidence of a new extreme that has been missed. (If you only have access to your web pages, then the sample trends web page may, or may not, cover the relevant period). Some plots record values every minute, and those high resolution plots are ideal for your search.
Using the Cumulus backup
If for any reason, the Alltimelog.txt cannot be used (maybe you deleted it in error), and you are within a week of when the rogue value updated the extreme, an alternative is to compare the alltime.ini with a previous version of the same file that has been stored in a backup directory. Basicallly, Cumulus takes a backup of most of the active data files when it starts up, and also at the start of the meteorological day (just after midnight for a lot of users). The backups are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours).
If you can find the latest backup stored for a date and time from just before the alltime.txt was updated with the rogue value, look in that backup copy to see the previous value that was in the file, and use that value when you follow editing instructions below.
Searching your standard log files
Any search through your monthly Standard_log_files has two disadvantages:
- those lines only record a small sample of spot values as by default they are only created every 10 minutes (so even if you choose to load all log files, it might not include the actual extreme you have missed recording).
- as the files are typically large, it takes some time to load the contents of the files.
Both the original Cumulus (it is not clear from which version, it might be from 1.7.x) and MX (from version 3.2.0 build 3056) provide all time record editors. These editors have the ability (as described below) to load up both the daily summary log file and all your Standard_log_files, so you can look at the extremes that are recorded in those files, and use that information to determine what to replace your rogue value with.
The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files.
Using the provided editors
Both Cumulus 1 and MX provide an editor (at most versions of the relevant flavour, update to a new release if your build does not include an editor).
When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:
select All-time records in the edit menu accessed from main screen in Cumulus 1,
select All Time records in the edit menu of MX admin interface.
The way the editor allows you to change a value depends on which flavour you are using:
- Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across). Press Save button to retain the change and exit.
- You have to select either a value or a time-stamp that you want to change before you can type in a replacement value (you have to type the new number in even if it is displayed by loading the log files). Then click the tick to retain the change, and that will allow you to select another value or time-stamp to edit, before you finally exit the editor.
Please remember any edit you make here will not affect the related extreme in your monthly-all-time extremes recorded in monthlyalltime.ini. Nor can you update Standard_log_files, or daily summary log, by any edit made here. If you are using MX, it has not affected related tables on your database server.
Actions after you have edited all-time entries
In all cases, an edit made to an entry in alltime.ini, means that you need to make the same change in monthlyalltime.ini. Look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed. How to edit the monthly-all-time extreme records are described in next section.
If you loaded up the monthly standard log files into the all-time editor, you should be able to see:
- if there is an entry for same time-stamp as that associated with the former rogue value as seen in Alltimelog.txt
- you should also be able to work out, from Alltimelog.txt, whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
If the correction you have made is in the current month, you will also need to change the entries in month.ini and year.ini. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same data folder, with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter.
The all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.
Correction of Monthly All Time Extreme records
From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
Initialisation of monthly-all-time extreme records
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one Copy button, the one in the header row, all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking OK button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see Calculate Missing Values).
How do I correct my monthly all-time records?
In many respects, the instruction for the all-time editing above also apply to monthly all-time.
- If you are a Cumulus 1 user, you do not have a monthlyalltimelog.txt file to look in to see the log entry with the previous and new values. This information was logged into a file in the diagnostic files folder. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.
- If you are a MX user, see if you have a monthlyalltimelog.txt file in the same data folder, and then you will know what value to revert to.
Here the old and new values can be looked up in the monthlyalltimelog.txt file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in Diags_folder.
There are the same alternative ways to look for the values to enter in the monthly-all-time entries that you need to change, as were described earlier to look up value for all-time. If you have already done an all-time edit, then look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed for monthly-all-time.
The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.
Actions after you have edited monthly-all-time entries
If you loaded up the monthly standard log files into the monthly-all-time editor, you should be able to see:
- if there is an entry for same time-stamp as that associated with the former rogue value as seen in monthlyalltimelog.txt
- you should also be able to work out, from monthlyalltimelog.txt, whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
If the correction you have made is in the current month, you will also need to change the entries in month.ini and year.ini. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same data folder, with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.
The monthly-all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.
Correction of extremes for today
The Cumulus Edit' menu includes a Today's rain option where you can adjust the total rainfall for today (e.g. if you or the wind have knocked your rain gauge) as described below. There is no facility provided to edit any other content of today.ini, but since today.ini is used to create lines in dayfile.txt, you can follow instructions in Amending_dayfile to make any necessary corrections for past days.
There is an error in today's total rainfall
Easy - correct today's total using the 'today's rain' editor on the edit menu.
- select 'Today's rain in the edit menu accessed from main screen in Cumulus 1,
- select Today's rain in the edit menu of MX admin interface.
This edit will actually alter the start of day rainfall counter figure:
- If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter
- If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter
Please note that this edit does not affect "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the Standard_log_files, your database will retain incorrect values, as these are not updated by this correction.
Please see Today.ini#Editing_rainfall_in_today.ini_within_Cumulus for details of how to edit related fields.
Correction of extremes for month-to-date
As mentioned in passing above, a rogue value may get recorded in a month.ini file.
You should ensure you have got the daily summary log (dayfile.txt) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this month.
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit This year's records. The editors work in exactly the same way as was described for all-time above.
- In Cumulus 1, Copy' buttons enable you to copy a record from dayfile. Click OK to save.
- In MX, the dayfile value/timestamp, and the Logfile value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click 'Tick to save.
Correction of extremes for past month
- Cumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
- From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the month.ini at the start of a new year, it saves the old month.ini (whenever it was last updated) as a file with a name like month201501.ini.
MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.
Correction of extremes for year-to-date
As mentioned in passing above, a rogue value may get recorded in a year.ini file.
You should ensure you have got the daily summary log (dayfile.txt) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this year.
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit This year's records. The editors work in exactly the same way as was described for all-time above.
Correction of extremes for past year
- Cumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
- From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the year.ini at the start of a new year, it saves the old year.ini (whenever it was last updated) as a file with a name like year2015.ini.
MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.
Some definitions
Rogue value
In this article, the term rogue value is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.
Here are a typical examples:
- it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
- perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
- an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong
Flavour, Release, Version, and Build
Flavour is used to represent the original Cumulus, Cumulus 2, and Cumulus MX, collectively. Where the text says applicability is dependant on flavour, it means that the action you do depends on whether you are installing/running Cumulus MX or the original Cumulus software.
Release is used to signify what the Cumulus developer makes available for download after there has been a modification to the software. For most Cumulus 1 builds, the release consisted of a executable that would create all the folders and files needed to run that software. For Cumulus 2, releases were a zip file and were numbered using a identifier with many digits. MX releases are as a zip file that is labelled with the build number.
Version here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.
Build number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number. For recent MX releases, the developer has changed version number every time there is a new build released.