Correcting Extremes: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
m (minor resequencing)
(6 intermediate revisions by the same user not shown)
Line 23: Line 23:
!style="width:300px"|Notes
!style="width:300px"|Notes
|-
|-
|For current day so far
! scope="row"|For current day so far
|[[today.ini]]
|[[today.ini]]
| Editor for "Today's rain" (no editor for other derivatives)
| Editor for "Today's rain" (no editor for other derivatives)
Line 29: Line 29:
|Many entries in this file (for non-midnight rollover, use is made of [[yesterday.ini]] too) get transferred to [[dayfile.txt]] at end of day.  
|Many entries in this file (for non-midnight rollover, use is made of [[yesterday.ini]] too) get transferred to [[dayfile.txt]] at end of day.  
|-
|-
| For past days
! scope="row"| For past days
| [[dayfile.txt]]
| [[dayfile.txt]]
| See [[Amending dayfile]]
| See [[Amending dayfile]]
| Web tags only exist for [[Webtags#Yesterday|yesterday]]
| Web tags only exist for [[Webtags#Yesterday|yesterday]]
| Often used as source for corrections - see  [[#Accuracy Note]]  
| Often used as source for corrections - see  [[#How editing accuracy depends on source selected|Depends on source selected Note]]  
|-
|-
|For current month-to-date
! scope="row"|For current month-to-date
|[[month.ini]]
|[[month.ini]]
| Editor for "This month's records"
| Editor for "This month's records"
Line 41: Line 41:
| Please see [[#Accuracy Note]]  
| Please see [[#Accuracy Note]]  
|-
|-
|For current year-to-date
|colspan="5" style="background:pink;"|monthyyyyMM.ini are archived copies for past months
|-
! scope="row"|For current year-to-date
|[[year.ini]]
|[[year.ini]]
| Editor for "This year's records"
| Editor for "This year's records"
|[[Webtags#Yearly|thisyear.htm]]
|[[Webtags#Yearly|thisyear.htm]]
| Please see [[#Accuracy Note]]  
| Please see [[#How editing accuracy depends on source selected|Accuracy Note]]
|-
|colspan="5" style="background:pink;"|yearyyyy.ini are archived copies for past years
|-
|-
|For all readings since a '''start date''
! scope="row"|For all readings since a '''start date''
|[[alltime.ini]]
|[[alltime.ini]]
| Editor for "All Time Records"
| Editor for "All Time Records"
Line 53: Line 57:
|See table below for start date
|See table below for start date
|-
|-
|For a particular month in all years
! scope="row"|For a particular month in all years
|[[monthlyalltime.ini]]
|[[monthlyalltime.ini]]
| Editor for "Monthly Records"
| Editor for "Monthly Records"
|[[Webtags#Monthly_All_Time_Records|monthlyrecord.htm]]
|[[Webtags#Monthly_All_Time_Records|monthlyrecord.htm]]
|
| Similar to previous row, but different start date, and separate extremes maintained for each month (regardless of the year)
|}
|}
Explaining columns in above table:
Explaining columns in above table:
# The first column is self-explanatory
# The first (label) column is self-explanatory
# The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
# The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
# The third column gives the name for the '''Edit''' menu item to choose to edit these extreme records
# The third column gives the name for the '''Edit''' menu item to choose to edit these extreme records
# The links in fourth column leads you to more information about the web tags associated with that period,  you can incorporate those in your own [[Customised_templates|templates]].
# The links in fourth column leads you to more information about the web tags associated with that period,  you can incorporate those in your own [[Customised_templates|templates]].
=General Editing Advice=
This Wiki page talks about [[#Rogue value|''rogue'' values]], meaning any incorrect entry in one or more of the [[:Category:Ini Files|extreme record files]].  You may have a feel as to which files in above table will need correction, but  generally it is best to start with all-time and finish with daily extreme records, not the reverse order.
The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values.
It is highly recommended that you always start your extreme record correction by seeing if the error is present in the [[Alltime.ini]] file that holds all-time extreme records. That approach is best partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier.  For the all-time extreme records, each individual update is logged in  [[Alltimelog.txt]] from version 1.8.9 onwards. You may get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded. If the error has affected all-time, looking at the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit.
If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the [[Monthlyalltime.ini|monthly-all-time]] file.
* From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
* If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected. 
* If you use MX, then [[Monthlyalltimelog.txt]]) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded.
Again, finding the rogue value has affected monthly-all-time gives you a date, that is a steer to whether the [[Month.ini|this month]] extremes file will need correction, and whether [[Year.ini|this year]] extreme records file will need correction.  They are relatively small files, so it should be easy to use the built-in editors to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like '''month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.
 
In working through the various files in above table, remember that if the rogue value was recorded today, then [[today.ini]] will be wrong:
* In Cumulus 1, you can edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file
* In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see [[MX on Linux]] or [[MX on Windows OS]]) and edit the file using a text editor or programmer's editor that will not add unwanted content to the file.
If the rogue value relates to yesterday, then you must edit [[Amending dayfile|dayfile.txt]], and you may choose to update [[Yesterday.ini|yesterday.ini]] although this latter file will be overwritten at next rollover.
==Correcting multiple extremes==
There are two cases to consider:
===Correcting all extremes===
Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units.
In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any [[Extra Sensor Files]], any [[Standard log files]], in [[dayfile.txt]], plus the multiple [[Category:Ini Files|extreme record files]].
The built-in editors only correct one extreme record at a time, so they cannot be used for such a task.
It is important to remember that there are [[Calculate_Missing_Values#Some_definitions|source and derived values]] in Cumulus.  If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see [[Calculate Missing Values]] page for further advice.
The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see [https://cumulus.hosiene.co.uk/viewtopic.php?p=155539#p155539 changed rainfall units example]), or to use a spreadsheet (be careful not to affects dates or times) like '''Libre Office Calc''' where you can paste special a multiplier to all cells in a particular column.
===Correcting extremes within a period===
There are a further two sub-cases that fall in this category. Both are near impossible to resolve!
Both Cumulus 1 and MX have had bugs in some releases of their software.  This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process!
[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt,  for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries.  The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent '''not-working'''. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null.  Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations.  However, there is no official solution to this problem!
=Built in extreme record editors=
[[File:Badge v1.png]] Cumulus 1 gained extreme record editing functionality from version 1.9.1 6th April 2011. See screenshot [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|Edit menu]] for how to select which file to edit, once on required editing screen, follow instructions on screen:
* Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across and click "Copy" to copy figure from identified log file to extreme record file). Press Save button to retain the change and exit.
Cumulus MX gradually gained extreme record editing functionality in releases from 3.1.1 - b3054 to 3.4.0 - b3064 (1 Nov 2019 to 19 Feb 2020), with a major redesign of user interface in release 3.18.0 (b.3189 14 June 2022):
* In latest MX release, if you click on an individual extreme record, then a pop-up appears where you can directly edit value and time-stamp to typed new content
* To load data from [[Standard log files]] you have to click on a button, from release 3.2.0 - build 3056 - only the relevant [[dayfile.txt]] entries are shown by default
* In latest MX release, if you click on a value in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy value across or not, this does not update date/time
* In latest MX release, if you click on a date/time-stamp in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy date/time across or not, this does not update value
* There is some validation on editing, you cannot empty the content of any extreme record
The built-in editors do allow you to load the standard monthly log files as well as the daily summary file (dayfile.txt). The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files. However, do exercise caution about using values from such sources, because as will be explained next, they may not hold the extreme values.


==How editing accuracy depends on source selected==
==How editing accuracy depends on source selected==


The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:
The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:
# The figure taken from a search for that extreme by examining all entries in the [[dayfile.txt]] for that period
# The (hopefully more accurate) figure taken from a search for that extreme by examining all entries in the [[dayfile.txt]] for that period
# The figure taken from a search for that extreme by examining all entries in the [[Standard log files]] for that period
# The (usually less accurate) figure taken from a search for that extreme by examining all entries in the [[Standard log files]] for that period


Normally the first returns the more accurate result (unless the '''dayfile.txt''' line, either was created with a rogue value, or has been corrupted). Let me explain why:
Obviously, it is possible that the dayfile.txt file has been corrupted, and if you are using MX, then you may want to read [[Calculate_Missing_Values#CreateMissing.exe|about a utility that can recreate dayfile.txt]].
 
But generally, dayfile.txt is the best starting point for recreating any longer period extreme records, to understand why read on:
* Using [[Standard log files]] as source for recalculating past extremes:
* Using [[Standard log files]] as source for recalculating past extremes:
** Let us assume you are using the default logging interval of 10 minutes
** Let us assume you are using the default logging interval of 10 minutes
Line 86: Line 163:
** In March 2021, a new utility '''Create Records''' was planned (for use with MX only), as at July 2021 no progress has been made in coding it.  It appears that this utility will read '''dayfile.txt''' and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other [[:Category:Ini Files|files like monthly-all-time and all-time]]. ''<big>Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available</big>''.
** In March 2021, a new utility '''Create Records''' was planned (for use with MX only), as at July 2021 no progress has been made in coding it.  It appears that this utility will read '''dayfile.txt''' and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other [[:Category:Ini Files|files like monthly-all-time and all-time]]. ''<big>Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available</big>''.


However, if you either have had an electrical supply problem or there has been corruption of data within your weather station generating a rogue value, it is possible that the files currently being used by Cumulus get corrupted.  Provided you act, as soon as possible afterwards, [[Calculate_Missing_Values#Reading_archive_data|you can make use]] of the [[Backup folder|backups]] that Cumulus takes. All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see [[today.ini]]) - either with the end of day or start of day contents.  For this month and this year extreme records, it is often better to find the previous extreme by examining a backup copy, providing a true extreme has not happened since, than by using the values displayed in the editor as those might be corrupted in the same problem.


For all-time (and for MX only, monthly-all-time) extreme records, there is a log [[Alltimelog.txt]] (and for MX only,[[Monthlyalltimelog.txt]]) that logs each time the extreme is updated, so you can get an accurate previous value from there.
==Cautionary warning and other files to examine==
 
File corruption can happen as a response to an electrical supply problem, a temporary input/output problem within the operating system, or the storage device being used, as well as after corruption of data within your weather station generating a rogue value fed through to current files.
 
Any or all of the [[:Category:Ini Files| Extreme Record .ini Files]] ''currently in use by Cumulus'' may get corrupted, as Cumulus gains exclusive write access that can overwrite the entire file during an update. 
 
Remember the values displayed in the built-in editor from dayfile.txt or monthly standard log files just might have been corrupted in the same problem. Cumulus only appends new lines to the end of these files, so it should never overwrite the whole file, but it is possible for a connection problem to make Cumulus start a new file. 
 
Therefore it is worth noting where else you can look to find values and date/time-stamps to use when correcting rogue data, and the next few subsections make some suggestions.
 
===Update tracking logs===
 
Cumulus 1 logs most extreme updates in files stored in [[Diags folder]].  Read that cross-reference for more guidance. As already mentioned, there is a log [[Alltimelog.txt]] that tracks the updates to all-time extremes.
 
Cumulus MX can log useful information in [[MXdiags folder]], depending on settings mentioned in that cross-reference, but it maintains two logs [[Alltimelog.txt]] and [[Monthlyalltimelog.txt]] as already mentioned.
 
These tracking logs can, in certain circumstances, be the best place to look up previous values as replacements for rogue values, when the built-in editors reveal that the rogue values exist in dayfile.txt and (possibly) the standard monthly log, so you can't within the editors find the correct value you seek.
 
 
===Looking at graphical representations===
 
Many people find it easy to interpolate replacements for rogue values by looking at graphical representations of their weather data covering before and after the time when the rogue figure got recorded.
 
In Cumulus 1, the obvious place to look is select-a-graphs (available from version 1.2 released on 5th April 2004).


==Arrangement of information on this page==
In Cumulus MX, later releases also have a select a chart feature, that may be more useful than the standard charts (in both interface web pages and default web pages).


The page continues by first describing corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).
Some plots record values every minute, and those high resolution plots are ideal for your search.


There is more information in [[:Category:Log_Files]] and the pages relating to individual files, for all of the extreme holding files.
===Using the Cumulus backup===


==Why might your extremes need to be corrected?==
Cumulus makes backups of the extreme record files that are kept in folders within the [[Backup folder|backup sub-folder]] in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours), so this method cannot be used for rogue values that are a week old or older.


If there is a [[Correcting_Extremes#Rogue_value|rogue value]] read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.
If you notice the rogue update when it happens, remember provided you act, as soon as possible afterwards, [[Calculate_Missing_Values#Reading_archive_data|you may be able to make use]] of the earlier version of the relevant extreme records file, as a source of correct extreme records before the corruption by a rogue figure.


It is also possible that you have discovered that you made a mistake in setting up or calibrating a sensor, and this leads you to identifying a constant/multiplier adjustment
All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see [[today.ini]]), with their contents just as they were either with the end of day or start of day.  It is therefore possible no true extreme has happened since the most recent backup, or maybe by comparing two recent back-ups you can obtain guidance on when the last true extreme occurred.  Obviously, such back-up files are no use for correcting daily extremes, but for this month, monthly-all-time, this year, and all-time, extreme records, updates to extremes don't always happen every day, especially when near end of a month. Therefore there is a good chance that you can find the previous extreme by examining a backup copy, providing a true extreme has not happened since.
to be applied to adjust all values over the past period.


A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
===Searching recent history===
 
Cumulus 1 only provides one way to access the [[Webtags#Recent_History|Recent_History]], and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags.
 
Cumulus MX stores its [[Recent history]] in a SQLite3 database that you can read.  To avoid making this page too technical, it won't tell you what software can read that database, but if you have the technical knowledge, you can look up details of suitable packages to install to read the entries in the database, and perhaps find correct extremes recorded there, as that for the last week stores measurements at (normally) one minute resolution.
 
 
==Extreme monitoring start-dates==
 
As Cumulus has developed, various derived values have been added that it calculates in addition to whatever your weather station supplies. At some releases, these extras are only available via web tags for current values, and it may be some significant time later that a release makes them available as all-time, or other period, extremes.  You may be able to track these changes by examining "changes.txt" for Cumulus 1 or "updates.txt" for MX, but those sources are not comprehensive.
 
''There is no guarantee that this Wiki content has been checked, or that it is up to date.  Any contributor is welcome to make corrections or bring it up to date''
 
For simplicity, this article will only document the development of yearly functionality and attempts to record when various extreme records became available in that context.


==All-time extreme functionality==
It should be obvious that full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, for Cumulus 1, daily extreme functionality was added first, then all-time, followed by this month/year extreme functionality, and finally monthly-all-time.  For MX, generally extras were added as current values first, and later the extremes for all the various periods were added together, but development does not always happen in a consistent way!


For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time.  Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.


[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt,  for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).  
This section has intentionally been kept brief, so it does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.


[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries.  The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.


[[Image:Icon info.png|left|30px]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
[[Image:Icon info.png|left|30px]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
Line 123: Line 231:
!style="width:100px"|First in Build
!style="width:100px"|First in Build
|-
|-
|highest/lowest apparent temperature
! scope="row"|highest/lowest apparent temperature
|26 Oct 2010
|26 Oct 2010
|1.9.1 beta
|1.9.1 beta
|957
|957
|-
|-
|highest/lowest feels like temperature
! scope="row"|highest/lowest feels like temperature
|24 June 2020
|24 June 2020
|3.6.10
|3.6.10
|3086
|3086
|-
|-
|highest Canadian Humidity Index (humidex)
! scope="row"|highest Canadian Humidity Index (humidex)
|28 Jul 2020
| 28 Jul 2020
|3.7.0
| 3.7.0
|3089
|3089
|-
|-
|highest minimum temperature
! scope="row"|highest minimum temperature
|15 April 2004
| 15 April 2004
|1.2.2
| 1.2.2
|(lost)
|(lost)
|-
|-
|highest USA heat index
! scope="row"|highest USA heat index
|29 Aug 2010
| 29 Aug 2010
|1.9.0 beta
| 1.9.0 beta
|955
|955
|-
|-
|wettest month
! scope="row"|wettest month
|5 April 2004
| 5 April 2004
|1.2
| 1.2
|(lost)
|(lost)
|-
|-
|highest daily wind run
| 24 hour rainfall
| 30 Apr 2022
| 3.16.0
| 3182
|-
! scope="row"|highest daily wind run
|3 Jul 2011
|3 Jul 2011
|1.9.2 beta
|1.9.2 beta
|1001
|1001
|-
| Sunshine hours
| 31 July 2021
| 3.12.0
| 3134
|}
|}


Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki.  Consequently,  all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page.
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki.  Consequently,  all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page.
=Correction of All Time Extreme records=
Cumulus software makes it easy to correct the all-time extremes (held in [[alltime.ini]]). 
This is because whenever Cumulus makes an update to that file, from version 1.8.9 onwards, it logs the previous and new values to the [[Alltimelog.txt]] file. Consequently, if you detect a rogue value, you can look up the latter file to determine what how to revert an entry in the former file.
==Alternative ways to find a replacement for a rogue value==
Of course, it is possible that the old value in '''Alltimelog.txt''' is not appropriate. It might be that after the rogue value was stored in '''alltime.ini''', a new extreme was seen, and this new extreme was different to the previous value stored in '''Alltimelog.txt''', but it did not cause an update in '''alltime.ini''' because of the rogue value that was stored there being more extreme.
Please see [[#Accuracy Note]] for other accuracy points.
===Looking at graphical representations===
One way to check whether the above speculation is correct is by looking at Cumulus 1 select-a-graphs (available from version 1.2 released on 5th April 2004), or MX charts in the admin interface (available in , if you can find a plot that covers the period between when the false extreme was recorded and now you are ready to do a correction, you can look for evidence of a new extreme that has been missed.  (If you only have access to your web pages, then the sample trends web page may, or may not, cover the relevant period).  Some plots record values every minute, and those high resolution plots are ideal for your search.
===Using the Cumulus backup===
If for any reason, the '''Alltimelog.txt''' cannot be used (maybe you deleted it in error), and you are within a week of when the rogue value updated the extreme, an alternative is to compare the '''alltime.ini''' with a previous version of the same file that has been stored in a [[backup_folder|''backup'' directory]]. Basicallly, Cumulus takes a backup of most of the active data files when it starts up, and also at the start of the meteorological day (just after midnight for a lot of users). The backups are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours).
If you can find the latest backup stored for a date and time from just before the ''alltime.txt'' was updated with the rogue value, look in that backup copy to see the previous value that was in the file, and use that value when you follow editing instructions below.
===Searching your standard log files===
Any search through your monthly [[Standard_log_files]] has two disadvantages:
#those lines only record a small sample of spot values as by default they are only created every 10 minutes (so even if you choose to load all log files, it might not include the actual extreme you have missed recording).
#as the files are typically large, it takes some time to load the contents of the files.
Both the original Cumulus (it is not clear from which version, it might be from 1.7.x) and MX (from version 3.2.0 build 3056) provide all time record editors.  These editors have the ability (as described below) to load up both the [[dayfile.txt|daily summary log file]] and all your [[Standard_log_files]], so you can look at the extremes that are recorded in those files, and use that information to determine what to replace your rogue value with.
The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files.
==Using the provided editors==
Both Cumulus 1 and MX provide an editor (at most versions of the relevant flavour, update to a new release if your build does not include an editor).
When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:
* [[File:Badge v1.png]] select '''All-time records''' in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
* [[File:Badge vMx.png]]select '''All Time records''' in the [[MX_Administrative_Interface#All_time_records|edit menu]] of MX admin interface.
The way the editor allows you to change a value depends on which flavour you are using:
* [[File:Badge v1.png]] Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across). Press Save button to retain the change and exit.
* [[File:Badge vMx.png]] You have to select either a value or a time-stamp that you want to change before you can type in a replacement value (you have to type the new number in even if it is displayed by loading the log files). Then click the tick to retain the change, and that will allow you to select another value or time-stamp to edit, before you finally exit the editor.
Please remember any edit you make here will not affect the related extreme in your monthly-all-time extremes recorded in [[monthlyalltime.ini]]. Nor can you update [[Standard_log_files]], or [[dayfile.txt|daily summary log]], by any edit made here. If you are using MX, it has not affected related tables on your database server.
==Actions after you have edited all-time entries==
''In all cases, an edit made to an entry in '''alltime.ini''', means that you need to make the same change in '''monthlyalltime.ini'''. Look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed. How to edit the monthly-all-time extreme records are described in next section.
If you loaded up the monthly standard log files into the all-time editor, you should be able to see:
*if there is an entry for same time-stamp as that associated with the former rogue value as seen in '''Alltimelog.txt'''
*you should also be able to work out, from '''Alltimelog.txt''', whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
If the correction you have made is in the current month, you will also need to change the entries in '''month.ini''' and '''year.ini'''. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter.
The all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.


=Correction of Monthly All Time Extreme records=
=Correction of Monthly All Time Extreme records=
Line 230: Line 287:
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]).
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]).


== How do I correct my monthly all-time records? ==


In many respects, the instruction for the all-time editing above also apply to monthly all-time.
=Correction of extremes for past days=
*[[File:Badge v1.png]] If you are a Cumulus 1 user, you do not have a [[monthlyalltimelog.txt]] file to look in to see the log entry with the previous and new values. This information was logged into a file in [[Diags_folder|the diagnostic files folder]]. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.
* [[File:Badge vMx.png]] If you are a MX user, see if you have a [[monthlyalltimelog.txt]] file in the same [[data_folder| data folder]], and then you will know what value to revert to.
Here the old and new values can be looked up in the [[monthlyalltimelog.txt]] file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in [[Diags_folder]].


 
Since '''today.ini''' is used to create lines in '''dayfile.txt''', you can follow instructions in [[Amending_dayfile]] to make any necessary corrections for past days.
There are the same alternative ways to look for the values to enter in the monthly-all-time entries that you need to change, as were described earlier to look up value for all-time. If you have already done an all-time edit, then look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed for monthly-all-time.
 
The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.
 
==Actions after you have edited monthly-all-time entries==
 
If you loaded up the monthly standard log files into the monthly-all-time editor, you should be able to see:
*if there is an entry for same time-stamp as that associated with the former rogue value as seen in '''monthlyalltimelog.txt'''
*you should also be able to work out, from '''monthlyalltimelog.txt''', whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
 
If the correction you have made is in the current month, you will also need to change the entries in '''month.ini''' and '''year.ini'''. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like '''month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.
 
The monthly-all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.


=Correction of extremes for today=
=Correction of extremes for today=


The Cumulus '''Edit' menu includes a '''Today's rain''' option where you can adjust the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus|total rainfall for today]] (e.g. if you or the wind have knocked your rain gauge) as described below. There is no facility provided to edit any other content of [[today.ini]], but since '''today.ini''' is used to create lines in '''dayfile.txt''', you can follow instructions in [[Amending_dayfile]] to make any necessary corrections for past days.
As described below, the Cumulus '''Edit' menu includes a '''Today's rain''' option where you can adjust the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus|total rainfall for today]] (e.g. if you or the wind have knocked your rain gauge), but note this does not correct "rain this hour", "rain since midnight", "maximum daily rain rate", "maximum 24 hour rain", or any extremes (for this month/year or all time).  


There is no facility provided to edit any other content of [[today.ini]].  Manual correction may be possible, but do read advice on [[today.ini]] page.


== There is an error in today's total rainfall ==
== There is an error in today's total rainfall ==
Line 268: Line 308:
*If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter
*If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter


Please note that this edit does not affect "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]], your database will retain incorrect values, as these are not updated by this correction.
Please note that this edit does not affect "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]], your database will retain incorrect values, as these are not updated by this correction.
 
Please see  [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit related fields.
 
=Correction of extremes for month-to-date=
 
As mentioned in passing above, a rogue value may get recorded in a [[month.ini]] file.
 
You should ensure you have got the daily summary log ([[Amending_dayfile|dayfile.txt]]) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this month. 
 
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit '''This year's records'''. The editors work in exactly the same way as was described for all-time above.
*[[File:Badge v1.png]]In Cumulus 1, '''Copy''' buttons enable you to copy a record from '''dayfile'''. Click '''OK'' to save.
* [[File:Badge vMx.png]] In MX, the [[dayfile.txt|dayfile]] value/timestamp, and the [[Standard_log_files|Logfile]] value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click '''Tick'' to save.


Please see  [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit those related fields.


=Correction of extremes for past month=
=Correction of extremes for past month=
Line 290: Line 319:
MX does not provide any functionality to read this old file, let alone edit it.  However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.
MX does not provide any functionality to read this old file, let alone edit it.  However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.


=Correction of extremes for  year-to-date=
As mentioned in passing above, a rogue value may get recorded in a [[year.ini]] file.
You should ensure you have got the daily summary log ([[Amending_dayfile|dayfile.txt]]) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this year. 
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit '''This year's records'''. The editors work in exactly the same way as was described for all-time above.
*[[File:Badge v1.png]]In Cumulus 1, '''Copy''' buttons enable you to copy a record from '''dayfile'''. Click '''OK'' to save.
* [[File:Badge vMx.png]] In MX, the [[dayfile.txt|dayfile]] value/timestamp, and the [[Standard_log_files|Logfile]] value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click '''Tick'' to save.


=Correction of extremes for past year=
=Correction of extremes for past year=
Line 328: Line 348:
'''Version''' here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.
'''Version''' here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.


'''Build''' number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number. For recent MX releases, the developer has changed version number every time there is a new build released.
'''Build''' number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number.
 
Steve Loft generally went through a lot of beta releases identified by build number before finally having a stable release with new version number. Most beta releases were available to everyone.
 
Mark Crossley has sometimes issued his beta commits identified by beta, beta 1, beta 2, etc. without changing build number, and sometimes incremented build number several times between changes to release version number. Most beta releases are made available to just a select few testers, via a sub-forum with limited access, or via direct email of zip files to particular people. The beta history is not documented in "updates.txt", which only quotes the more significant changes.

Revision as of 13:37, 5 August 2022

This page brings together text that was originally on other pages. It is designed to cover both Cumulus Version 1 Specific and Cumulus Version MX Specific, but do be aware that some terminology varies between the two flavours.



Introduction

As Cumulus processes each reading from your weather station, it checks that value (and any derived from it) against the extremes currently stored in various .ini files, and if necessary updates the extreme records that are affected.

All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered amongst pages covering the various files).

The extreme records that are maintained in this way are:

Period File storing extremes How to correct Link to web tag section Notes
For current day so far today.ini Editor for "Today's rain" (no editor for other derivatives) today.htm Many entries in this file (for non-midnight rollover, use is made of yesterday.ini too) get transferred to dayfile.txt at end of day.
For past days dayfile.txt See Amending dayfile Web tags only exist for yesterday Often used as source for corrections - see Depends on source selected Note
For current month-to-date month.ini Editor for "This month's records" thismonth.htm Please see #Accuracy Note
monthyyyyMM.ini are archived copies for past months
For current year-to-date year.ini Editor for "This year's records" thisyear.htm Please see Accuracy Note
yearyyyy.ini are archived copies for past years
For all readings since a 'start date alltime.ini Editor for "All Time Records" records.htm See table below for start date
For a particular month in all years monthlyalltime.ini Editor for "Monthly Records" monthlyrecord.htm Similar to previous row, but different start date, and separate extremes maintained for each month (regardless of the year)

Explaining columns in above table:

  1. The first (label) column is self-explanatory
  2. The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
  3. The third column gives the name for the Edit menu item to choose to edit these extreme records
  4. The links in fourth column leads you to more information about the web tags associated with that period, you can incorporate those in your own templates.


General Editing Advice

This Wiki page talks about rogue values, meaning any incorrect entry in one or more of the extreme record files. You may have a feel as to which files in above table will need correction, but generally it is best to start with all-time and finish with daily extreme records, not the reverse order.

The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values.

It is highly recommended that you always start your extreme record correction by seeing if the error is present in the Alltime.ini file that holds all-time extreme records. That approach is best partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier. For the all-time extreme records, each individual update is logged in Alltimelog.txt from version 1.8.9 onwards. You may get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded. If the error has affected all-time, looking at the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit.

If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the monthly-all-time file.

  • From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
  • If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected.
  • If you use MX, then Monthlyalltimelog.txt) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded.

Again, finding the rogue value has affected monthly-all-time gives you a date, that is a steer to whether the this month extremes file will need correction, and whether this year extreme records file will need correction. They are relatively small files, so it should be easy to use the built-in editors to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same data folder, with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.


In working through the various files in above table, remember that if the rogue value was recorded today, then today.ini will be wrong:

  • In Cumulus 1, you can edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file
  • In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see MX on Linux or MX on Windows OS) and edit the file using a text editor or programmer's editor that will not add unwanted content to the file.

If the rogue value relates to yesterday, then you must edit dayfile.txt, and you may choose to update yesterday.ini although this latter file will be overwritten at next rollover.

Correcting multiple extremes

There are two cases to consider:

Correcting all extremes

Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units.

In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any Extra Sensor Files, any Standard log files, in dayfile.txt, plus the multiple.

The built-in editors only correct one extreme record at a time, so they cannot be used for such a task.

It is important to remember that there are source and derived values in Cumulus. If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see Calculate Missing Values page for further advice.

The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see changed rainfall units example), or to use a spreadsheet (be careful not to affects dates or times) like Libre Office Calc where you can paste special a multiplier to all cells in a particular column.

Correcting extremes within a period

There are a further two sub-cases that fall in this category. Both are near impossible to resolve!

Both Cumulus 1 and MX have had bugs in some releases of their software. This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process!

Badge v1.pngThere were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in File:Changes.zip, although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within Cumulus forum announcements, but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in month.ini and year.ini depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt, for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in dayfile.txt for the whole of that period).

Badge vMx.pngCumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The updates.txt that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in the relevant support forum will yield more information in return for a lot more effort.

A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.

In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent not-working. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null. Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations. However, there is no official solution to this problem!

Built in extreme record editors

Badge v1.png Cumulus 1 gained extreme record editing functionality from version 1.9.1 6th April 2011. See screenshot Edit menu for how to select which file to edit, once on required editing screen, follow instructions on screen:

  • Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across and click "Copy" to copy figure from identified log file to extreme record file). Press Save button to retain the change and exit.

Cumulus MX gradually gained extreme record editing functionality in releases from 3.1.1 - b3054 to 3.4.0 - b3064 (1 Nov 2019 to 19 Feb 2020), with a major redesign of user interface in release 3.18.0 (b.3189 14 June 2022):

  • In latest MX release, if you click on an individual extreme record, then a pop-up appears where you can directly edit value and time-stamp to typed new content
  • To load data from Standard log files you have to click on a button, from release 3.2.0 - build 3056 - only the relevant dayfile.txt entries are shown by default
  • In latest MX release, if you click on a value in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy value across or not, this does not update date/time
  • In latest MX release, if you click on a date/time-stamp in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy date/time across or not, this does not update value
  • There is some validation on editing, you cannot empty the content of any extreme record


The built-in editors do allow you to load the standard monthly log files as well as the daily summary file (dayfile.txt). The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files. However, do exercise caution about using values from such sources, because as will be explained next, they may not hold the extreme values.

How editing accuracy depends on source selected

The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:

  1. The (hopefully more accurate) figure taken from a search for that extreme by examining all entries in the dayfile.txt for that period
  2. The (usually less accurate) figure taken from a search for that extreme by examining all entries in the Standard log files for that period

Obviously, it is possible that the dayfile.txt file has been corrupted, and if you are using MX, then you may want to read about a utility that can recreate dayfile.txt.

But generally, dayfile.txt is the best starting point for recreating any longer period extreme records, to understand why read on:

  • Using Standard log files as source for recalculating past extremes:
    • Let us assume you are using the default logging interval of 10 minutes
    • Unlike some other weather station software available (which logs highest and lowest since previous log), Cumulus logs spot values
    • That means the Monthly log files do not capture any extremes recorded in the time (by default 599 seconds) between logs
    • Therefore the detailed log files are not normally the most accurate source
    • Please note, this less accurate way of deducing daily extremes/totals (to update dayfile.txt) is used by Cumulus software:
  • Using dayfile.txt as source for recalculating past extremes
    • MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
    • Therefore extremes recorded in today.ini' (and from there into dayfile.ini) are based on the full sampling done by Cumulus
    • This means none, or very few, extremes are missed
    • In March 2021, a new utility Create Records was planned (for use with MX only), as at July 2021 no progress has been made in coding it. It appears that this utility will read dayfile.txt and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other files like monthly-all-time and all-time. Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available.


Cautionary warning and other files to examine

File corruption can happen as a response to an electrical supply problem, a temporary input/output problem within the operating system, or the storage device being used, as well as after corruption of data within your weather station generating a rogue value fed through to current files.

Any or all of the Extreme Record .ini Files currently in use by Cumulus may get corrupted, as Cumulus gains exclusive write access that can overwrite the entire file during an update.

Remember the values displayed in the built-in editor from dayfile.txt or monthly standard log files just might have been corrupted in the same problem. Cumulus only appends new lines to the end of these files, so it should never overwrite the whole file, but it is possible for a connection problem to make Cumulus start a new file.

Therefore it is worth noting where else you can look to find values and date/time-stamps to use when correcting rogue data, and the next few subsections make some suggestions.

Update tracking logs

Cumulus 1 logs most extreme updates in files stored in Diags folder. Read that cross-reference for more guidance. As already mentioned, there is a log Alltimelog.txt that tracks the updates to all-time extremes.

Cumulus MX can log useful information in MXdiags folder, depending on settings mentioned in that cross-reference, but it maintains two logs Alltimelog.txt and Monthlyalltimelog.txt as already mentioned.

These tracking logs can, in certain circumstances, be the best place to look up previous values as replacements for rogue values, when the built-in editors reveal that the rogue values exist in dayfile.txt and (possibly) the standard monthly log, so you can't within the editors find the correct value you seek.


Looking at graphical representations

Many people find it easy to interpolate replacements for rogue values by looking at graphical representations of their weather data covering before and after the time when the rogue figure got recorded.

In Cumulus 1, the obvious place to look is select-a-graphs (available from version 1.2 released on 5th April 2004).

In Cumulus MX, later releases also have a select a chart feature, that may be more useful than the standard charts (in both interface web pages and default web pages).

Some plots record values every minute, and those high resolution plots are ideal for your search.

Using the Cumulus backup

Cumulus makes backups of the extreme record files that are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours), so this method cannot be used for rogue values that are a week old or older.

If you notice the rogue update when it happens, remember provided you act, as soon as possible afterwards, you may be able to make use of the earlier version of the relevant extreme records file, as a source of correct extreme records before the corruption by a rogue figure.

All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see today.ini), with their contents just as they were either with the end of day or start of day. It is therefore possible no true extreme has happened since the most recent backup, or maybe by comparing two recent back-ups you can obtain guidance on when the last true extreme occurred. Obviously, such back-up files are no use for correcting daily extremes, but for this month, monthly-all-time, this year, and all-time, extreme records, updates to extremes don't always happen every day, especially when near end of a month. Therefore there is a good chance that you can find the previous extreme by examining a backup copy, providing a true extreme has not happened since.

Searching recent history

Cumulus 1 only provides one way to access the Recent_History, and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags.

Cumulus MX stores its Recent history in a SQLite3 database that you can read. To avoid making this page too technical, it won't tell you what software can read that database, but if you have the technical knowledge, you can look up details of suitable packages to install to read the entries in the database, and perhaps find correct extremes recorded there, as that for the last week stores measurements at (normally) one minute resolution.


Extreme monitoring start-dates

As Cumulus has developed, various derived values have been added that it calculates in addition to whatever your weather station supplies. At some releases, these extras are only available via web tags for current values, and it may be some significant time later that a release makes them available as all-time, or other period, extremes. You may be able to track these changes by examining "changes.txt" for Cumulus 1 or "updates.txt" for MX, but those sources are not comprehensive.

There is no guarantee that this Wiki content has been checked, or that it is up to date. Any contributor is welcome to make corrections or bring it up to date

For simplicity, this article will only document the development of yearly functionality and attempts to record when various extreme records became available in that context.

It should be obvious that full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, for Cumulus 1, daily extreme functionality was added first, then all-time, followed by this month/year extreme functionality, and finally monthly-all-time. For MX, generally extras were added as current values first, and later the extremes for all the various periods were added together, but development does not always happen in a consistent way!


This section has intentionally been kept brief, so it does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.


Icon info.png

The start date referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:

Parameter First released First in Version First in Build
highest/lowest apparent temperature 26 Oct 2010 1.9.1 beta 957
highest/lowest feels like temperature 24 June 2020 3.6.10 3086
highest Canadian Humidity Index (humidex) 28 Jul 2020 3.7.0 3089
highest minimum temperature 15 April 2004 1.2.2 (lost)
highest USA heat index 29 Aug 2010 1.9.0 beta 955
wettest month 5 April 2004 1.2 (lost)
24 hour rainfall 30 Apr 2022 3.16.0 3182
highest daily wind run 3 Jul 2011 1.9.2 beta 1001
Sunshine hours 31 July 2021 3.12.0 3134

Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked (lost), and there is some vagueness in information mentioned elsewhere in this page.

Correction of Monthly All Time Extreme records

From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.

Initialisation of monthly-all-time extreme records

Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one Copy button, the one in the header row, all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking OK button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see Calculate Missing Values).


Correction of extremes for past days

Since today.ini is used to create lines in dayfile.txt, you can follow instructions in Amending_dayfile to make any necessary corrections for past days.

Correction of extremes for today

As described below, the Cumulus Edit' menu includes a Today's rain option where you can adjust the total rainfall for today (e.g. if you or the wind have knocked your rain gauge), but note this does not correct "rain this hour", "rain since midnight", "maximum daily rain rate", "maximum 24 hour rain", or any extremes (for this month/year or all time).

There is no facility provided to edit any other content of today.ini. Manual correction may be possible, but do read advice on today.ini page.

There is an error in today's total rainfall

Easy - correct today's total using the 'today's rain' editor on the edit menu.

  • Badge v1.pngselect 'Today's rain in the edit menu accessed from main screen in Cumulus 1,
  • Badge vMx.pngselect Today's rain in the edit menu of MX admin interface.

This edit will actually alter the start of day rainfall counter figure:

  • If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter
  • If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter

Please note that this edit does not affect "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the Standard_log_files, your database will retain incorrect values, as these are not updated by this correction.

Please see Today.ini#Editing_rainfall_in_today.ini_within_Cumulus for details of how to edit those related fields.

Correction of extremes for past month

  • Badge v1.pngCumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
  • Badge vMx.png From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the month.ini at the start of a new year, it saves the old month.ini (whenever it was last updated) as a file with a name like month201501.ini.

MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.


Correction of extremes for past year

  • Badge v1.pngCumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
  • Badge vMx.png From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the year.ini at the start of a new year, it saves the old year.ini (whenever it was last updated) as a file with a name like year2015.ini.

MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.


Some definitions

Rogue value

In this article, the term rogue value is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.

Here are a typical examples:

  • it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
  • perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
  • an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong

Flavour, Release, Version, and Build

Flavour is used to represent the original Cumulus, Cumulus 2, and Cumulus MX, collectively. Where the text says applicability is dependant on flavour, it means that the action you do depends on whether you are installing/running Cumulus MX or the original Cumulus software.

Release is used to signify what the Cumulus developer makes available for download after there has been a modification to the software. For most Cumulus 1 builds, the release consisted of a executable that would create all the folders and files needed to run that software. For Cumulus 2, releases were a zip file and were numbered using a identifier with many digits. MX releases are as a zip file that is labelled with the build number.

Version here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.

Build number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number.

Steve Loft generally went through a lot of beta releases identified by build number before finally having a stable release with new version number. Most beta releases were available to everyone.

Mark Crossley has sometimes issued his beta commits identified by beta, beta 1, beta 2, etc. without changing build number, and sometimes incremented build number several times between changes to release version number. Most beta releases are made available to just a select few testers, via a sub-forum with limited access, or via direct email of zip files to particular people. The beta history is not documented in "updates.txt", which only quotes the more significant changes.