Difference between revisions of "Correcting Extremes"

From Cumulus Wiki
m (All-time extreme functionality)
m (Correcting an error in today's total rainfall: extend list of where might send rainfall data)
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page brings together text that was originally on other pages. It is designed to cover both
+
[[Category:Cumulus Files]][[Category:Ini Files]]
{{Version badge 1}} and {{Template:Version badge Mx}}, but do be aware that some terminology varies between the two flavours.
+
=Terminology=
  
[[Category:Cumulus Files]][[Category:Ini Files]]
+
For simplicity, the terminology "extremes" is used on this page, the meaning includes:
 +
# The ''totals'' maintained (such as rainfall, chill hours, and the various "degree days"), and
 +
#  The high/low "extreme records" for various periods (such as all-time, and this month).
 +
 
 +
 
 +
This Wiki page has been created to cover both {{Version badge 1}} and {{Template:Version badge Mx}}, but do be aware that some terminology varies between the two flavours. All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered between Wiki pages for each specific file, this redesign brings together all such text here).
  
 +
==Rogue value==
  
 +
Throughout this Wiki the term '''rogue value''' is used to mean you see a value somewhere in Cumulus that you believe should not be there.
  
 +
Generally, "rogue" usage refers to a single data point. However, where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files. Specifically, on this Wiki page, the meaning covers any incorrect entry in one or more of the [[:Category:Ini Files|extreme record files]].
  
=Introduction=
+
Here are some typical examples:
 +
* it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
 +
* perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
 +
* an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong
  
As Cumulus processes each reading from your weather station, it checks that value (and any [[Calculate_Missing_Values#Derived spot values|derived]] from it) against the extremes currently stored in various [[:Category:Log_Files|.ini files]], and if necessary updates the extreme records that are affected.
+
=How Cumulus software tracks "extremes"=
  
All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered amongst pages covering the various files).
+
As Cumulus weather software processes each reading from your weather station, it checks that value (and any [[Calculate_Missing_Values#Derived spot values|derived]] from it) against the extremes currently stored in various RAM variables, and if necessary updates the extreme records that are affected. Please note these extreme records are held by Cumulus software in internal variables. Periodically, these totals and high/low "extreme records" are written out to [[:Category:Log_Files|.ini files]] (ensuring such "extremes" are kept when MX is stopped and restarted).
  
The extreme records that are maintained in this way are:
+
The totals/extreme records that are maintained in this way are:
  
 
{| class="wikitable" border="1"
 
{| class="wikitable" border="1"
Line 23: Line 34:
 
!style="width:300px"|Notes
 
!style="width:300px"|Notes
 
|-
 
|-
|For current day so far
+
! scope="row"|For current day so far
 
|[[today.ini]]
 
|[[today.ini]]
 
| Editor for "Today's rain" (no editor for other derivatives)
 
| Editor for "Today's rain" (no editor for other derivatives)
Line 29: Line 40:
 
|Many entries in this file (for non-midnight rollover, use is made of [[yesterday.ini]] too) get transferred to [[dayfile.txt]] at end of day.  
 
|Many entries in this file (for non-midnight rollover, use is made of [[yesterday.ini]] too) get transferred to [[dayfile.txt]] at end of day.  
 
|-
 
|-
| For past days
+
! scope="row"| For past days
 
| [[dayfile.txt]]
 
| [[dayfile.txt]]
 
| See [[Amending dayfile]]
 
| See [[Amending dayfile]]
 
| Web tags only exist for [[Webtags#Yesterday|yesterday]]
 
| Web tags only exist for [[Webtags#Yesterday|yesterday]]
| Often used as source for corrections - see  [[#Accuracy Note]]  
+
| Often used as source for corrections - see  [[#How editing accuracy depends on source selected|Depends on source selected Note]]  
 
|-
 
|-
|For current month-to-date
+
! scope="row"|For current month-to-date
 
|[[month.ini]]
 
|[[month.ini]]
 
| Editor for "This month's records"
 
| Editor for "This month's records"
Line 41: Line 52:
 
| Please see [[#Accuracy Note]]  
 
| Please see [[#Accuracy Note]]  
 
|-
 
|-
|For current year-to-date
+
|colspan="5" style="background:pink;"|monthyyyyMM.ini are archived copies for past months
 +
|-
 +
! scope="row"|For current year-to-date
 
|[[year.ini]]
 
|[[year.ini]]
 
| Editor for "This year's records"
 
| Editor for "This year's records"
 
|[[Webtags#Yearly|thisyear.htm]]
 
|[[Webtags#Yearly|thisyear.htm]]
| Please see [[#Accuracy Note]]  
+
| Please see [[#How editing accuracy depends on source selected|Accuracy Note]]
 +
|-
 +
|colspan="5" style="background:pink;"|yearyyyy.ini are archived copies for past years
 
|-
 
|-
|For all readings since a '''start date''
+
! scope="row"|For all readings since a '''start date''
 
|[[alltime.ini]]
 
|[[alltime.ini]]
 
| Editor for "All Time Records"
 
| Editor for "All Time Records"
Line 53: Line 68:
 
|See table below for start date
 
|See table below for start date
 
|-
 
|-
|For a particular month in all years
+
! scope="row"|For a particular month in all years
 
|[[monthlyalltime.ini]]
 
|[[monthlyalltime.ini]]
 
| Editor for "Monthly Records"
 
| Editor for "Monthly Records"
 
|[[Webtags#Monthly_All_Time_Records|monthlyrecord.htm]]
 
|[[Webtags#Monthly_All_Time_Records|monthlyrecord.htm]]
|
+
| Similar to previous row, but different start date, and separate extremes maintained for each month (regardless of the year)
 
|}
 
|}
 
Explaining columns in above table:
 
Explaining columns in above table:
# The first column is self-explanatory
+
# The first (label) column is self-explanatory
 
# The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
 
# The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
 
# The third column gives the name for the '''Edit''' menu item to choose to edit these extreme records
 
# The third column gives the name for the '''Edit''' menu item to choose to edit these extreme records
 
# The links in fourth column leads you to more information about the web tags associated with that period,  you can incorporate those in your own [[Customised_templates|templates]].
 
# The links in fourth column leads you to more information about the web tags associated with that period,  you can incorporate those in your own [[Customised_templates|templates]].
  
==Accuracy Note==
 
  
The editors built into Cumulus, for extremes over a period of a month or more, give you the ability to display, for each extreme record:
 
# The figure taken from a search for that extreme by examining entries in the [[dayfile.txt]] for that period
 
# The figure taken from a search for that extreme by examining entries in the [[Standard log files]] for that period
 
  
Normally the first returns the more accurate result (unless the '''dayfile.txt''' line, either was created with a rogue value, or has been corrupted). Let me explain why:
+
=Built in extreme record editors=
 +
 
 +
[[File:Badge v1.png]] Cumulus 1 gained extreme record editing functionality from version 1.9.1 6th April 2011. See screenshot [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|Edit menu]] for how to select which file to edit, once on required editing screen, follow instructions on screen:
 +
* Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across and click "Copy" to copy figure from identified log file to extreme record file). Press Save button to retain the change and exit.
 +
 
 +
Cumulus MX gradually gained extreme record editing functionality in releases from 3.1.1 - b3054 to 3.4.0 - b3064 (1 Nov 2019 to 19 Feb 2020), with a major redesign of user interface in release 3.18.0 (b.3189 14 June 2022):
 +
* In latest MX release, if you click on an individual extreme record, then a pop-up appears where you can directly edit value and time-stamp to typed new content
 +
* To load data from [[Standard log files]] you have to click on a button, from release 3.2.0 - build 3056 - only the relevant [[dayfile.txt]] entries are shown by default
 +
* In latest MX release, if you click on a value in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy value across or not, this does not update date/time
 +
* In latest MX release, if you click on a date/time-stamp in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy date/time across or not, this does not update value
 +
* There is some validation on editing, you cannot empty the content of any extreme record
 +
 
 +
 
 +
The built-in editors do allow you to load the standard monthly log files as well as the daily summary file (dayfile.txt). The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files. However, do exercise caution about using values from such sources, because as will be explained next, they may not hold the extreme values.
 +
 
 +
==How editing accuracy depends on source selected==
 +
 
 +
The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:
 +
# The (hopefully more accurate) figure taken from a search for that extreme by examining all entries in the [[dayfile.txt]] for that period
 +
# The (usually less accurate) figure taken from a search for that extreme by examining all entries in the [[Standard log files]] for that period
 +
 
 +
Obviously, it is possible that the dayfile.txt file has been corrupted, and if you are using MX, then you may want to read [[Calculate_Missing_Values#CreateMissing.exe|about a utility that can recreate dayfile.txt]].
 +
 
 +
But generally, dayfile.txt is the best starting point for recreating any longer period extreme records, to understand why read on:
 
* Using [[Standard log files]] as source for recalculating past extremes:
 
* Using [[Standard log files]] as source for recalculating past extremes:
 
** Let us assume you are using the default logging interval of 10 minutes
 
** Let us assume you are using the default logging interval of 10 minutes
Line 77: Line 111:
 
** That means the [[Monthly log files]] do not capture any extremes recorded in the time (by default 599 seconds) between logs
 
** That means the [[Monthly log files]] do not capture any extremes recorded in the time (by default 599 seconds) between logs
 
** Therefore the detailed log files are not normally the most accurate source
 
** Therefore the detailed log files are not normally the most accurate source
 +
** Please note, this less accurate way of deducing daily extremes/totals (to update dayfile.txt) is used by Cumulus software:
 +
*** For Legacy Cumulus 1: [[Amending_dayfile#Create_Missing_on_legacy_dayfile_editor|Create Missing in legacy dayfile editor]].
 +
*** For MX: [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] utility.
 
* Using [[dayfile.txt]] as source for recalculating past extremes
 
* Using [[dayfile.txt]] as source for recalculating past extremes
 
** MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
 
** MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
 
** Therefore extremes recorded in '''today.ini''' (and from there into '''dayfile.ini'') are based on the full sampling done by Cumulus
 
** Therefore extremes recorded in '''today.ini''' (and from there into '''dayfile.ini'') are based on the full sampling done by Cumulus
 
** This means none, or very few, extremes are missed
 
** This means none, or very few, extremes are missed
 +
** In March 2021, a new utility '''Create Records''' was planned (for use with MX only), as at July 2021 no progress has been made in coding it.  It appears that this utility will read '''dayfile.txt''' and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other [[:Category:Ini Files|files like monthly-all-time and all-time]]. ''<big>Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available</big>''.
 +
 +
 +
==Cautionary warning and other files to examine==
 +
 +
File corruption can happen as a response to an electrical supply problem, a temporary input/output problem within the operating system, or the storage device being used, as well as after corruption of data within your weather station generating a rogue value fed through to current files.
 +
 +
Any or all of the [[:Category:Ini Files| Extreme Record .ini Files]] ''currently in use by Cumulus'' may get corrupted, as Cumulus gains exclusive write access that can overwrite the entire file during an update. 
 +
 +
Remember the values displayed in the built-in editor from dayfile.txt or monthly standard log files just might have been corrupted in the same problem. Cumulus only appends new lines to the end of these files, so it should never overwrite the whole file, but it is possible for a connection problem to make Cumulus start a new file. 
 +
 +
Therefore it is worth noting where else you can look to find values and date/time-stamps to use when correcting rogue data, and the next few subsections make some suggestions.
 +
 +
===Update tracking logs===
 +
 +
Cumulus 1 logs most extreme updates in files stored in [[Diags folder]].  Read that cross-reference for more guidance. As already mentioned, there is a log [[Alltimelog.txt]] that tracks the updates to all-time extremes.
 +
 +
Cumulus MX can log useful information in [[MXdiags folder]], depending on settings mentioned in that cross-reference, but it maintains two logs [[Alltimelog.txt]] and [[Monthlyalltimelog.txt]] as already mentioned.
 +
 +
These tracking logs can, in certain circumstances, be the best place to look up previous values as replacements for rogue values, when the built-in editors reveal that the rogue values exist in dayfile.txt and (possibly) the standard monthly log, so you can't within the editors find the correct value you seek.
 +
 +
 +
===Looking at graphical representations===
 +
 +
Many people find it easy to interpolate replacements for rogue values by looking at graphical representations of their weather data covering before and after the time when the rogue figure got recorded.
 +
 +
In Cumulus 1, the obvious place to look is select-a-graphs (available from version 1.2 released on 5th April 2004).
 +
 +
In Cumulus MX, later releases also have a select a chart feature, that may be more useful than the standard charts (in both interface web pages and default web pages).
 +
 +
Some plots record values every minute, and those high resolution plots are ideal for your search.
 +
 +
===Using the Cumulus backup===
 +
 +
Cumulus makes backups of the extreme record files that are kept in folders within the [[Backup folder|backup sub-folder]] in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours), so this method cannot be used for rogue values that are a week old or older.
 +
 +
If you notice the rogue update when it happens, remember provided you act, as soon as possible afterwards, [[Calculate_Missing_Values#Reading_archive_data|you may be able to make use]] of the earlier version of the relevant extreme records file, as a source of correct extreme records before the corruption by a rogue figure.
 +
 +
All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see [[today.ini]]), with their contents just as they were either with the end of day or start of day.  It is therefore possible no true extreme has happened since the most recent backup, or maybe by comparing two recent back-ups you can obtain guidance on when the last true extreme occurred.  Obviously, such back-up files are no use for correcting daily extremes, but for this month, monthly-all-time, this year, and all-time, extreme records, updates to extremes don't always happen every day, especially when near end of a month. Therefore there is a good chance that you can find the previous extreme by examining a backup copy, providing a true extreme has not happened since.
 +
 +
===Searching recent history===
 +
 +
Cumulus 1 only provides one way to access the [[Webtags#Recent_History|Recent_History]], and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags.
 +
 +
Cumulus MX stores its [[Recent history]] in a SQLite3 database that you can read/edit as explained at [[Cumulusmx.db#Reading.2Fediting_database_table_outside_MX]].
  
However, if you either have had an electrical supply problem or there has been corruption of data within your weather station generating a rogue value, it is possible that the files currently being used by Cumulus get corrupted.  Provided you act, as soon as possible afterwards, you can make use of the [[Backup folder|backups]] that Cumulus takes. All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see [[today.ini]] - either  with the end of day or start of day contents.  For this month and this year extreme records, it is often better to find the previous extreme by examining a backup copy, providing a true extreme has not happened since, than by using the values displayed in the editor as those might be corrupted in the same problem.
 
  
For all-time (and for MX only, monthly-all-time) extreme records, there is a log [[Alltimelog.txt]] (and for MX only,[[Monthlyalltimelog.txt]]) that logs each time the extreme is updated, so you can get an accurate previous value from there.
 
  
==Arrangement of information on this page==
+
=General Editing Advice=
  
The page continues by first describing corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).
+
The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values.
  
There is more information in [[:Category:Log_Files]] and the pages relating to individual files, for all of the extreme holding files.
+
You may have a feel as to which files in above table will need correction, but if in doubt it is highly recommended that you always start your extreme record correction by seeing if the error is present in the [[Alltime.ini]] file that holds all-time extreme records. That approach is best, partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier.
  
==Why might your extremes need to be corrected?==
 
  
If there is a [[Correcting_Extremes#Rogue_value|rogue value]] read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.
+
==Extreme monitoring start-dates==
  
It is also possible that you have discovered that you made a mistake in setting up or calibrating a sensor, and this leads you to identifying a constant/multiplier adjustment
+
As Cumulus has developed, various derived values have been added that it calculates in addition to whatever your weather station supplies. At some releases, these extras are only available via web tags for current values, and it may be some significant time later that a release makes them available as all-time, or other period, extremes.  You may be able to track these changes by examining "changes.txt" for Cumulus 1 or "updates.txt" for MX, but those sources are not comprehensive.
to be applied to adjust all values over the past period.
 
  
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
+
''There is no guarantee that this Wiki content has been checked, or that it is up to date.  Any contributor is welcome to make corrections or bring it up to date''
 +
 
 +
For simplicity, this article will only document the development of yearly functionality and attempts to record when various extreme records became available in that context.
  
==All-time extreme functionality==
+
It should be obvious that full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, for Cumulus 1, daily extreme functionality was added first, then all-time, followed by this month/year extreme functionality, and finally monthly-all-time.  For MX, generally extras were added as current values first, and later the extremes for all the various periods were added together, but development does not always happen in a consistent way!
  
For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time.  Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.
 
  
[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt,  for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).  
+
This section has intentionally been kept brief, so it does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.
  
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
 
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries.  The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.
 
  
 
[[Image:Icon info.png|left|30px]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
 
[[Image:Icon info.png|left|30px]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
Line 119: Line 196:
 
!style="width:100px"|First in Build
 
!style="width:100px"|First in Build
 
|-
 
|-
|highest/lowest apparent temperature
+
! scope="row"|highest/lowest apparent temperature
 
|26 Oct 2010
 
|26 Oct 2010
 
|1.9.1 beta
 
|1.9.1 beta
 
|957
 
|957
 
|-
 
|-
|highest/lowest feels like temperature
+
! scope="row"|highest/lowest feels like temperature
 
|24 June 2020
 
|24 June 2020
 
|3.6.10
 
|3.6.10
 
|3086
 
|3086
 
|-
 
|-
|highest Canadian Humidity Index (humidex)
+
! scope="row"|highest Canadian Humidity Index (humidex)
|28 Jul 2020
+
| 28 Jul 2020
|3.7.0
+
| 3.7.0
 
|3089
 
|3089
 
|-
 
|-
|highest minimum temperature
+
! scope="row"|highest minimum temperature
|15 April 2004
+
| 15 April 2004
|1.2.2
+
| 1.2.2
 
|(lost)
 
|(lost)
 
|-
 
|-
|highest USA heat index
+
! scope="row"|highest USA heat index
|29 Aug 2010
+
| 29 Aug 2010
|1.9.0 beta
+
| 1.9.0 beta
 
|955
 
|955
 
|-
 
|-
|wettest month
+
! scope="row"|wettest month
|5 April 2004
+
| 5 April 2004
|1.2
+
| 1.2
 
|(lost)
 
|(lost)
 
|-
 
|-
|highest daily wind run
+
| 24 hour rainfall
 +
| 30 Apr 2022
 +
| 3.16.0
 +
| 3182
 +
|-
 +
! scope="row"|highest daily wind run
 
|3 Jul 2011
 
|3 Jul 2011
 
|1.9.2 beta
 
|1.9.2 beta
 
|1001
 
|1001
 +
|-
 +
| Sunshine hours
 +
| 31 July 2021
 +
| 3.12.0
 +
| 3134
 
|}
 
|}
  
 
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki.  Consequently,  all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page.
 
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki.  Consequently,  all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page.
  
=Correction of All Time Extreme records=
 
  
Cumulus software makes it easy to correct the all-time extremes (held in [[alltime.ini]]). 
+
==Correcting multiple extremes==
  
This is because whenever Cumulus makes an update to that file, from version 1.8.9 onwards, it logs the previous and new values to the [[Alltimelog.txt]] file. Consequently, if you detect a rogue value, you can look up the latter file to determine what how to revert an entry in the former file.
+
There are two cases to consider:
  
 +
===Correcting extremes recorded for every logging entry (plus every day/month/year of long period)===
  
==Alternative ways to find a replacement for a rogue value==
+
Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units.
  
Of course, it is possible that the old value in '''Alltimelog.txt''' is not appropriate. It might be that after the rogue value was stored in '''alltime.ini''', a new extreme was seen, and this new extreme was different to the previous value stored in '''Alltimelog.txt''', but it did not cause an update in '''alltime.ini''' because of the rogue value that was stored there being more extreme.
+
In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any [[Extra Sensor Files]], any [[Standard log files]], in [[dayfile.txt]], plus the multiple [[Category:Ini Files|extreme record files]].
  
Please see [[#Accuracy Note]] for other accuracy points.
+
The built-in editors only correct one extreme record at a time, so they cannot be used for such a task.  
  
===Looking at graphical representations===
+
It is important to remember that there are [[Calculate_Missing_Values#Some_definitions|source and derived values]] in Cumulus.  If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see [[Calculate Missing Values]] page for further advice.
  
One way to check whether the above speculation is correct is by looking at Cumulus 1 select-a-graphs (available from version 1.2 released on 5th April 2004), or MX charts in the admin interface (available in , if you can find a plot that covers the period between when the false extreme was recorded and now you are ready to do a correction, you can look for evidence of a new extreme that has been missed.  (If you only have access to your web pages, then the sample trends web page may, or may not, cover the relevant period).  Some plots record values every minute, and those high resolution plots are ideal for your search.
+
The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see [https://cumulus.hosiene.co.uk/viewtopic.php?p=155539#p155539 changed rainfall units example]), or to use a spreadsheet (be careful not to affects dates or times) like '''Libre Office Calc''' where you can paste special a multiplier to all cells in a particular column.
  
===Using the Cumulus backup===
+
===Correcting extremes just for a few logging entries  (plus selected days/months/years in a short period)===
  
If for any reason, the '''Alltimelog.txt''' cannot be used (maybe you deleted it in error), and you are within a week of when the rogue value updated the extreme, an alternative is to compare the '''alltime.ini''' with a previous version of the same file that has been stored in a [[backup_folder|''backup'' directory]]. Basicallly, Cumulus takes a backup of most of the active data files when it starts up, and also at the start of the meteorological day (just after midnight for a lot of users). The backups are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours).
+
There are a further two sub-cases that fall in this category. Both are near impossible to resolve!
  
If you can find the latest backup stored for a date and time from just before the ''alltime.txt'' was updated with the rogue value, look in that backup copy to see the previous value that was in the file, and use that value when you follow editing instructions below.
+
Both Cumulus 1 and MX have had bugs in some releases of their software. This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process!
  
===Searching your standard log files===
+
[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt,  for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).
  
Any search through your monthly [[Standard_log_files]] has two disadvantages:
+
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
#those lines only record a small sample of spot values as by default they are only created every 10 minutes (so even if you choose to load all log files, it might not include the actual extreme you have missed recording).
+
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries.  The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.
#as the files are typically large, it takes some time to load the contents of the files.
 
  
Both the original Cumulus (it is not clear from which version, it might be from 1.7.x) and MX (from version 3.2.0 build 3056) provide all time record editors.  These editors have the ability (as described below) to load up both the [[dayfile.txt|daily summary log file]] and all your [[Standard_log_files]], so you can look at the extremes that are recorded in those files, and use that information to determine what to replace your rogue value with.
+
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
 
 
The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files.
 
 
 
==Using the provided editors==
 
 
 
Both Cumulus 1 and MX provide an editor (at most versions of the relevant flavour, update to a new release if your build does not include an editor).
 
 
 
When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:
 
* [[File:Badge v1.png]] select '''All-time records''' in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
 
* [[File:Badge vMx.png]]select '''All Time records''' in the [[MX_Administrative_Interface#All_time_records|edit menu]] of MX admin interface.
 
 
 
The way the editor allows you to change a value depends on which flavour you are using:
 
* [[File:Badge v1.png]] Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across). Press Save button to retain the change and exit.
 
* [[File:Badge vMx.png]] You have to select either a value or a time-stamp that you want to change before you can type in a replacement value (you have to type the new number in even if it is displayed by loading the log files). Then click the tick to retain the change, and that will allow you to select another value or time-stamp to edit, before you finally exit the editor.
 
 
 
Please remember any edit you make here will not affect the related extreme in your monthly-all-time extremes recorded in [[monthlyalltime.ini]]. Nor can you update [[Standard_log_files]], or [[dayfile.txt|daily summary log]], by any edit made here. If you are using MX, it has not affected related tables on your database server.
 
  
==Actions after you have edited all-time entries==
+
In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent '''not-working'''. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null.  Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations.  However, there is no official solution to this problem!
  
''In all cases, an edit made to an entry in '''alltime.ini''', means that you need to make the same change in '''monthlyalltime.ini'''. Look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed. How to edit the monthly-all-time extreme records are described in next section.
+
==All-time corrections==
  
If you loaded up the monthly standard log files into the all-time editor, you should be able to see:
+
For the all-time extreme records, each individual update is logged in [[Alltimelog.txt]] from version 1.8.9 onwards. Depending on sequence of extreme values, you may get an accurate previous value from this tracking log. 
*if there is an entry for same time-stamp as that associated with the former rogue value as seen in '''Alltimelog.txt'''
+
* The tracking log will not tell you a correct high/low extreme record if ''the rogue extreme occurred before the actual high/low extreme'' was experienced. This is because the rogue extreme stopped any subsequent true extreme being recorded.  
*you should also be able to work out, from '''Alltimelog.txt''', whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
+
* If the ''actual high/low extreme that you want to retain was recorded before the rogue extreme'', then you can see that true value, and its time-stamp, in the tracking log. Based on that time-stamp, the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
 
  
If the correction you have made is in the current month, you will also need to change the entries in '''month.ini''' and '''year.ini'''. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter.
 
  
The all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.
+
==Introduction of Monthly All Time Extreme records==
 
 
 
 
=Correction of Monthly All Time Extreme records=
 
  
 
From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
 
From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
Line 226: Line 291:
 
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]).
 
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]).
  
== How do I correct my monthly all-time records? ==
+
==Monthly-all-time corrections==
  
In many respects, the instruction for the all-time editing above also apply to monthly all-time.
+
If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the [[Monthlyalltime.ini|monthly-all-time]] file.  
*[[File:Badge v1.png]] If you are a Cumulus 1 user, you do not have a [[monthlyalltimelog.txt]] file to look in to see the log entry with the previous and new values. This information was logged into a file in [[Diags_folder|the diagnostic files folder]]. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.  
+
* From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.  
* [[File:Badge vMx.png]] If you are a MX user, see if you have a [[monthlyalltimelog.txt]] file in the same [[data_folder| data folder]], and then you will know what value to revert to.
+
* If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected.
Here the old and new values can be looked up in the [[monthlyalltimelog.txt]] file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in [[Diags_folder]].
+
* If you use MX, then [[Monthlyalltimelog.txt]]) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded.
  
  
There are the same alternative ways to look for the values to enter in the monthly-all-time entries that you need to change, as were described earlier to look up value for all-time. If you have already done an all-time edit, then look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed for monthly-all-time.
+
==Correction of extremes for past year==
  
The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.
+
An earlier correction may have identified that the rogue value was in a past year, so this sub-section explores whether you can continue correction pathway:
 +
* [[File:Badge v1.png]]Cumulus 1 never allows you to see a '''year.ini''' file when the year is completed, because at the end of the year it is initialised ready for the new year.
 +
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, the MX beta (3.0.0), and later MX releases, at the start of a new year, saves the old year.ini (whenever it was last updated) as a file with a name like '''year2015.ini''', and only then overwrites the ''year.ini'' file.  
 +
** Although MX does not provide any functionality to read this old file, let alone edit it, you may want to use a standard text editor to amend the relevant part of this old file. Your edit to either ''alltime.ini'' or ''monthlyalltime.ini'' should have told you what old value in old file is wrong, and told you correct value to replace that.
  
==Actions after you have edited monthly-all-time entries==
+
==Correction of extremes for past month==
  
If you loaded up the monthly standard log files into the monthly-all-time editor, you should be able to see:
+
An earlier correction may have identified that the rogue value was in a past month, so this sub-section explores whether you can continue correction pathway:
*if there is an entry for same time-stamp as that associated with the former rogue value as seen in '''monthlyalltimelog.txt'''
+
* [[File:Badge v1.png]]Cumulus 1 never allows you to see a '''month.ini''' file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
*you should also be able to work out, from '''monthlyalltimelog.txt''', whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.  
+
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, the MX beta (3.0.0), and later MX releases, at the end of a month, saves the old '''month.ini''' (whenever it was last updated) as a file with a name like '''month201501.ini''' (i.e. "month", followed by year, followed by month number, and with file type ".ini"), before writing standard "reset high/low values" to '''month.ini'''.  
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
+
** Although MX does not provide any functionality to read this old file, let alone edit it, you may want to use a standard text editor to amend the relevant part of this old file. Your edit to either ''alltime.ini'' or ''monthlyalltime.ini'' should have told you what old value in old file is wrong, and told you correct value to replace that.
  
If the correction you have made is in the current month, you will also need to change the entries in '''month.ini''' and '''year.ini'''. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like '''month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.
 
  
The monthly-all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.
+
==Current month/year corrections==
  
=Correction of extremes for today=
+
If your earlier correction (''finding how the rogue value has affected monthly-all-time has given you a date'' in the current month/year), that is a steer to whether the [[Month.ini|this month]] extremes file will need correction, and whether [[Year.ini|this year]] extreme records file will need correction. 
  
The Cumulus '''Edit' menu includes a '''Today's rain''' option where you can adjust the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus|total rainfall for today]] (e.g. if you or the wind have knocked your rain gauge) as described below. There is no facility provided to edit any other content of [[today.ini]], but since '''today.ini''' is used to create lines in '''dayfile.txt''', you can follow instructions in [[Amending_dayfile]] to make any necessary corrections for past days.
+
They are relatively small files, so it should be easy to use the [[#Built in extreme record editors]] to edit them.
  
  
== There is an error in today's total rainfall ==
+
==Correction of extremes for past days==
  
Easy - correct today's total using the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus | 'today's rain']] editor on the edit menu.
+
If the rogue value relates to yesterday, or an earlier day, then you must edit [[Amending dayfile|dayfile.txt]] to make any necessary corrections for past days.
* [[File:Badge v1.png]]select 'Today's rain in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
 
* [[File:Badge vMx.png]]select '''Today's rain''' in the [[MX_Administrative_Interface#Today.27s_rain.27|edit menu]] of MX admin interface.
 
  
This edit will actually alter the start of day rainfall counter figure:
+
It is entirely optional whether you choose to update [[Yesterday.ini|yesterday.ini]] if that contains a rogue value. That file is overwritten at both midnight and at next rollover, so in general there is no benefit gained from any editing.
*If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter
 
*If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter
 
  
Please note that this edit does not affect "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]], your database will retain incorrect values, as these are not updated by this correction.
+
=== Correcting an error in rainfall for a past day===
  
Please see  [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit related fields.
+
For the legacy Cumulus 1, rainfall corrections are covered at [[FAQ#My_station_invented_some_rain_that_didn.27t_really_occur.2C_and_I_want_to_set_it_to_zero_.28or_some_other_figure.29]].
  
=Correction of extremes for month-to-date=
+
The total rainfall for a month, or 12-month season (rainfall labelled as ''Rain this year'' can start counting from zero on first [[Meteorological day]] of any month chosen by user (Interface: '''Station settings''' &rarr; ''Annual Rainfall'' %rarr; '''Start of rainfall season'''), is held in an internal (RAM) variable in Cumulus software. When you start Cumulus software, the program looks in [[dayfile.txt]] and reads the amount for each past day (of current month/season) in that file, then adds the rainfall for today-so-far. For rain this season, Cumulus flavour alters calculation:
 +
* Cumulus 1 - if the year entered in [[Cumulus_Screenshots#Station]] "Annual Rainfall" frame, box labelled ''Year'', matches the current '''calendar''' year, then the amount in box labelled ''Amount'' is added.  Note if your season does not start in "January" then this only affects the part of the season that is in the specified year.
 +
* Cumulus MX - if your rainfall season starts in a month within year specified in box labelled "Year to which year-to-date amount applies", then the amount in box labelled "Year-to-date amount" is added to your rain for that season as shown in  ''Rain this year'' (and the &lt;ryear&gt; tag if used for [[MySqlConnect|Custom SQL]], MQTT, [[Cumulus_MX_Local_API]], or [[Cumulus_template_file]])
  
As mentioned in passing above, a rogue value may get recorded in a [[month.ini]] file.  
+
So if the rainfall reported for current month, or current "season" (year) is wrong, you need to correct [[dayfile.txt]], both Cumulus 1 and MX have [[Amending_dayfile#Editors_built_into_Cumulus|built-in dayfile.txt editors]].
  
You should ensure you have got the daily summary log ([[Amending_dayfile|dayfile.txt]]) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this month. 
 
  
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit '''This year's records'''. The editors work in exactly the same way as was described for all-time above.
+
 
*[[File:Badge v1.png]]In Cumulus 1, '''Copy''' buttons enable you to copy a record from '''dayfile'''. Click '''OK'' to save.
+
==Today corrections==
* [[File:Badge vMx.png]] In MX, the [[dayfile.txt|dayfile]] value/timestamp, and the [[Standard_log_files|Logfile]] value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click '''Tick'' to save.
 
  
 +
As described below, the Cumulus '''Edit' menu includes a '''Today's rain''' option where you can adjust the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus|total rainfall for today]] (e.g. if you or the wind have knocked your rain gauge).
  
=Correction of extremes for past month=
+
''There is no facility provided to edit any other content'' of [[today.ini]].  Manual correction may be possible, but do read advice on [[today.ini]] page, in particular noting that MX only reads "today.ini" when first started, MX uses an internal array that represents content of file while MX is running.
  
*[[File:Badge v1.png]]Cumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
+
In working through the various files in above table, remember that if the rogue value was recorded today, then [[today.ini]] will be wrong:
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the month.ini at the start of a new year, it saves the old month.ini (whenever it was last updated) as a file with a name like '''month201501.ini'''.  
+
* In Cumulus 1, you possibly could edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file
 +
* In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see [[MX on Linux]] or [[MX on Windows OS]]) and edit the file using a text editor, or programmer's editor, that will not add unwanted content to the file.
  
MX does not provide any functionality to read this old file, let alone edit it.  However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.
 
  
=Correction of extremes for  year-to-date=
+
=== Correcting an error in today's total rainfall ===
  
As mentioned in passing above, a rogue value may get recorded in a [[year.ini]] file.  
+
You can correct rainfall total (for current [[meteorological day]] since rollover).  
  
You should ensure you have got the daily summary log ([[Amending_dayfile|dayfile.txt]]) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this year.
+
Just use the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus | 'today's rain']] editor on the edit menu.
 +
* [[File:Badge v1.png]]select ''Today's rain'' in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
 +
* [[File:Badge vMx.png]]select '''Today's rain''' in the [[MX_Administrative_Interface#Today.27s_rain.27|edit menu]] of MX interface.
  
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit '''This year's records'''. The editors work in exactly the same way as was described for all-time above.
+
This edit will actually alter '''the start of day rainfall counter figure held by Cumulus internally''' (RAM):
*[[File:Badge v1.png]]In Cumulus 1, '''Copy''' buttons enable you to copy a record from '''dayfile'''. Click '''OK'' to save.
+
* If you want today's rain to seem less (perhaps you, an animal, or the wind, knocked the rain gauge), Cumulus will increase the start of day counter
* [[File:Badge vMx.png]] In MX, the [[dayfile.txt|dayfile]] value/timestamp, and the [[Standard_log_files|Logfile]] value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click '''Tick'' to save.
+
* If you want today's rain to seem greater (perhaps the rain gauge got blocked by a leaf, or a bird), Cumulus will decrease the start of day counter
 
+
* Be aware that if you stop Cumulus, and restart it, before end of meteorological day, internally held variables are lost, so the start of day figure will revert
=Correction of extremes for past year=
 
 
 
*[[File:Badge v1.png]]Cumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
 
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the year.ini at the start of a new year, it saves the old year.ini (whenever it was last updated) as a file with a name like '''year2015.ini'''.
 
 
 
MX does not provide any functionality to read this old file, let alone edit it.  However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.
 
  
 +
Please note that this edit ''does not affect'' "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", ''nor does it update'' ANY EXISTING data log line so it has correct rainfall counter reading. Please see  [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit those related fields so when Cumulus is restarted all are corrected.
  
 +
If you send rainfall data to:
 +
* any third-party external web site
 +
* MQTT or other local system
 +
* any database tables (e.g. if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]])
 +
''then be aware that'' that this correction does '''not send any update''' for any messages/values '''already sent'''.
  
 
=Some definitions=
 
=Some definitions=
  
==Rogue value==
 
 
In this article, the term '''rogue value''' is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.
 
 
Here are a typical examples:
 
* it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
 
* perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
 
* an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong
 
  
 
==Flavour, Release, Version, and Build==
 
==Flavour, Release, Version, and Build==
Line 324: Line 382:
 
'''Version''' here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.
 
'''Version''' here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.
  
'''Build''' number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number. For recent MX releases, the developer has changed version number every time there is a new build released.
+
'''Build''' number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number.
 +
 
 +
Steve Loft generally went through a lot of beta releases identified by build number before finally having a stable release with new version number. Most beta releases were available to everyone.
 +
 
 +
Mark Crossley has sometimes issued his beta commits identified by beta, beta 1, beta 2, etc. without changing build number, and sometimes incremented build number several times between changes to release version number. Most beta releases are made available to just a select few testers, via a sub-forum with limited access, or via direct email of zip files to particular people. The beta history is not documented in "updates.txt", which only quotes the more significant changes.

Latest revision as of 09:21, 9 September 2022

Terminology

For simplicity, the terminology "extremes" is used on this page, the meaning includes:

  1. The totals maintained (such as rainfall, chill hours, and the various "degree days"), and
  2. The high/low "extreme records" for various periods (such as all-time, and this month).


This Wiki page has been created to cover both Cumulus Version 1 Specific and Cumulus Version MX Specific, but do be aware that some terminology varies between the two flavours. All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered between Wiki pages for each specific file, this redesign brings together all such text here).

Rogue value

Throughout this Wiki the term rogue value is used to mean you see a value somewhere in Cumulus that you believe should not be there.

Generally, "rogue" usage refers to a single data point. However, where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files. Specifically, on this Wiki page, the meaning covers any incorrect entry in one or more of the extreme record files.

Here are some typical examples:

  • it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
  • perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
  • an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong

How Cumulus software tracks "extremes"

As Cumulus weather software processes each reading from your weather station, it checks that value (and any derived from it) against the extremes currently stored in various RAM variables, and if necessary updates the extreme records that are affected. Please note these extreme records are held by Cumulus software in internal variables. Periodically, these totals and high/low "extreme records" are written out to .ini files (ensuring such "extremes" are kept when MX is stopped and restarted).

The totals/extreme records that are maintained in this way are:

Period File storing extremes How to correct Link to web tag section Notes
For current day so far today.ini Editor for "Today's rain" (no editor for other derivatives) today.htm Many entries in this file (for non-midnight rollover, use is made of yesterday.ini too) get transferred to dayfile.txt at end of day.
For past days dayfile.txt See Amending dayfile Web tags only exist for yesterday Often used as source for corrections - see Depends on source selected Note
For current month-to-date month.ini Editor for "This month's records" thismonth.htm Please see #Accuracy Note
monthyyyyMM.ini are archived copies for past months
For current year-to-date year.ini Editor for "This year's records" thisyear.htm Please see Accuracy Note
yearyyyy.ini are archived copies for past years
For all readings since a 'start date alltime.ini Editor for "All Time Records" records.htm See table below for start date
For a particular month in all years monthlyalltime.ini Editor for "Monthly Records" monthlyrecord.htm Similar to previous row, but different start date, and separate extremes maintained for each month (regardless of the year)

Explaining columns in above table:

  1. The first (label) column is self-explanatory
  2. The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
  3. The third column gives the name for the Edit menu item to choose to edit these extreme records
  4. The links in fourth column leads you to more information about the web tags associated with that period, you can incorporate those in your own templates.


Built in extreme record editors

Badge v1.png Cumulus 1 gained extreme record editing functionality from version 1.9.1 6th April 2011. See screenshot Edit menu for how to select which file to edit, once on required editing screen, follow instructions on screen:

  • Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across and click "Copy" to copy figure from identified log file to extreme record file). Press Save button to retain the change and exit.

Cumulus MX gradually gained extreme record editing functionality in releases from 3.1.1 - b3054 to 3.4.0 - b3064 (1 Nov 2019 to 19 Feb 2020), with a major redesign of user interface in release 3.18.0 (b.3189 14 June 2022):

  • In latest MX release, if you click on an individual extreme record, then a pop-up appears where you can directly edit value and time-stamp to typed new content
  • To load data from Standard log files you have to click on a button, from release 3.2.0 - build 3056 - only the relevant dayfile.txt entries are shown by default
  • In latest MX release, if you click on a value in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy value across or not, this does not update date/time
  • In latest MX release, if you click on a date/time-stamp in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy date/time across or not, this does not update value
  • There is some validation on editing, you cannot empty the content of any extreme record


The built-in editors do allow you to load the standard monthly log files as well as the daily summary file (dayfile.txt). The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files. However, do exercise caution about using values from such sources, because as will be explained next, they may not hold the extreme values.

How editing accuracy depends on source selected

The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:

  1. The (hopefully more accurate) figure taken from a search for that extreme by examining all entries in the dayfile.txt for that period
  2. The (usually less accurate) figure taken from a search for that extreme by examining all entries in the Standard log files for that period

Obviously, it is possible that the dayfile.txt file has been corrupted, and if you are using MX, then you may want to read about a utility that can recreate dayfile.txt.

But generally, dayfile.txt is the best starting point for recreating any longer period extreme records, to understand why read on:

  • Using Standard log files as source for recalculating past extremes:
    • Let us assume you are using the default logging interval of 10 minutes
    • Unlike some other weather station software available (which logs highest and lowest since previous log), Cumulus logs spot values
    • That means the Monthly log files do not capture any extremes recorded in the time (by default 599 seconds) between logs
    • Therefore the detailed log files are not normally the most accurate source
    • Please note, this less accurate way of deducing daily extremes/totals (to update dayfile.txt) is used by Cumulus software:
  • Using dayfile.txt as source for recalculating past extremes
    • MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
    • Therefore extremes recorded in today.ini' (and from there into dayfile.ini) are based on the full sampling done by Cumulus
    • This means none, or very few, extremes are missed
    • In March 2021, a new utility Create Records was planned (for use with MX only), as at July 2021 no progress has been made in coding it. It appears that this utility will read dayfile.txt and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other files like monthly-all-time and all-time. Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available.


Cautionary warning and other files to examine

File corruption can happen as a response to an electrical supply problem, a temporary input/output problem within the operating system, or the storage device being used, as well as after corruption of data within your weather station generating a rogue value fed through to current files.

Any or all of the Extreme Record .ini Files currently in use by Cumulus may get corrupted, as Cumulus gains exclusive write access that can overwrite the entire file during an update.

Remember the values displayed in the built-in editor from dayfile.txt or monthly standard log files just might have been corrupted in the same problem. Cumulus only appends new lines to the end of these files, so it should never overwrite the whole file, but it is possible for a connection problem to make Cumulus start a new file.

Therefore it is worth noting where else you can look to find values and date/time-stamps to use when correcting rogue data, and the next few subsections make some suggestions.

Update tracking logs

Cumulus 1 logs most extreme updates in files stored in Diags folder. Read that cross-reference for more guidance. As already mentioned, there is a log Alltimelog.txt that tracks the updates to all-time extremes.

Cumulus MX can log useful information in MXdiags folder, depending on settings mentioned in that cross-reference, but it maintains two logs Alltimelog.txt and Monthlyalltimelog.txt as already mentioned.

These tracking logs can, in certain circumstances, be the best place to look up previous values as replacements for rogue values, when the built-in editors reveal that the rogue values exist in dayfile.txt and (possibly) the standard monthly log, so you can't within the editors find the correct value you seek.


Looking at graphical representations

Many people find it easy to interpolate replacements for rogue values by looking at graphical representations of their weather data covering before and after the time when the rogue figure got recorded.

In Cumulus 1, the obvious place to look is select-a-graphs (available from version 1.2 released on 5th April 2004).

In Cumulus MX, later releases also have a select a chart feature, that may be more useful than the standard charts (in both interface web pages and default web pages).

Some plots record values every minute, and those high resolution plots are ideal for your search.

Using the Cumulus backup

Cumulus makes backups of the extreme record files that are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours), so this method cannot be used for rogue values that are a week old or older.

If you notice the rogue update when it happens, remember provided you act, as soon as possible afterwards, you may be able to make use of the earlier version of the relevant extreme records file, as a source of correct extreme records before the corruption by a rogue figure.

All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see today.ini), with their contents just as they were either with the end of day or start of day. It is therefore possible no true extreme has happened since the most recent backup, or maybe by comparing two recent back-ups you can obtain guidance on when the last true extreme occurred. Obviously, such back-up files are no use for correcting daily extremes, but for this month, monthly-all-time, this year, and all-time, extreme records, updates to extremes don't always happen every day, especially when near end of a month. Therefore there is a good chance that you can find the previous extreme by examining a backup copy, providing a true extreme has not happened since.

Searching recent history

Cumulus 1 only provides one way to access the Recent_History, and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags.

Cumulus MX stores its Recent history in a SQLite3 database that you can read/edit as explained at Cumulusmx.db#Reading.2Fediting_database_table_outside_MX.


General Editing Advice

The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values.

You may have a feel as to which files in above table will need correction, but if in doubt it is highly recommended that you always start your extreme record correction by seeing if the error is present in the Alltime.ini file that holds all-time extreme records. That approach is best, partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier.


Extreme monitoring start-dates

As Cumulus has developed, various derived values have been added that it calculates in addition to whatever your weather station supplies. At some releases, these extras are only available via web tags for current values, and it may be some significant time later that a release makes them available as all-time, or other period, extremes. You may be able to track these changes by examining "changes.txt" for Cumulus 1 or "updates.txt" for MX, but those sources are not comprehensive.

There is no guarantee that this Wiki content has been checked, or that it is up to date. Any contributor is welcome to make corrections or bring it up to date

For simplicity, this article will only document the development of yearly functionality and attempts to record when various extreme records became available in that context.

It should be obvious that full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, for Cumulus 1, daily extreme functionality was added first, then all-time, followed by this month/year extreme functionality, and finally monthly-all-time. For MX, generally extras were added as current values first, and later the extremes for all the various periods were added together, but development does not always happen in a consistent way!


This section has intentionally been kept brief, so it does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.


Icon info.png

The start date referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:

Parameter First released First in Version First in Build
highest/lowest apparent temperature 26 Oct 2010 1.9.1 beta 957
highest/lowest feels like temperature 24 June 2020 3.6.10 3086
highest Canadian Humidity Index (humidex) 28 Jul 2020 3.7.0 3089
highest minimum temperature 15 April 2004 1.2.2 (lost)
highest USA heat index 29 Aug 2010 1.9.0 beta 955
wettest month 5 April 2004 1.2 (lost)
24 hour rainfall 30 Apr 2022 3.16.0 3182
highest daily wind run 3 Jul 2011 1.9.2 beta 1001
Sunshine hours 31 July 2021 3.12.0 3134

Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked (lost), and there is some vagueness in information mentioned elsewhere in this page.


Correcting multiple extremes

There are two cases to consider:

Correcting extremes recorded for every logging entry (plus every day/month/year of long period)

Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units.

In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any Extra Sensor Files, any Standard log files, in dayfile.txt, plus the multiple.

The built-in editors only correct one extreme record at a time, so they cannot be used for such a task.

It is important to remember that there are source and derived values in Cumulus. If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see Calculate Missing Values page for further advice.

The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see changed rainfall units example), or to use a spreadsheet (be careful not to affects dates or times) like Libre Office Calc where you can paste special a multiplier to all cells in a particular column.

Correcting extremes just for a few logging entries (plus selected days/months/years in a short period)

There are a further two sub-cases that fall in this category. Both are near impossible to resolve!

Both Cumulus 1 and MX have had bugs in some releases of their software. This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process!

Badge v1.pngThere were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in File:Changes.zip, although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within Cumulus forum announcements, but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in month.ini and year.ini depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt, for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in dayfile.txt for the whole of that period).

Badge vMx.pngCumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The updates.txt that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in the relevant support forum will yield more information in return for a lot more effort.

A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.

In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent not-working. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null. Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations. However, there is no official solution to this problem!

All-time corrections

For the all-time extreme records, each individual update is logged in Alltimelog.txt from version 1.8.9 onwards. Depending on sequence of extreme values, you may get an accurate previous value from this tracking log.

  • The tracking log will not tell you a correct high/low extreme record if the rogue extreme occurred before the actual high/low extreme was experienced. This is because the rogue extreme stopped any subsequent true extreme being recorded.
  • If the actual high/low extreme that you want to retain was recorded before the rogue extreme, then you can see that true value, and its time-stamp, in the tracking log. Based on that time-stamp, the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit.


Introduction of Monthly All Time Extreme records

From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.

Initialisation of monthly-all-time extreme records

Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one Copy button, the one in the header row, all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking OK button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see Calculate Missing Values).

Monthly-all-time corrections

If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the monthly-all-time file.

  • From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
  • If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected.
  • If you use MX, then Monthlyalltimelog.txt) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded.


Correction of extremes for past year

An earlier correction may have identified that the rogue value was in a past year, so this sub-section explores whether you can continue correction pathway:

  • Badge v1.pngCumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
  • Badge vMx.png From build 3035 released 2 Dec 2015, the MX beta (3.0.0), and later MX releases, at the start of a new year, saves the old year.ini (whenever it was last updated) as a file with a name like year2015.ini, and only then overwrites the year.ini file.
    • Although MX does not provide any functionality to read this old file, let alone edit it, you may want to use a standard text editor to amend the relevant part of this old file. Your edit to either alltime.ini or monthlyalltime.ini should have told you what old value in old file is wrong, and told you correct value to replace that.

Correction of extremes for past month

An earlier correction may have identified that the rogue value was in a past month, so this sub-section explores whether you can continue correction pathway:

  • Badge v1.pngCumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
  • Badge vMx.png From build 3035 released 2 Dec 2015, the MX beta (3.0.0), and later MX releases, at the end of a month, saves the old month.ini (whenever it was last updated) as a file with a name like month201501.ini (i.e. "month", followed by year, followed by month number, and with file type ".ini"), before writing standard "reset high/low values" to month.ini.
    • Although MX does not provide any functionality to read this old file, let alone edit it, you may want to use a standard text editor to amend the relevant part of this old file. Your edit to either alltime.ini or monthlyalltime.ini should have told you what old value in old file is wrong, and told you correct value to replace that.


Current month/year corrections

If your earlier correction (finding how the rogue value has affected monthly-all-time has given you a date in the current month/year), that is a steer to whether the this month extremes file will need correction, and whether this year extreme records file will need correction.

They are relatively small files, so it should be easy to use the #Built in extreme record editors to edit them.


Correction of extremes for past days

If the rogue value relates to yesterday, or an earlier day, then you must edit dayfile.txt to make any necessary corrections for past days.

It is entirely optional whether you choose to update yesterday.ini if that contains a rogue value. That file is overwritten at both midnight and at next rollover, so in general there is no benefit gained from any editing.

Correcting an error in rainfall for a past day

For the legacy Cumulus 1, rainfall corrections are covered at FAQ#My_station_invented_some_rain_that_didn.27t_really_occur.2C_and_I_want_to_set_it_to_zero_.28or_some_other_figure.29.

The total rainfall for a month, or 12-month season (rainfall labelled as Rain this year can start counting from zero on first Meteorological day of any month chosen by user (Interface: Station settingsAnnual Rainfall %rarr; Start of rainfall season), is held in an internal (RAM) variable in Cumulus software. When you start Cumulus software, the program looks in dayfile.txt and reads the amount for each past day (of current month/season) in that file, then adds the rainfall for today-so-far. For rain this season, Cumulus flavour alters calculation:

  • Cumulus 1 - if the year entered in Cumulus_Screenshots#Station "Annual Rainfall" frame, box labelled Year, matches the current calendar year, then the amount in box labelled Amount is added. Note if your season does not start in "January" then this only affects the part of the season that is in the specified year.
  • Cumulus MX - if your rainfall season starts in a month within year specified in box labelled "Year to which year-to-date amount applies", then the amount in box labelled "Year-to-date amount" is added to your rain for that season as shown in Rain this year (and the <ryear> tag if used for Custom SQL, MQTT, Cumulus_MX_Local_API, or Cumulus_template_file)

So if the rainfall reported for current month, or current "season" (year) is wrong, you need to correct dayfile.txt, both Cumulus 1 and MX have built-in dayfile.txt editors.


Today corrections

As described below, the Cumulus Edit' menu includes a Today's rain option where you can adjust the total rainfall for today (e.g. if you or the wind have knocked your rain gauge).

There is no facility provided to edit any other content of today.ini. Manual correction may be possible, but do read advice on today.ini page, in particular noting that MX only reads "today.ini" when first started, MX uses an internal array that represents content of file while MX is running.

In working through the various files in above table, remember that if the rogue value was recorded today, then today.ini will be wrong:

  • In Cumulus 1, you possibly could edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file
  • In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see MX on Linux or MX on Windows OS) and edit the file using a text editor, or programmer's editor, that will not add unwanted content to the file.


Correcting an error in today's total rainfall

You can correct rainfall total (for current meteorological day since rollover).

Just use the 'today's rain' editor on the edit menu.

  • Badge v1.pngselect Today's rain in the edit menu accessed from main screen in Cumulus 1,
  • Badge vMx.pngselect Today's rain in the edit menu of MX interface.

This edit will actually alter the start of day rainfall counter figure held by Cumulus internally (RAM):

  • If you want today's rain to seem less (perhaps you, an animal, or the wind, knocked the rain gauge), Cumulus will increase the start of day counter
  • If you want today's rain to seem greater (perhaps the rain gauge got blocked by a leaf, or a bird), Cumulus will decrease the start of day counter
  • Be aware that if you stop Cumulus, and restart it, before end of meteorological day, internally held variables are lost, so the start of day figure will revert

Please note that this edit does not affect "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", nor does it update ANY EXISTING data log line so it has correct rainfall counter reading. Please see Today.ini#Editing_rainfall_in_today.ini_within_Cumulus for details of how to edit those related fields so when Cumulus is restarted all are corrected.

If you send rainfall data to:

  • any third-party external web site
  • MQTT or other local system
  • any database tables (e.g. if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the Standard_log_files)

then be aware that that this correction does not send any update for any messages/values already sent.

Some definitions

Flavour, Release, Version, and Build

Flavour is used to represent the original Cumulus, Cumulus 2, and Cumulus MX, collectively. Where the text says applicability is dependant on flavour, it means that the action you do depends on whether you are installing/running Cumulus MX or the original Cumulus software.

Release is used to signify what the Cumulus developer makes available for download after there has been a modification to the software. For most Cumulus 1 builds, the release consisted of a executable that would create all the folders and files needed to run that software. For Cumulus 2, releases were a zip file and were numbered using a identifier with many digits. MX releases are as a zip file that is labelled with the build number.

Version here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.

Build number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number.

Steve Loft generally went through a lot of beta releases identified by build number before finally having a stable release with new version number. Most beta releases were available to everyone.

Mark Crossley has sometimes issued his beta commits identified by beta, beta 1, beta 2, etc. without changing build number, and sometimes incremented build number several times between changes to release version number. Most beta releases are made available to just a select few testers, via a sub-forum with limited access, or via direct email of zip files to particular people. The beta history is not documented in "updates.txt", which only quotes the more significant changes.