5,838
edits
Correcting Extremes: Difference between revisions
m
no edit summary
m (→Introduction: corrected link) |
mNo edit summary |
||
|
=Introduction=
As Cumulus processes each reading from your weather station, it checks that value (and any [[Correcting_Missing_Values#Derived spot values|derived]] from it) against the extremes currently stored in
{| class="wikitable" border="1"
|-
!style="width:200px"|Period
!style="width:50px"|File
!style="width:50px"|Example web page
!style="width:200px"|Notes
|-
|For current day so far
|[[today.ini]]
|[[Webtags#Today|today.htm]]
|Many entries in this file get transferred to [[dayfile.txt]] at end of day
|-
|For current month-to-date
|[[month.ini]]
|[[Webtags#Monthly|thismonth.htm]]
|
|-
|For current year-to-date
|[[year.ini]]
|[[Webtags#Yearly|thisyear.htm]]
|
|-
|For all readings since a '''start date''
|[[alltime.ini]]
|[[Webtags#All_Time|records.htm]]
|See table below for start date
|-
|For a particular month in all years
|[[monthlyalltime.ini]]
|
|}
Following the links in the second column leads you to more information about the relevant file. Following links in third column leads you to more information about the web tags that you can incorporate in your own [[Customised_templates|templates]].
The purpose of this article is to help you to edit incorrect entries in any of the .ini files in the second column, to understand where else to investigate, and where necessary amend other log files.
The article starts with corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).
There is more information in [[:Category:Log_Files]] and the pages relating to individual files, for all of the extreme holding files.
{{TOCright}}
==Why might your extremes need to be corrected?==
If there is a rogue value read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.
It is also possible that you have discovered that you made a mistake in setting up or calibrating a sensor, and this leads you to identifying a constant/multiplier adjustment
to be applied to adjust all values over the past period.
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
==All-time extreme functionality==
For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time. Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.
{{Version badge 1}}There were bugs introduced sometimes in builds of the original Cumulus (1). Information about a few of the bugs and fixes can be found in [[at File:Changes.zip]], although that does not cover any 1.7.x versions, and does not detail bugs created and fixed within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort as there are a lot of posts to search. (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log for the relevant period, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.
[[File:Icon_info.png]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has added more extreme records to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
{| class="wikitable" border="1"
|-
!style="width:200px"|Parameter
!style="width:50px"|First released
!style="width:50px"|First in Version
!style="width:50px"|First in Build
|-
|highest/lowest apparent temperature
|26 Oct 2010
|1.9.1 beta
|957
|-
|highest/lowest feels like temperture
|24 June 2020
|3.6.10
|3086
|-
|highest Canadian Humidity Index (humidex)
|28 Jul 2020
|3.7.0
|3089
|-
|highest minimum temperature
|15 April 2004
|1.2.2
|(lost)
|-
|highest USA heat index
|29 Aug 2010
|1.9.0 beta
|955
|-
|wettest month
|5 April 2004
|1.2
|(lost)
|-
|highest daily windrun
|3 Jul 2011
|1.9.2 beta
|1001
|}
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elesewhere in this page.
=Correction of All Time Extreme records=
Cumulus software makes it easy to correct the all-time extremes (held in [[alltime.ini]]).
This is because whenever Cumulus makes an update to that file, from version 1.8.9 onwards, it logs the previous and new values to the [[Alltimelog.txt]] file. Consequently, if you detect a rogue value, you can look up the latter file to determine what how to revert an entry in the former file.
==Alternative ways to find a replacement for a rogue value==
Of course, it is possible that the old value in '''Alltimelog.txt''' is not appropriate. It might be that after the rogue value was stored in '''alltime.ini''', a new extreme was seen, and this new extreme was different to the previous value stored in '''Alltimelog.txt''', but it did not cause an update in '''alltime.ini''' because of the rogue value that was stored there being more extreme.
===Looking at graphical representations===
One way to check whether the above speculation is correct is by looking at Cumulus 1 select-a-graphs (available from version 1.2 released on 5th April 2004), or MX charts in the admin interface (available in , if you can find a plot that covers the period between when the false extreme was recorded and now you are ready to do a correction, you can look for evidence of a new extreme that has been missed. (If you only have access to your web pages, then the sample trends web page may, or may not, cover the relevant period). Some plots record values every minute, and those high resolution plots are ideal for your search.
===Using the Cumulus backup===
If for any reason, the '''Alltimelog.txt''' cannot be used (maybe you deleted it in error), and you are within a week of when the rogue value updated the extreme, an alternative is to compare the '''alltime.ini''' with a previous version of the same file that has been stored in a [[backup_folder|''backup'' directory]]. Basicallly, Cumulus takes a backup of most of the active data files when it starts up, and also at the start of the meteorological day (just after midnight for a lot of users). The backups are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours).
If you can find the latest backup stored for a date and time from just before the ''alltime.txt'' was updated with the rogue value, look in that backup copy to see the previous value that was in the file, and use that value when you follow editing instructions below.
===Searching your standard log files===
Any search through your monthly [[Standard_log_files]] has two disadvantages:
#those lines only record a small sample of spot values as by default they are only created every 10 minutes (so even if you choose to load all log files, it might not include the actual extreme you have missed recording).
#as the files are typically large, it takes some time to load the contents of the files.
Both the original Cumulus (it is not clear from which version, it might be from 1.7.x) and MX (from version 3.2.0 build 3056) provide all time record editors. These editors have the ability (as described below) to load up both the [[dayfile.txt|daily summary log file]] and all your [[Standard_log_files]], so you can look at the extremes that are recorded in those files, and use that information to determine what to replace your rogue value with.
The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files.
==Using the provided editors==
Both Cumulus 1 and MX provide an editor (at most versions of the relevant flavour, update to a new release if your build does not include an editor).
When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:
* [[File:Badge v1.png]] select '''All-time records''' in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
* [[File:Badge vMx.png]]select '''All Time records''' in the [[MX_Administrative_Interface#Today.27s_rain.27|edit menu]] of MX admin interface.
* [[File:Badge vMx.png]] You have to select either a value or a time-stamp that you want to change before you can type in a replacement value (you have to type the new number in even if it is displayed by loading the log files). Then click the tick to retain the change, and that will allow you to select another value or time-stamp to edit, before you finally exit the editor.
Please remember
==Actions after you have edited all-time entries==
''In all cases, an edit made to an entry in '''alltime.ini''', means that you need to make the same change in '''monthlyalltime.ini'''. Look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed. How to edit the monthly-all-time extreme records are described in next section.
If you loaded up the monthly standard log files into the all-time editor, you should be able to see:
*if there is an entry for same time-stamp as that associated with the former rogue value as seen in '''Alltimelog.txt'''
*you should also be able to work out, from '''Alltimelog.txt''', whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
If the correction you have made is in the current month, you will also need to change the entries in '''month.ini''' and '''year.ini'''. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter.
The all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.
=Correction of Monthly All Time Extreme records=
From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
==Initialisation of monthly-all-time extreme records==
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Creating_Missing_Values]]).
== How do I correct my monthly all-time records? ==
In many respects, the instruction for the all-time editing above also apply to monthly all-time.
*[[File:Badge v1.png]] If you are a Cumulus 1 user, you do not have a [[monthlyalltimelog.txt]] file to look in to see the log entry with the previous and new values. This information was logged into a file in [[Diags_folder|the diagnostic files folder]]. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.
* [[File:Badge vMx.png]] If you are a MX user, see if you have a [[monthlyalltimelog.txt]] file in the same [[data_folder| data folder]], and then you will know what value to revert to.
Here the old and new values can be looked up in the [[monthlyalltimelog.txt]] file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in [[Diags_folder]].
There are the same alternative ways to look for the values to enter in the monthly-all-time entries that you need to change, as were described earlier to look up value for all-time. If you have already done an all-time edit, then look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed for monthly-all-time.
The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.
==Actions after you have edited monthly-all-time entries==
If you loaded up the monthly standard log files into the monthly-all-time editor, you should be able to see:
*if there is an entry for same time-stamp as that associated with the former rogue value as seen in '''monthlyalltimelog.txt'''
*you should also be able to work out, from '''monthlyalltimelog.txt''', whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.
If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).
If the correction you have made is in the current month, you will also need to change the entries in '''month.ini''' and '''year.ini'''. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like '''month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.
The monthly-all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.
=Correction of extremes for today=
The Cumulus '''Edit' menu includes a '''Today's rain''' option where you can adjust the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus|total rainfall for today]] (e.g. if you or the wind have knocked your rain gauge) as described below. There is no facility provided to edit any other content of [[today.ini]], but since '''today.ini''' is used to create lines in '''dayfile.txt''', you can follow instructions in [[Amending_dayfile]] to make any necessary corrections for past days.
== There is an error in today's total rainfall ==
Easy - correct today's total using the [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus | 'today's rain']] editor on the edit menu.
* [[File:Badge v1.png]]select 'Today's rain in the [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|edit menu]] accessed from main screen in Cumulus 1,
* [[File:Badge vMx.png]]select '''Today's rain''' in the [[MX_Administrative_Interface#Today.27s_rain.27|edit menu]] of MX admin interface.
This edit will actually alter the start of day rainfall counter figure:
*If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter
*If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter
Please note that this edit does not affect "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]], your database will retain incorrect values, as these are not updated by this correction.
Please see [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit related fields.
=Correction of extremes for month-to-date=
As mentioned in passing above, a rogue value may get recorded in a [[month.ini]] file.
You should ensure you have got the daily summary log ([[Amending_dayfile|dayfile.txt]]) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this month.
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit '''This year's records'''. The editors work in exactly the same way as was described for all-time above.
*[[File:Badge v1.png]]In Cumulus 1, '''Copy''' buttons enable you to copy a record from '''dayfile'''. Click '''OK'' to save.
* [[File:Badge vMx.png]] In MX, the [[dayfile.txt|dayfile]] value/timestamp, and the [[Standard_log_files|Logfile]] value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click '''Tick'' to save.
=Correction of extremes for past month=
*[[File:Badge v1.png]]Cumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the month.ini at the start of a new year, it saves the old month.ini (whenever it was last updated) as a file with a name like '''month201501.ini'''.
MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.
=Correction of extremes for year-to-date=
As mentioned in passing above, a rogue value may get recorded in a [[year.ini]] file.
You should ensure you have got the daily summary log ([[Amending_dayfile|dayfile.txt]]) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this year.
Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit '''This year's records'''. The editors work in exactly the same way as was described for all-time above.
*[[File:Badge v1.png]]In Cumulus 1, '''Copy''' buttons enable you to copy a record from '''dayfile'''. Click '''OK'' to save.
* [[File:Badge vMx.png]] In MX, the [[dayfile.txt|dayfile]] value/timestamp, and the [[Standard_log_files|Logfile]] value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click '''Tick'' to save.
=Correction of extremes for past year=
*[[File:Badge v1.png]]Cumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
* [[File:Badge vMx.png]] From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the year.ini at the start of a new year, it saves the old year.ini (whenever it was last updated) as a file with a name like '''year2015.ini'''.
MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.
=Some definitions=
| |||