5,838
edits
Correcting Extremes: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m
Rewrite of out of date page for MX 3.18.0 changes
m (→Introduction: restored accidentally deleted cell) |
m (Rewrite of out of date page for MX 3.18.0 changes) |
||
|
| See [[Amending dayfile]]
| Web tags only exist for [[Webtags#Yesterday|yesterday]]
| Often used as source for corrections - see [[#
|-
! scope="row"|For current month-to-date
| Editor for "This year's records"
|[[Webtags#Yearly|thisyear.htm]]
| Please see [[#How editing accuracy depends on source selected|Accuracy Note]]
|-
|colspan="5" style="background:pink;"|yearyyyy.ini are archived copies for past years
# The third column gives the name for the '''Edit''' menu item to choose to edit these extreme records
# The links in fourth column leads you to more information about the web tags associated with that period, you can incorporate those in your own [[Customised_templates|templates]].
=General Editing Advice=
This Wiki page talks about [[#Rogue value|''rogue'' values]], meaning any incorrect entry in one or more of the [[:Category:Ini Files|extreme record files]]. You may have a feel as to which files in above table will need correction, but generally it is best to start with all-time and finish with daily extreme records, not the reverse order.
The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values.
It is highly recommended that you always start your extreme record correction by seeing if the error is present in the [[Alltime.ini]] file that holds all-time extreme records. That approach is best partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier. For the all-time extreme records, each individual update is logged in [[Alltimelog.txt]] from version 1.8.9 onwards. You may get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded. If the error has affected all-time, looking at the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit.
If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the [[Monthlyalltime.ini|monthly-all-time]] file.
* From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
* If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected.
* If you use MX, then [[Monthlyalltimelog.txt]]) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded.
Again, finding the rogue value has affected monthly-all-time gives you a date, that is a steer to whether the [[Month.ini|this month]] extremes file will need correction, and whether [[Year.ini|this year]] extreme records file will need correction. They are relatively small files, so it should be easy to use the built-in editors to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same [[data_folder| data folder]], with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like '''month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.
In working through the various files in above table, remember that if the rogue value was recorded today, then [[today.ini]] will be wrong:
* In Cumulus 1, you can edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file
* In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see [[MX on Linux]] or [[MX on Windows OS]]) and edit the file using a text editor or programmer's editor that will not add unwanted content to the file.
If the rogue value relates to yesterday, then you must edit [[Amending dayfile|dayfile.txt]], and you may choose to update [[Yesterday.ini|yesterday.ini]] although this latter file will be overwritten at next rollover.
==Correcting multiple extremes==
There are two cases to consider:
===Correcting all extremes===
Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units.
In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any [[Extra Sensor Files]], any [[Standard log files]], in [[dayfile.txt]], plus the multiple [[Category:Ini Files|extreme record files]].
The built-in editors only correct one extreme record at a time, so they cannot be used for such a task.
It is important to remember that there are [[Calculate_Missing_Values#Some_definitions|source and derived values]] in Cumulus. If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see [[Calculate Missing Values]] page for further advice.
The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see [https://cumulus.hosiene.co.uk/viewtopic.php?p=155539#p155539 changed rainfall units example]), or to use a spreadsheet (be careful not to affects dates or times) like '''Libre Office Calc''' where you can paste special a multiplier to all cells in a particular column.
===Correcting extremes within a period===
There are a further two sub-cases that fall in this category. Both are near impossible to resolve!
Both Cumulus 1 and MX have had bugs in some releases of their software. This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process!
[[File:Badge v1.png]]There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in [[File:Changes.zip]], although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within [https://cumulus.hosiene.co.uk/viewforum.php?f=2 Cumulus forum announcements], but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in '''month.ini''' and '''year.ini''' depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt, for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in '''dayfile.txt''' for the whole of that period).
[[File:Badge vMx.png]]Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records
reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The '''updates.txt''' that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in [https://cumulus.hosiene.co.uk/viewforum.php?f=40 the relevant support forum] will yield more information in return for a lot more effort.
A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.
In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent '''not-working'''. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null. Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations. However, there is no official solution to this problem!
=Built in extreme record editors=
[[File:Badge v1.png]] Cumulus 1 gained extreme record editing functionality from version 1.9.1 6th April 2011. See screenshot [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu|Edit menu]] for how to select which file to edit, once on required editing screen, follow instructions on screen:
* Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across and click "Copy" to copy figure from identified log file to extreme record file). Press Save button to retain the change and exit.
Cumulus MX gradually gained extreme record editing functionality in releases from 3.1.1 - b3054 to 3.4.0 - b3064 (1 Nov 2019 to 19 Feb 2020), with a major redesign of user interface in release 3.18.0 (b.3189 14 June 2022):
* In latest MX release, if you click on an individual extreme record, then a pop-up appears where you can directly edit value and time-stamp to typed new content
* To load data from [[Standard log files]] you have to click on a button, from release 3.2.0 - build 3056 - only the relevant [[dayfile.txt]] entries are shown by default
* In latest MX release, if you click on a value in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy value across or not, this does not update date/time
* In latest MX release, if you click on a date/time-stamp in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy date/time across or not, this does not update value
* There is some validation on editing, you cannot empty the content of any extreme record
The built-in editors do allow you to load the standard monthly log files as well as the daily summary file (dayfile.txt). The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files. However, do exercise caution about using values from such sources, because as will be explained next, they may not hold the extreme values.
==How editing accuracy depends on source selected==
# The (usually less accurate) figure taken from a search for that extreme by examining all entries in the [[Standard log files]] for that period
Obviously, it is possible that the dayfile.txt file has been corrupted, and if you are using MX, then you may want to read [[Calculate_Missing_Values#CreateMissing.exe|about a utility that can recreate dayfile.txt]].
But generally, dayfile.txt is the best starting point for recreating any longer period extreme records, to understand why read on:
* Using [[Standard log files]] as source for recalculating past extremes:
** Let us assume you are using the default logging interval of 10 minutes
** In March 2021, a new utility '''Create Records''' was planned (for use with MX only), as at July 2021 no progress has been made in coding it. It appears that this utility will read '''dayfile.txt''' and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other [[:Category:Ini Files|files like monthly-all-time and all-time]]. ''<big>Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available</big>''.
==Cautionary warning and other files to examine==
File corruption can happen as a response to an electrical supply problem, a temporary input/output problem within the operating system, or the storage device being used, as well as after corruption of data within your weather station generating a rogue value fed through to current files.
Any or all of the [[:Category:Ini Files| Extreme Record .ini Files]] ''currently in use by Cumulus'' may get corrupted, as Cumulus gains exclusive write access that can overwrite the entire file during an update.
Remember the values displayed in the built-in editor from dayfile.txt or monthly standard log files just might have been corrupted in the same problem. Cumulus only appends new lines to the end of these files, so it should never overwrite the whole file, but it is possible for a connection problem to make Cumulus start a new file.
Therefore it is worth noting where else you can look to find values and date/time-stamps to use when correcting rogue data, and the next few subsections make some suggestions.
===Update tracking logs===
Cumulus 1 logs most extreme updates in files stored in [[Diags folder]]. Read that cross-reference for more guidance. As already mentioned, there is a log [[Alltimelog.txt]] that tracks the updates to all-time extremes.
Cumulus MX can log useful information in [[MXdiags folder]], depending on settings mentioned in that cross-reference, but it maintains two logs [[Alltimelog.txt]] and [[Monthlyalltimelog.txt]] as already mentioned.
These tracking logs can, in certain circumstances, be the best place to look up previous values as replacements for rogue values, when the built-in editors reveal that the rogue values exist in dayfile.txt and (possibly) the standard monthly log, so you can't within the editors find the correct value you seek.
===Looking at graphical representations===
Many people find it easy to interpolate replacements for rogue values by looking at graphical representations of their weather data covering before and after the time when the rogue figure got recorded.
In Cumulus 1, the obvious place to look is select-a-graphs (available from version 1.2 released on 5th April 2004).
In Cumulus MX, later releases also have a select a chart feature, that may be more useful than the standard charts (in both interface web pages and default web pages).
Some plots record values every minute, and those high resolution plots are ideal for your search.
===Using the Cumulus backup===
Cumulus makes backups of the extreme record files that are kept in folders within the [[Backup folder|backup sub-folder]] in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours), so this method cannot be used for rogue values that are a week old or older.
If you notice the rogue update when it happens, remember provided you act, as soon as possible afterwards, [[Calculate_Missing_Values#Reading_archive_data|you may be able to make use]] of the earlier version of the relevant extreme records file, as a source of correct extreme records before the corruption by a rogue figure.
All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see [[today.ini]]), with their contents just as they were either with the end of day or start of day. It is therefore possible no true extreme has happened since the most recent backup, or maybe by comparing two recent back-ups you can obtain guidance on when the last true extreme occurred. Obviously, such back-up files are no use for correcting daily extremes, but for this month, monthly-all-time, this year, and all-time, extreme records, updates to extremes don't always happen every day, especially when near end of a month. Therefore there is a good chance that you can find the previous extreme by examining a backup copy, providing a true extreme has not happened since.
===Searching recent history===
Cumulus 1 only provides one way to access the [[Webtags#Recent_History|Recent_History]], and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags.
Cumulus MX stores its [[Recent history]] in a SQLite3 database that you can read. To avoid making this page too technical, it won't tell you what software can read that database, but if you have the technical knowledge, you can look up details of suitable packages to install to read the entries in the database, and perhaps find correct extremes recorded there, as that for the last week stores measurements at (normally) one minute resolution.
==Extreme monitoring start-dates==
As Cumulus has developed, various derived values have been added that it calculates in addition to whatever your weather station supplies. At some releases, these extras are only available via web tags for current values, and it may be some significant time later that a release makes them available as all-time, or other period, extremes. You may be able to track these changes by examining "changes.txt" for Cumulus 1 or "updates.txt" for MX, but those sources are not comprehensive.
''There is no guarantee that this Wiki content has been checked, or that it is up to date. Any contributor is welcome to make corrections or bring it up to date''
For simplicity, this article will only document the development of yearly functionality and attempts to record when various extreme records became available in that context.
It should be obvious that full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, for Cumulus 1, daily extreme functionality was added first, then all-time, followed by this month/year extreme functionality, and finally monthly-all-time. For MX, generally extras were added as current values first, and later the extremes for all the various periods were added together, but development does not always happen in a consistent way!
This section has intentionally been kept brief, so it does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.
[[Image:Icon info.png|left|30px]]The '''start date''' referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:
|-
! scope="row"|highest Canadian Humidity Index (humidex)
| 28 Jul 2020
| 3.7.0
|3089
|-
! scope="row"|highest minimum temperature
| 15 April 2004
| 1.2.2
|(lost)
|-
! scope="row"|highest USA heat index
| 29 Aug 2010
| 1.9.0 beta
|955
|-
! scope="row"|wettest month
| 5 April 2004
| 1.2
|(lost)
|-
| 24 hour rainfall
| 30 Apr 2022
| 3.16.0
| 3182
|-
! scope="row"|highest daily wind run
|1.9.2 beta
|1001
|-
| Sunshine hours
| 31 July 2021
| 3.12.0
| 3134
|}
Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked ''(lost)'', and there is some vagueness in information mentioned elsewhere in this page.
=Correction of Monthly All Time Extreme records=
Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one '''Copy''' button, the one ''in the header row'', all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking '''OK''' button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see [[Calculate Missing Values]]).
=Correction of extremes for today=
*If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter
Please note that this edit does not affect "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the [[Standard_log_files]], your database will retain incorrect values, as these are not updated by this correction.
Please see [[Today.ini#Editing_rainfall_in_today.ini_within_Cumulus]] for details of how to edit related fields.
MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either ''alltime.ini'' or ''monthlyalltime.ini'' and know what old value is wrong, and what should the value for future.
=Correction of extremes for past year=
'''Version''' here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.
'''Build''' number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number
Steve Loft generally went through a lot of beta releases identified by build number before finally having a stable release with new version number. Most beta releases were available to everyone.
Mark Crossley has sometimes issued his beta commits identified by beta, beta 1, beta 2, etc. without changing build number, and sometimes incremented build number several times between changes to release version number. Most beta releases are made available to just a select few testers, via a sub-forum with limited access, or via direct email of zip files to particular people. The beta history is not documented in "updates.txt", which only quotes the more significant changes.
| |||
