5,838
edits
Today.ini: Difference between revisions
no edit summary
No edit summary |
|||
|
[[Category:Ini Files]]
= Introduction to tracking of Daily Extremes/Totals/Averages =
Steve Loft first wrote Cumulus software in 2003, although it was not shared with the public until 27th January 2004. His (no longer available) web site claimed (amongst other reasons) that Cumulus was invented to cope with [[Meteorological_day|tracking high and low for days starting at 9 a.m.]] as (at that time) no other software could do that. Therefore, it can be assumed that even in 2003, Cumulus software was tracking the highest/lowest for all time as well as for each day . (For daily period, there is also tracking of some totals, and in some cases by dividing by number of samples averages can be calculated). This Wiki page covers all aspects of the daily tracking.
The legacy Cumulus 1 software has a '''Recent extremes''' frame on its [[Cumulus_Screenshots#Main_Screen|main screen]]. In release 1.0 (27th January 2004) this only showed extremes for today, but from 1.1 (17th February 2004) it also shows extremes for yesterday. The interface provided with Cumulus MX is also able to show the extremes for today and yesterday (on a single web page).
=Essential reading=
Cumulus 1 has a [[Downloads#Resources|Help file]] that is part of the Cumulus 1 installation package; that provides a simple explanation for each file.
David Jamieson created this Wiki page on 27 August 2009, to cover "today.ini" and "yesterday.ini" files. His introductory text simply stated these files were for tracking extremes in the two days and new files were created at 9am or midnight rollover). David also included a listing of an example just for "today.ini". Since then, the Wiki has attempted to become reference documentation that answers questions that arise frequently in the support forum. This does mean its pages have become harder to read, and thus there is now an attempt to split "essential" facts that give a basic understanding (as in following subsections) from "technical" facts (more complex with instructions on how to cope with problems).
Therefore for a quick read, stop after this essential section or use the page index above to skip quickly to whatever is of interest to you.
== How Cumulus tracks extremes and why two files were introduced ==
Cumulus software handles daily tracking by storing values internally, i.e. within its use of random access memory. As the software processes (the time interval for doing this varies between weather station type and also on exactly which release you are running) data from the weather station, incoming values are compared against internally held values, and when appropriate the values being processed update the internal values.
Cumulus initially made the assumption that it would be left running continuously, so holding values internally enabled them to be shown on that screen. However, subsequently it was realised that Cumulus had to be stopped and restarted to install a new version (and Microsoft's updates system restarts computers as part of its installation process). Information held in RAM is lost when the software is closed.
Apparently when version 1.3 was released (18 January 2005) it did not include "today.ini" nor "yesterday.ini". An attempt to track the history of the "today.ini" file appears [[Changes to this file at particular releases|below]]. Unfortunately, Steve Loft lost some of his notes, and the Cumulus 1 version history in his [[Downloads#Earlier_Versions|changes.txt]] is incomplete. So it is pure guesswork that version 1.4 resolved the problem of losing daily information on closing Cumulus by introducing a '''today.ini'' file to hold the daily extremes and total records.
Steve Loft did document that at rollover, the daily extreme/average records were "transferred" (there are some differences) from "today.ini" to [[yesterday.ini]] to make it easier to display them on the main screen of the legacy software. The sub-section below covers this in more detail.
When you close Cumulus, it will write the final values for highs and lows and their timestamps to today.ini as part of the close down process.
==End of day actions==
End of day actions happen when Cumulus detects that it is processing weather data for the rollover time [either midnight, or 9am (or 10am) depending on your configuration and season], this might be during the processing of archive data while Cumulus is catching up after it has been restarted, or in normal running when those clock times are reached.
Oversimplifying the process a bit, the contents of today.ini, goes to two places:
# yesterday.ini
# a new line appended onto [[dayfile.txt]]
After that a new today.ini is created populated with initial values for each extreme/total entry.
The three files have a few differences in content, so rolling-over does involve a little editing work:
# The multiple lines in the [General] section of today.ini shrink to just one item in yesterday.ini (Date) and one item in dayfile.txt (abbreviated date)
# The 'Total' and 'Samples' values in "today.ini" become the single 'AvgTemp' in "yesterday.ini" and "dayfile.txt"
=== Complexity if you are recording sunshine hours ===
Sunshine hours are recorded starting at midnight, regardless of what rollover time is used.
For MX only, the sunshine hours are partly recorded in [[yesterday.ini]] and partly in today.ini. So even if your rollover is 9 am, there will be an update to both files at midnight which is when the current sunshine hours figure in today.ini is copied to yesterday.ini before the figure in today.ini is reset to zero.
==Where are the files stored?==
The files '''today.ini''' and '''[[yesterday.ini]]''' are located in the [[Data folder|"data" sub-folder]] of the path where your cumulus.exe (or CumulusMX.exe) is installed.
(NOTE: Microsoft Windows Operating Systems, may relocate some files, as explained at [[FAQ#I_can.E2.80.99t_find_my_data_files.21| FAQ on location of data log files]]).
When Cumulus is left running, a daily backup of all the files is created as part of this rollover in a subfolder 'daily' of the [[Backup_folder|backup folder]]. Depending on the release you are running, the files included in the backup may be a snapshot of their content just before rollover, just after rollover, or at the first standard interval after rollover.
== How Cumulus updates the file ==
This depends on which flavour of Cumulus you run. The [[#End of day actions|end of day process]] is same for both flavours, but updates during the day work differently for the legacy software and for MX.
For '''today.ini''', it is important to be aware that MX can read a file created by Cumulus 1 (MX can read the time format in the value for '''Timestamp=''' parameter whether it is in Cumulus 1 or MX format), but Cumulus 1 cannot read a file that has been updated by MX.
Cumulus MX and the legacy software handle the updating of "today.ini" differently, this difference is critical should you want to [[Correcting_Extremes#Correction_of_extremes_for_today|edit out rogue data]].
=== How Cumulus 1 updates the file ===
Steve Loft never shared his souce code, so what follows is just a guess at how the update might work. The frequency at which data is read from a weather station varies depending on the type, but is at least every 30 seconds.
The indications are that Cumulus 1 has an internal one minute timer that triggers the logging of readings (after conversion to units selected) to an internally held
[[Recent history|recent history database]] and the comparison of those readings against existing internally held extremes/totals. The file is updated immediently afterwards, each update only changes those lines (within sections) where the extreme/total/count/time has changed, and other lines retain the same content as before the update.
Whilst you are strongly advised not to manually edit the file with Cumulus running, because access to the file cannot be shared, if you are able to complete the edit between one real-time interval and the next, any change you make is retained.
(NOTE: A full set of latest spot readings are logged to a file at a configurable interval that might be every 10 or 30 minutes).
===How MX replaces the file===
Note the subtle difference in this sub-section header. MX does all updates to the today extremes/totals/counts only to internally held values. Although the [[Software#Latest_MX_Source_Code|source code]] is available, you need to be more technical than the person typing this to understand at what frequency these internal updates occur. It is probably every time data is read from the weather station, but might be just when the externally stored [[Recent history|recent history database]] is updated.
Critically, MX only updates the "today.ini" file at the configurable [[Standard log files|Standard interval]] used for logging the spot values (might be every 10 or 30 minutes). Each update is a ''rewrite of the entire file contents'', from the internally (RAM) held values.
You would be wasting your time should you try a manual edit of the file while MX is running, as the next MX update will overwrite any manual changes! Don't be misled by the fact that a sharing violation is less likely to be an issue (because of the longer interval between MX updates).
== Format of the file ==
The files are text files, consisting of many lines. Some lines consist of a single piece of text surrounded by square brackets, these are the section names. The sections (for "today.ini" after the first [General]) can be in any order, Cumulus will maintain whatever order the sections are currently in.
Under each section name, there is a list of parameters <code>attribute=value</code>, with one parameter per line. The attribute names are defined by Cumulus, but can appear in any order:
* Where a time-stamp is stored, note that in "today.ini" only the hour and minute parts of a time are stored.
* Where an extreme/total value is stored, note that it is always post conversion to the units selected by the Cumulus user.
** It was mentioned earlier that the file exists to store values that Cumulus holds in RAM. Internally those numbers are in binary (base 2), but in the file the numbers are expressed to base 10. The value of any integer part of numbers is unchanged between the two bases, but decimal parts in base 2 and base 10 do not convert exactly, therefore in the file you may see some strange looking numbers with lots of decimal places.
The key difference between all [[:Category:Ini_Files]] for the different flavours is the formatting of any time-stamps that include a date:
* [[File:Badge v1.png]] Cumulus 1 used the format specified in Control Panel for your region settings, for UK that would typically be <code>day/month/year (space) hour:minute</code> in today.ini
* Cumulus 2 (withdrawn) worked differently, all date/time stamps were converted to UTC, and stored in ISO 8601 format of <code>yyyy-MM-ddTHH:mm:ss</code>(using the net specifiers that MX uses).
* [[File:Badge vMx.png]] Cumulus MX uses the same format as Cumulus 2, but all date/times are expressed according to date/time read from the computer running MX, so use whatever time-zone you have selected on that device.
Please look at the category page (link above) to read more on formatting differences between the legacy Cumulus and MX.
===Typical Sections within file===
As Cumulus is developed it is adding further sections, but the legacy Cumulus (by 1.9.4) used the following:
* The [General] section stores current date and time.
* The [Wind} section stores the highest wind speed and highest gust, it holds the sum of wind speeds as wind run, and details for the dominant wind.
* The [Temp], or temperature, section stores the highest and lowest temperature, the sum of all temperatures from every sample (Total), and the number of Samples in that total, the Cumulative Chill Hours total (for the season), and the cumulative Heating and Cooling Degree Days for the current day-so-far.
* The [Pressure], [Humidity], [AppTemp] (for apparent temperature), and [Dewpoint] sections just hold Highs and Lows.
* The [WindChill] section only holds lowest, the [HeatIndex] section only has highest.
* The [Rain] section holds a lot of different parameters, including the Start count that derives most rain outputs, and the LastTip date-time.
* Other sections present in 1.9.4 are [ET], [Solar], [NOAA], and [FineOffset]; whether these are updated depends on what sensors you have, and whether you have set up NOAA reports.
MX development has been far more rapid than the legacy software, and there have been a lot of changes to the content of the two files. Unfortunately, the MX release announcements rarely go into enough detail to permit good documentation, so all mentions relating to MX on this page are guesses from examination of the file, and it cannot be guanteed that the information is correct for whatever MX release you may be running!
A typical MX release will include all the 1.9.4 sections, plus:
* The '''[Records]''' section contains one line denoting when extreme records were last revised. For example "Alltime=2020-03-06T06:42:13", indicates when the all-time extreme records was last updated. In the example file, there were various extreme records broken at that time on 6 March (lowest temperature, lowest apparent temperature, and greatest wind chill), but none have been broken since.
* Other sections (depending on MX release) may include [FeelsLike], [Humidex], [Lightning], and [TempMidnight]; whether these are updated depends on what sensors you have.
===Composite Example of 'today.ini' file===
'''Notes in round brackets () are not part of the file, they simply explain elements of the composite.'''
* This is an example made up of composites that may not all be present in an actual file.
* This composite contains time-stamp formats used by Cumulus 1.x.y and Cumulus MX in the few places where they differ.
* Because this example is made up of composites, the times shown are not all consistent, in a real file no time anywhere will be later than the time at the top!
* The order the sections appear in this composite may not match your file; as mentioned above the section order can be edited.
<pre>
[General]
Date=29/09/
Timestamp= (
CurrentYear=
CurrentMonth=9
CurrentDay=13
Start=1923.59997558594
Yesterday=0
LastTip=
HourlyHigh=0
HHourlyTime=00:00
LatestYearlyReport=
[FineOffset] (
FOSensorClockTime=(the format here is different for Cumulus 1 e.g. "29/09/
FOStationClockTime=(the format here is different for Cumulus 1 e.g. "29/09/
FOSolarClockTime=0001-01-01T00:00:00 (this parameter only appears in later MX releases, the time shown is the default if no solar sensor)
</pre>
<br>
<hr>
<br>
=Non-essential information=
You can skip the following subsections unless you have a particular need to read them.
== First use of Cumulus ==
When you use Cumulus software for the very first time, it records a start date, and assumes you have no data anywhere for earlier than that start time.
Some weather station types have an internal logger that can be accessed for historic data. It is theoretically possible, but not recommended, to read in that historic data, so it is included in your Cumulus extreme tracking. The minimum content for "today.ini" is the "[General]" section. If you stop Cumulus, manually edit "today.ini" to remove all other sections, and then restart Cumulus, then the software will attempt to read that historic data. Please read [[FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station | FAQ here]] for full guidance. Remember this only applies when you are first starting use of Cumulus with a weather station.
== Restart and Catch-up ==
If you restart Cumulus during the day it will read the ''today.ini'' file at startup, so it can resume tracking extremes of the key parameters starting from latest stored values in today.ini. If your weather station type permits, a restart of Cumulus can go into a catch-up mode during which it reads any historic data from the weather station, for the period while Cumulus was not running, before starting the reading of the current data.
During that catch-up of historic data, internally held daily extremes, and the "today.ini" file, will be updated with revised highs and lows as Cumulus processes the historic data from the station's memory; and if necessary Cumulus will do a roll-over (see [[#End of day actions]]) as it processes the readings for the relevant time.
On restart Cumulus writes a backup of today.ini (and some of the other Cumulus files) into the [[Backup_folder|backup folder]] found below the folder with the cumulus.exe (or CumulusMX.exe). With Cumulus stopped, you can copy the today.ini file in either a restart backup, or a daily backup (see [[#Where are the files stored?]]), in conjuction with copying the other files in that backup folder into their original folders (mostly [[data_folder|data sub-folder]] overwriting the files in those destinations. When you restart Cumulus, the tracking will begin again as if the time has been rewound back to the date those copied files were last updated. This rewinding works best if historic data can be read from your weather station. One example of when this might be useful is if you spot a rogue value very soon after it has been recorded, the rewinding often brings in correct data for the recent period. Another example is if your computer on rebooting initially shows the wrong time and so Cumulus records some data against that wrong time; a rewind can eliminate the wrongly timed records and replace them with records timed correctly.
You are strongly advised not to stop/restart Cumulus close to either midnight or your rollover time. Steve Loft defines "close" in this context as within whatever time you have set as interval between logging of spot values (e.g. 10 or 30 minutes). The potential problems were significantly worse for earlier versions of Cumulus 1, but restart problems have been reduced in newer builds of C1. In general, MX is more tolerant over restart timings, but the way its code works you will encounter more problems with accuracy of output if MX is stopped for more than the few minutes needed for an upgrade (or computer reboot).
== Editing rainfall in today.ini within Cumulus ==
This sub-section applies to Cumulus 1 and Cumulus MX.
At the time of writing, no Cumulus release contains an editor for today's daily extreme records as stored in "today.ini". There is just one exception, both flavours do contain an editor for 'Today's rain'.
Cumulus internally stores a "start of day rain counter". This is copied into "today.ini" into the ''Start='' line within the '[rain]' section. Subtracting that from the current rain counter value allows Cumulus to calculate (in your chosen units) the toatl rainfall for today. The provided editor changes the internal "start of day rain counter" so that the current rainfall total becomes what you enter into the editor.
It is important to note:
* The rainfall counter was invented in Cumulus 1 as a way that could calculate rain when the software was made to serve multiple weather station types
* The counter is based on a piece of data that can be read from a particular weather station type, it might be based on an all-time total rainfall, or based on a rainfall this year figure
* The counter is not intended to be of any interest to the Cumulus user
* Any increase in the rainfall counter is treated as representing valid rain (spike removal functionality exists only for rain rate and rain in last hour, not the total rain)
* Cumulus has special code to detect if the counter value decreases, without going into the complexity, Cumulus will normally reset its internal start of day counter based on the decrease in the number received.
* The "Today's rain" editor does not affect any derived rainfall output;
** The following in "today.ini" are not affected:
*** last tip time-stamp ("LastTip=")
*** highest rainfall rate so far today and time-stamp ("High=" and "HTime=")
*** hourly high amount and time-stamp ("HourlyHigh=" and "HHourlyTime=")
*** Highest 24 hour amount
** Outside "today.ini" it does not affect any recent records nor monthly, yearly, monthly-all-time, or all-time extremes:
*** The recent history entries whether held internally (Cumulus 1), or for MX in [[Cumulusmx.db]], are not amended so any tags used for web page data, local API, custom SQL, MQTT, HTTP (see [[Webtags/Parameters#Cumulus_MX_software|this Wiki section]]) will report incorrect values
*** Total Rainfall this month, rainfall this year/season will only be recalculated if Cumulus is stopped and restarted after the edit
*** Highest rainfall rate, highest hourly rainfall, highest 24-hour rainfall, highest daily rainfall; applying to these longer periods are not changed unles each extreme record is individually manually edited (Cumulus provides an editor for most of these, but cannot generally suggest what new value to use).
*** The consecutive wet/dry days ("ConsecutiveRainDays=" and "ConsecutiveDryDays=") in this month/year - remember these counts are to last roll-over, they exclude today
** The editor does not alter any lines already logged in [[Standard_log_files |log file]] for the current month. This means you will see distorted graphs attempting to portray rainfall within a day
** The editor does not update any database records that may be affected by the change of the rainfall total
== Dealing with rogue values ==
The "today.ini" file is written to throughout the current day, as
If your weather station reports a rogue value, an incorrect update to a High, Low, or Total, may get stored in this file. At end of day, it will then be stored in [[dayfile.txt]]. Cumulus developer advice is that instead of manually editing "today.ini", you should wait until the day has been stored in "dayfile.txt" and then use the editor for that log file to make the desired changes.
The rogue value may also affect extreme records held for this month, this year, monthly-all-time, and/or all-time. Cumulus does provide editing functionality for most (not all) entries in the files holding thgose extreme records and you can read instructions on [[Correcting Extremes]] Wiki page. Here it is sufficient to say it is worth looking in the diagnostics, to see if you can spot when the problem occurred, because that helps you work out what may be affected:
* [[File:Badge vMx.png]] [[MXdiags folder]]
* [[File:Badge v1.png]] [[Diags_folder]]
=== Manual editing of "today.ini" ===
The developer advice to wait until next day has a flaw that every process that happens in the meantime sees wrong data, and that might include sending data to several external sites and a number of extra custom processes you might have in your system.
Therefore this Wiki page will now give some advice on how to manually edit the file:
# You must stop Cumulus (please see earlier in this page for details as reason depends on flavour you are running)
# Take a back-up of existing file, or rename it (if you decide it easiest to create a new file rather than edit the existing one)
# Any plain text file editor can be used (that includes coding editors like Geany, Notepad++, NoteTab, and many others)
# Be careful to ensure any change maintains existing format (integer, decimal,time, date/time, text) paying attention to any punctuation (including decimal commas or decimal points, direction type of any slashes)
# For readability you can insert blank lines into files '''today.ini''' and '''yesterday.ini''', Cumulus will not mind.
# It is up to you to work out what new value/time to type, Cumulus won't accept nulls, but there are some "initial values/dates" that it will accept (I won't tell you these here, because they depend on release you are running and you should not be editing this file unless you have enough technical understanding to work out what Cumulus will accept)
# Remember changing extremes for [[Calculate_Missing_Values#Some_definitions|source value extremes and derived extremes]] is complicated:
#* Please see specific advice in sub-sections below
==== Edits related to pressure ====
Should be straight forward, the maximum/minimum can be edited without this affecting anything else. You should be able to look in either [[Standard_log_files |log file]] or for MX in [[Cumulusmx.db]] for pressure readings earlier or later in the day to find the new extreme value/time to replace the rogue pressure value and time.
==== Edits relating to Temperature ====
#If you change a temperature source extreme (i.e. highest/lowest temperature) you should also be taking the old value out of 'Total' line and decreasing by one the'Samples' count
# You cannot work out from any change in a temperature source extreme (i.e.. highest/lowest temperature) how to change the related derived extremes (e.g. wind chill, apparent temperature, feels like, dew point). Each derived value is worked out by combining spot values at a particular time, so you have to recalculate as many spot derived values as possible in order to work out the new derived extreme.
==== Edits related to humidity or wind speed ====
Obviously changes to humidity, wind speed, also have an effect on derived temperature extremes mentioned above, but again you have to recalculate as many spot derived values as possible in order to work out the new derived extreme.
====Edits related to rainfall ====
If you want to edit ''rainfall since midnight'' in today.ini, it should be fairly easy to calculate a new value. Remember you also need to manually edit entries in [[Standard_log_files |log file]] and for MX in [[Cumulusmx.db]].
To edit the "rainfall counter" in [[Standard_log_files |log file]] remember that for any decrease in the actual rain, you need to increase the rain counter, and vice versa. It is probably easiest to work backwards through time working out the alteration to the rain counter for the actual rainfall increment, in the period since the preceding entry.
Other derivatives like "maximum hourly rain", "highest rainfall rate", or "highest 24 hour rainfall", all will have to be guessed or recalculated manually, there is no easy way to work these out as past values for the day are not logged anywhere.
==Cross-references for explanations of key parameters==
[[Wind_measurement | Bearing and Direction]],
[[Windrun]];
[[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | Temp]],
[[Average temperature | Total and Samples]],
[[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | Humidity]],
[[Heat/cold degree days and Chill hours | Heating degree days]],
[[Heat/cold degree days and Chill hours | Cooling degree days]],
[[Apparent temperature | AppTemp]],
[[Heat index]],
[[wind chill]],
[[Heat/cold degree days and Chill hours | Chill hours]],
[[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | Dewpoint]];
[[Webtags|Rain]],
[[FAQ#The_longest_dry.2Fwet_spell_for_this_month.2Fyear_includes_days_from_last_month.2Fyear | ConsecutiveRainDays]], [[FAQ#The_longest_dry.2Fwet_spell_for_this_month.2Fyear_includes_days_from_last_month.2Fyear | ConsecutiveDryDays]];
[[Reports folder|NOAA]]
== Changes to this file at particular releases ==
''This section may not be a complete history, and may not be kept up to date; so don't take it as authoritative''
* 3.
** Added '''TempMidnight''' section for tracking of min/max temperature based on day starting at midnight, to complement tracking starting at rollover time
* 3.17.0 build 3184 (released 23 May 2022)
** Some changes to handling of lighning (regarding when no strikes)
* 3.12.0 beta build 3134 (released 29 July 2021)
** Added '''Lightning''' section (lightning distance and last strike time)
* 3.11.0 build 3129 (released on 7 May 2021)
**Fix: End of day backup now always runs at rollover, so like most releases of the legacy software, the stored file represents the true end of day position.
*** For example in the '''[General]''' section, the ''Date'' (calendar date) and ''Timestamp'' attributes will relate to when it was last updated, but the three '''Current'' attributes will relate to the meteorological date just ended.
*** For the today.ini in the backup/daily folder, last update time-stamp will indicate time as at one update cycle before end of day, but file last modified will show that was at end-of-day
** New: Added (to [Temp]) GrowingDegreeDaysThisYear1=, and GrowingDegreeDaysThisYear2=.
* 3.7.0 - build 3089 (released on 28 July 2020)
** Canadian Humidity Index (Humidex) added for highest extreme monitoring in today.ini (and other longer period extreme log files)
* Release 3.6.6 build 3082 (released on 1 June 2020)
** Values in the file had been stored (in all previous builds) using 15 decimal places (because of difficulty in representing decimal parts of numbers in binary), from this release revised so all values stored using 17 significant figures
* Release 3.6.3 - build 3079 (released on 21 May 2020)
** Fix long standing problem with today.ini becoming corrupted when Microsoft Windows is shutdown
* Release 3.6.0 build 3076 (released on 4 May 2020)
**
* Release 3.1.0
** Added '''Records''' section, this holds one attribute "Alltime" that holds a datetime stamp recording the most recent update to the [[alltime.ini]] file
*From version 1.9.4 build 1089 to release 3.10.5 build 3122
** The '''today.ini''' stored in the daily backup contains the position as at '''start of day''' (or in specified older MX releases, up to one UPDATE interval later), so it does not contain any information (except Cumulative Chill Hours) that relates to day that has just ended.
** It is believed that today.ini may have been introduced at version 1.4
*Version 1.0 27th January 2004 First release
**Included internal tracking of daily extremes, and display on main screen, but not storing in a file.
| |||