5,838
edits
Recent History (preserving history): Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
Recent History (preserving history) (view source)
Revision as of 11:35, 8 August 2021
, 8 August 2021Minor resequencing of page
m (→What is Recent History?: typo) |
m (Minor resequencing of page) |
||
|
{{Template:Version badge Mx}}{{Version badge 1}}This Wiki page applies to all Cumulus flavours (including the abandoned Cumulus 2).
[[Category:Terminology]]
=What is Recent History?=
Recent history functionality in Cumulus is the ability to store weather values, at up to 1 minute intervals, for the last 7 days.
It is important to realise, these are spot readings, and will miss any extremes that occur between the times when the recent history values are stored. The weather value that tends to vary the most is wind speed, so that is where peaks are most often missed.
It is also important to realise that weather stations are not supplying values to Cumulus on a continuous basis, depending on the weather station type values may be available at various intervals, either every so many seconds or perhaps at a rate less than once a minute. This can mean that the readings stored on a 1-minute basis may actually be duplicates if the station has not supplied a new value since the previous minute.
=How is the functionality used?=
The normal way to retrieve a specific value is by using [[Webtags#Recent_History|recent history tag names]] and [[Webtags/Parameters#Input_modification_Parameters|specifying how many minutes ago]] you want (to avoid using large numbers, parameters allow you to specify days ago, hours ago and minutes ago in any combination).
You'll get the nearest value if you ask for a time for which there is currently no exact match, and the first tag [[Webtags#Recent_History|listed here]] tells you that nearest time.
There are a number of internal calculations, such as calculating [[Average temperature]], [[Heat/cold degree days and Chill hours]], which use the recent history data:
* Mark Crossley has explicitly said these calculations use the recent history database for MX
* Steve Loft said they are calculated from data at one minute intervals, he did not explicitly say that the legacy software uses recent history data array
When the legacy Cumulus is generating images containing graphs for its [[Customised_templates#The_Standard_Templates_for_Cumulus_1|"Trends.htm"]] web page, Steve Loft did say these use data from recent history array and that not all points on those graphs are at one minute intervals if data is not available for all minutes.
MX uses data from recent history in its [[API_Graph_Data|Application Programming Interface]] for the standard [[MX_Administrative_Interface#Charts]]
MX optionally uses data from recent history (using several different .json files), see [[:Category:JSON_Files#Uploading_data_to_a_web_server_outside_MX]] to make the data available for plotting charts on your web server.
While Cumulus is left running,, every minute:
# Cumulus then works through subsequent table rows (or array elements), as it reads each one, it moves it to the previous (now empty) row (or array element)
# The number of table rows, or array elements, is actually variable, this is because of two reasons:
#
#
=Availability by Release=
The functionality was introduced from version 1.9.3 (beta build 1033 release 10 April 2012), in terms of working while the Cumulus software is running.
The functionality did not work before build 1098,in terms of initialising from archive records read from a station logger when Cumulus is restarted.
It is only from release 3.12.0, that values from a previous session of MX, are available to MX on restarting less that 7 days after previous session closed.
=How are recent history values stored?=
Depends on the release you are running:
* Release 3.12.0 onwards: Held in [[cumulusmx.db|RecentData table in cumulusmx.db]] (uses SQLite), so available outside MX
* Releases 3.0.0 to 3.11.4: Held in a SQLite database table that is stored in-memory within MX code
* Cumulus 2: Was held in an external SQLite database, together with other data that Cumulus 1 had held in text files
* In the legacy Cumulus 1, the values were stored in an array held within the Cumulus code.
=What happens when I need to stop and restart Cumulus?=
Cumulus software has been designed on the assumption that it is left running constantly. However, in practice users do stop and restart the software.
Other people do not want to leave their computer running all the time, and deliberately switch it off when they are not using it for some other purpose.
Earlier, it was described how
Some weather stations have a memory, or separate logging feature, that allows them to store weather data. Cumulus software, for these weather station types, has the option to read these history records from the weather station in its archive data reading process as the software starts. However, it is rare for the weather station to store data every minute, hence the resolution of archive data might be any period between every ten minutes and every 60 minutes, depending on settings.
The recent history data collected while Cumulus is running is lost when you stop Cumulus, unless you are running release 3.12.0 or later, as explained below. If it is retained, obviously it depends how much time has passed with MX stopped, as to how much lies within the 7 day period and can be used after restart.
(There are advantages if the weather station logging uses the same period as is set for the various [[Monthly log files]] that Cumulus can update, and MX does include some optional single use functionality to try to force your weather station to use the same interval).
==MX release 3.12.0 (beta build 3134) onwards==
The recent history is stored in a SQLite3 database table [[cumulusmx.db#Release 3.12.0 onwards|RecentData]] and therefore if you stop MX, the recent history data up to the time MX stopped has become persistent, and is available when MX starts again. Thus the charts, and internal calculations, mentioned above can make use of recent history data from the previous Cumulus session for the period 7 days ago to the time when MX was stopped.
If Cumulus MX is offline for a prolonged period, (and when you first run 3.12.0, as the '''RecentData''' table does not yet exist) then all the data for the previous just over 10 thousand minutes will be at this lower station logging interval resolution, as none will be available from the previous run of MX.
In this release range, MX holds the recent history values in a [https://cumulus.hosiene.co.uk/viewtopic.php?p=100098#p100098 SQLite database], but the database is held '''in-memory''', not as an external database, so it only exists while Cumulus is running. Therefore, if Cumulus stops, all this high resolution data is lost.
When you do restart MX, if your weather station can store historic data, then its logger is read during the restart as archive data for the period from 7 days ago until the time of restart, and obviously only available at the station logging interval resolution of that historic data (be it every 10 minutes, or every 30 minutes or whatever).
The 'recent historical data' is stored in an array stored by the Cumulus code, as explained earlier. This means it is lost when the legacy Cumulus is stopped.
The only way that the legacy Cumulus can build up the array for the period before Cumulus is restarted is if your weather station includes a memory that stores historical readings. I believe some builds had a bug relating to reading this data into the array, but such data read from your weather can be incorporated into the array elements for 7 days ago until when Cumulus was restarted.
=Which weather values are stored?=
For '''Current Conditions''', Cumulus can display values from the basic set of weather sensors it expects, from some derived values, and from various extra sensors, as per table below.
As Cumulus has been developed, more and more of these have been stored for Recent History. Those available in current MX release are listed at [[cumulusmx.db|RecentData table in cumulusmx.db]]. Those available for the legacy software are listed at [[Webtags#Table_of_Recent_History_tag_names_available]]. If your release does not calculate all the derivatives you want, please see [[#If the derivative you want is not available in your Cumulus release]].
{|-
!style="width:150px" | Basic "source" measurements
!style="width:150px" | Measurements derived from source ones
!style="width:150px" | Example extra sensors
|-
| Full details [[Calculate_Missing_Values#Source_value|here]], ''a defined minimum set of source values'':
# Current air temperature
# Current Relative Humidity
# At least one wind speed
# Current air pressure (absolute or sea-level)
| Full list [[Calculate_Missing_Values#Derived_spot_values|here]], for history of some see [[Feels Like|weather derivatives]], examples include:
* Dew Point
* Wind Chill
* Humidex
* Feels Like temperature
* Wet Bulb temperature
| Complex options, include:
* Extra temperatures
* Extra humidities
* Air Quality
* Solar
|}
=Warning when Daylight Saving Time starts or ends=
'''Please note that parameters specify time-stamped array element to retrieve based on counting back from current local time''' so the result for ''any period including when clocks change'' may not be quite what you anticipated.
== Changing from Summer to Winter time==
Remember, ''when clocks go back'', that for a whole hour following the clock change, clock times (now in winter time) repeat the previous clock times (that were in summer time). The way that Cumulus has coded this functionality, instead of the new sets of spot values being added after the old rows/elements, they overwrite the old table rows, or array elements, for the same times. Consequently, you cannot retrieve any values for the times that have been overwritten, and you must be careful when you specify to Cumulus which values you wish to retrieve.
==Changing from Winter to Summer time==
As for ''when the clocks go forward'', the new values do get added after the old rows/elements, but of course there will be one hour's worth of times that simply do not exist. Consequently, any attempt to retrieve values around the time of the clock change has to be careful not to specify any time in that non-existent period, as Cumulus will just return the value before, or the value after, depending on which is nearest.
==Technical note==
As explained earlier, you request an entry from Recent History by specifying 1 to 10 079 minutes ago. Cumulus looks at current clock time, calculates what time would be that long ago, and seeks the nearest datetime entry.
Since SQLite database tables use the row number as the primary key (see [[cumulusmx.db]]), the datetime column does not have to contain a unique value, so technically it would be possible for MX to create rows in the database that have same time as other rows when the clocks go back; equally technically when clocks go forward, it would technically be possible to calculate a revised timestamp so you don't have 60 minute specifiers for which the time never existed.
However, to do this would involve more complicated code, because MX would need to keep track of which row was last updated, keep track of when the clocks changed, and modify its search routine to find the correct row. Life is much easier if the Cumulus user makes allowance for any clock change when making their request.
=If the derivative you want is not available in your Cumulus release=
It was mentioned [[#Which weather values are stored?|earlier]] that more recent history web tag names have been added as Cumulus developed. In many cases, Cumulus has actually started calculating the derivative and using it for charts, in earlier releases than when it becomes available for others to use via web tags.
From 3.12.0, the availability of the database for viewing outside Cumulus, means that anyone who knows how to read a SQLite database can write a script, or program, to access any derivative that is in the database, but not available as a web tag, and can then pass the output to their web site by an appropriate technical process.
Alternatively, although Humidex, 'Apparent Temperature', 'Feels Like temperature', and other derivatives Cumulus can calculate, are not available in all releases, they can be calculated in a script from recent 'outside temperature', 'wind speed', and 'relative humidity' values (using the same time selection for all).
The relevant formulae using JavaScript, you need to adjust them for other languages, for some of these are shown below:
If you are in USA and use Fahrenheit instead of Celsius, you will need to omit the 5/9 term, but as the index is dimensionless no other conversion is needed. This example is for 3 hours ago, change the input modification parameter to suit your need.
H = (5/9 * (<#RecentHumidity h=3> /100 * svp - 10)) + <#RecentOutsideTemp h=3;
Note this apparent temperature formula uses Celsius for temperature and '''metres per second''' for wind speed. You will need to do the appropriate conversions from the quoted recent history tags if you use different units. The Australian Apparent temperature formula is same for Cumulus 1 and MX:
| |||
