Calculate Missing Values - Revision history

HansR: /* Derived spot values */

2022-10-01T11:36:36Z

Derived spot values

← Older revision		Revision as of 11:36, 1 October 2022
Line 166:		Line 166:
	The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.		The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.

	There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature, please see [[MX_Administrative_Interface#Changing_Settings]] and [[Cumulus.ini]] pages for how to find the settings.		There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature, please see the [[MX_Administrative_Interface#Changing_Settings\|interface]] and [[Cumulus.ini]] pages for how to find the settings.

	=Field Count Variations=		=Field Count Variations=

HansR: /* Derived spot values */

2022-10-01T11:34:32Z

Derived spot values

← Older revision		Revision as of 11:34, 1 October 2022
Line 150:		Line 150:
	** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)		** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)

	=Derived spot values=		= Derived spot values =

	Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.		Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.

	Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):		Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):
	* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity		* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity. The original legacy Cumulus 1, and CumulusMX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters\|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by CumulusMX.
		⚫	* '''Wet Bulb''', is not calculated by CumulusMX
	** The original legacy Cumulus, and MX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters\|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by MX.
⚫	* '''Wet Bulb''', is ~~only~~ calculated by ~~Cumulus 1, not MX~~
	* [[Wind_chill\|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.		* [[Wind_chill\|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
	* [[Humidex\|'''Canadian Humidity Index (Humidex)''']], [[Heat_index\|'''USA Heat Index''']], and [[Apparent_temperature\|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)		* [[Humidex\|'''Canadian Humidity Index (Humidex)''']], [[Heat_index\|'''USA Heat Index''']], and [[Apparent_temperature\|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
	** (The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like\|here]]).		** The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like\|here]].
	** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.		** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
	* [[Feels_Like\|'''Feels Like Temperature''']] is calculated by ~~the Cumulus MX flavour~~ only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like\|has varied]] in different releases~~, but use a variation on~~		* [[Feels_Like\|'''Feels Like Temperature''']] is calculated by CumulusMX only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like\|has varied]] in different releases.
	* [[Heat/cold_degree_days_and_Chill_hours\|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)		* [[Heat/cold_degree_days_and_Chill_hours\|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
	**A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.

	The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.		The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.

		⚫	There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature, please see [[MX_Administrative_Interface#Changing_Settings]] and [[Cumulus.ini]] pages for how to find the settings.

⚫	There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.
	* For MX, please see [[MX_Administrative_Interface#Changing_Settings]] and [[Cumulus.ini]] pages for how to find the settings
	* For legacy Cumulus 1, please see [[Cumulus_Screenshots#Station\|Configuration -->> Station]] menu and [[Cumulus.ini_(Cumulus_1)]] pages

	=Field Count Variations=		=Field Count Variations=

Sfws: /* CreateMissing.exe */ Rewritten for 3.20.0 release

2022-08-22T09:01:00Z

CreateMissing.exe: Rewritten for 3.20.0 release

Show changes

Sfws: /* CreateMissing.exe */ That sentence still read wrongly!

2022-04-10T10:33:39Z

CreateMissing.exe: That sentence still read wrongly!

← Older revision		Revision as of 10:33, 10 April 2022
Line 208:		Line 208:
	* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder\|data sub-folder]]		* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder\|data sub-folder]]
	** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once		** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once
	* It ~~adds~~ the following fields to [[Standard_log_files#List of fields in the file\|MMMYYlog.txt]] lines as it works through that file to work out daily extremes (see table below).		* If any of the following fields are not populated in [[Standard_log_files#List of fields in the file\|MMMYYlog.txt]] lines, the utility can populate them as it works through that file to work out daily extremes (see calculations in table below).
	** [[Wind chill\|Wind Chill]]		** [[Wind chill\|Wind Chill]]
	** [[Apparent_temperature\|Australian Apparent Temperature]]		** [[Apparent_temperature\|Australian Apparent Temperature]]
Line 215:		Line 215:
	** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb\|Wet Bulb temperature]]		** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb\|Wet Bulb temperature]]
	** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity\|Dew Point]]		** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity\|Dew Point]]
	** Rain since midnight)
	* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below		* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below

Sfws: /* CreateMissing.exe */ correcting incomplete sentence!

2022-04-10T10:29:22Z

CreateMissing.exe: correcting incomplete sentence!

← Older revision		Revision as of 10:29, 10 April 2022
Line 208:		Line 208:
	* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder\|data sub-folder]]		* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder\|data sub-folder]]
	** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once		** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once
	* It adds the following fields to [[Standard_log_files#List of fields in the file\|MMMYYlog.txt]] lines as it works through that file to work out daily extremes in ~~order to recreate~~ .		* It adds the following fields to [[Standard_log_files#List of fields in the file\|MMMYYlog.txt]] lines as it works through that file to work out daily extremes (see table below).
	** [[Wind chill\|Wind Chill]]		** [[Wind chill\|Wind Chill]]
	** [[Apparent_temperature\|Australian Apparent Temperature]]		** [[Apparent_temperature\|Australian Apparent Temperature]]
Line 603:		Line 603:
	# At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day		# At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day
	\|}		\|}


	==Using a PHP script on your web server==		==Using a PHP script on your web server==

Sfws: Update for post 3.12.0

2022-04-10T10:27:56Z

Update for post 3.12.0

Show changes

Sfws: /* CreateMissing.exe */

2022-04-09T17:56:50Z

CreateMissing.exe

← Older revision		Revision as of 17:56, 9 April 2022
Line 198:		Line 198:
	==CreateMissing.exe==		==CreateMissing.exe==

	This is only available if you already have MX installed and have been using it.		This is only available if you already have MX installed and have been using it. Put simply, this utility updates [[Standard_log_files#Number_of_fields_per_line_varies_by_release\|MMMYYlog.txt]] lines as it works through that file to work out daily extremes in order to recreate [[dayfile.txt]].

	These notes relate to "Create Missing Version 1.1.0" which is for "Cumulus MX release 3.12.0" and above. At the time of amending this section Create Missing version 1.0.2 had been tested and was found to have a bug.		These notes relate to "Create Missing Version 1.1.0" which is for "Cumulus MX release 3.12.0" and above. At the time of amending this section Create Missing version 1.0.2 had been tested and was found to have a bug.
Line 223:		Line 223:
	# options (use zero bearing, use 10 minute wind average, use speed or gust for wind average calculation, fix maximum humidity, read or calculate dew point, read or calculate wind chill, synchronisation time, , set clock option, read or calculate pressure trends, log extra sensors or not, ignore station clock or not, round wind speeds or not, air quality sensor or not, check that minimum number of sensors are updating or not, time for averaging wind bearings, time for averaging wind speed, and time for peak gust)		# options (use zero bearing, use 10 minute wind average, use speed or gust for wind average calculation, fix maximum humidity, read or calculate dew point, read or calculate wind chill, synchronisation time, , set clock option, read or calculate pressure trends, log extra sensors or not, ignore station clock or not, round wind speeds or not, air quality sensor or not, check that minimum number of sensors are updating or not, time for averaging wind bearings, time for averaging wind speed, and time for peak gust)

	The utility program, CreateMissing.exe, uses the same module to calculate derived values (like dew point, wind chill, apparent temperature, feels like, and Humidity Index) from source values as CumulusMX.exe. This is how it is able to insert missing fields in the standard log files. As it processes each line in any standard log file, if a particular field is missing, it will calculate it from other fields in the line if it can. The utility program, CreateMissing.exe, can update the various [[:Category:Ini Files]] with any new extremes found while it is calculating new fields for any standard log file.		The utility program, CreateMissing.exe, uses the same module to calculate derived values (like dew point, wind chill, apparent temperature, feels like, and Humidity Index) from source values as CumulusMX.exe. This is how it is able to [[Standard_log_files#Number_of_fields_per_line_varies_by_release\|insert missing fields in the standard log files]]. As it processes each line in any standard log file, if a particular field is missing, it will calculate it from other fields in the line if it can. The utility program, CreateMissing.exe, can update the various [[:Category:Ini Files]] with any new extremes found while it is calculating new fields for any standard log file.

	The utility program,CreateMissing.exe, looks in the [[Data folder\|data sub-folder]] to see if dayfile.txt exists (if it does, its file name has a '''.sav''' suffix added after the ''.txt''). If a file with .sav suffix already exists, the utility program will stop, as it cannot create a file if it already exists.		The utility program,CreateMissing.exe, looks in the [[Data folder\|data sub-folder]] to see if dayfile.txt exists (if it does, its file name has a '''.sav''' suffix added after the ''.txt''). If a file with .sav suffix already exists, the utility program will stop, as it cannot create a file if it already exists.

Sfws: Text replacement - "Category:MX txt Files" to "Category:Files_with_Comma_Separated_Values"

2021-12-31T21:30:33Z

Text replacement - "Category:MX txt Files" to "Category:Files_with_Comma_Separated_Values"

← Older revision		Revision as of 21:30, 31 December 2021
Line 5:		Line 5:
	* Solving problems when a line was not successfully written to the daily summary log file		* Solving problems when a line was not successfully written to the daily summary log file

	[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:~~MX txt Files~~]][[Category:Cumulus 1]]		[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:Files_with_Comma_Separated_Values]][[Category:Cumulus 1]]

	=How Cumulus Works=		=How Cumulus Works=

Sfws: /* OTHERS????= */

2021-07-15T09:08:54Z

OTHERS????=

← Older revision		Revision as of 09:08, 15 July 2021
Line 69:		Line 69:
	My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data. I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.		My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data. I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.

	==OTHERS????===		==OTHERS????==

	PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.		PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.

			Meanwhile, people should ask for help in the support forum.


	=Some definitions=		=Some definitions=

Sfws: /* Reading archive data */

2021-07-15T04:55:54Z

Reading archive data

← Older revision		Revision as of 04:55, 15 July 2021
Line 25:		Line 25:
	If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.		If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.

	This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can '''rewind''', by stopping Cumulus, optionally replacing files in [[Data folder\|'''data'' folder]] with earlier copies of those files from [[Backup folder\| '''backup''' folder or ''backup/daily'' folder]], and restarting Cumulus. Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.		This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can '''rewind''', by stopping Cumulus, optionally replacing files in [[Data folder\| '''data''' folder]] with earlier copies of those files from [[Backup folder\| '''backup''' folder or ''backup/daily'' folder]], and restarting Cumulus. Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.

	* For Cumulus 1, to stop Cumulus, you select [[Cumulus Screenshots\|Exit]] from the main menu.		* For Cumulus 1, to stop Cumulus, you select [[Cumulus Screenshots\|Exit]] from the main menu.

← Older revision		Revision as of 10:33, 10 April 2022
Line 208:		Line 208:
	* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder\|data sub-folder]]		* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder\|data sub-folder]]
	** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once		** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once
	* It ~~adds~~ the following fields to [[Standard_log_files#List of fields in the file\|MMMYYlog.txt]] lines as it works through that file to work out daily extremes (see table below).		* If any of the following fields are not populated in [[Standard_log_files#List of fields in the file\|MMMYYlog.txt]] lines, the utility can populate them as it works through that file to work out daily extremes (see calculations in table below).
	** [[Wind chill\|Wind Chill]]		** [[Wind chill\|Wind Chill]]
	** [[Apparent_temperature\|Australian Apparent Temperature]]		** [[Apparent_temperature\|Australian Apparent Temperature]]
Line 215:		Line 215:
	** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb\|Wet Bulb temperature]]		** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb\|Wet Bulb temperature]]
	** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity\|Dew Point]]		** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity\|Dew Point]]
	** Rain since midnight)
	* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below		* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below