5,838
edits
Amending dayfile: Difference between revisions
m
Updating for better coverage of legacy and MX differences
mNo edit summary |
m (Updating for better coverage of legacy and MX differences) |
||
|
[[Category:Files_with_Comma_Separated_Values]]
[[Category:Cumulus 1]]
[[Category:Cumulus Files]]
=How Cumulus uses the daily summary log=
#It will derive totals, averages, highs, and lows, from the data it reads, for each missing date
#*Note that normally dayfile.txt lines are created from [[Today.ini]] which logs the daily totals, averages, highs, and lows, from every reading taken from the weather station
#*Depending on your weather station, Cumulus is able to read values at least every minute (maybe every 10 seconds), and consequently update [[today.ini]] frequently (each minute in Legacy Cumulus/each logging interval in MX) if an extreme happens,
#*If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th (maybe 1/120th depending on weather station reading frequency) of the data,
#* Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence why getting missing lines from a backup is better)
==Legacy Workaround==
The legacy Cumulus 1 functionality is not designed to insert (or correct) individual missing extreme figures for daily summary lines that already have some fields in them, only to insert complete missing lines. However, we can workaround that constraint:
'''WORKAROUND FOR DAYFILE.TXT if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
# Click the ''Create Missing'' button
# Wait while Cumulus scans all the standard log files, and for each source or derived field in those files, tracks the highest and lowest for each meteorological day that is now missing in dayfile.txt.
#* As an aside, be aware that when the contents of [[today.ini]] was used to create a new line in dayfile.txt originally, the high and low that were copied across were derived from all source values that Cumulus obtained from your weather station (this will usually be at least once a minute, depending on your weather station type
#*Continuing this aside, the high and low that Cumulus is now deriving are based purely on those values stored in the [[Standard log files|standard data log]], by default that is just every 10 minutes, but it might be only every 30 minutes.
#The new lines that Cumulus 1 adds to your dayfile.txt while you use the editor, are also added to another log file '''dayfileeditlog.txt''', so we can access that to
#For simplicity, click '''OK''' to save the edited '''dayfile.txt''' file, and exit the editor.
#Rename the amended dayfile.txt as "dayfile(generated).txt".
#*You can use an editor designed for computer programmers or developers that can edit text file
#*You can use a spreadsheet application, such as "Libre Office Calc" or "Microsoft Excel", providing you obey the editing rules defined later
#
#
#Finally, copy your temporary "dayfile.txt" into the "Cumulus\data" folder, so from now on Cumulus accesses the file with the maximum number of fields present.
One note of caution:
*If a required source or derived value is not present in the standard data log lines, Cumulus will write into related dayfile.txt fields zero for values, and in the editor "00:00" (in normal running it would insert your rollover time) for time stamps, because it does not understand the concept of "Null".
*Cumulus 1 does not have the functionality to retrospectively calculate derived values (e.g. cannot work out apparent temperature) when it is reading source values (e.g. temperature, humidity, wind speed) in the standard data log line.
#*Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
#Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
#All figures must be within the range of sensible figures for that field (e.g. no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in
# Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing. Also ensure all lines have same end of line characters, this is one way accidental blank lines can be inserted!
# Don't
=== File specific Editing Rules ===
[[File:Open office (editing cumulus log files).png| right]]
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM. (If you are using old software, both Microsoft and Google editors, used to insert BOM automatically).
# All rows must ''start with date'' as a text field (not a date formatted field) and include at least 14 further fields ''in correct sequence''.
# Remember the month must be the middle figure in the text field representing date, USA date convention cannot apply within this logfile.
# The (meteorological) date format uses ''two digits for the year''. ▼
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types). ▼
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or Microsoft's '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file. ▼
▲#Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile.
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
#* If you move your software (any flavour) to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
▲#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types).
▲#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or Microsoft's '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years,
# Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
#* If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
# Rows can vary in length but only by missing off ''fields at the end''. The minimum number of fields after the date is 14, the maximum varies between different
#* (The variation between maximum number of fields may cause a problem, if you regress to an earlier release!)
# Each field has a pre-defined format, and the same format must always be used in that field position.
#No fields will accept letters.
#*Some fields (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
#* Most value fields are in ''real number format'' using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
# Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
#when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a ''lowest or highest value'' and that must be paired with a time-stamp in the next field.
#* Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
# Time stamp fields must always be in ''format HH:mm'' i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
#**Be aware you will have problems if you, or your editing software, add seconds.
#* If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
#* Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
# Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
| |||