5,838
edits
Dayfile.txt: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
revised '''Important Rules''': to make them cope with MX and non-windows devices
m (→Using the daily summary log on your web-site: spelling correction) |
(revised '''Important Rules''': to make them cope with MX and non-windows devices) |
||
|
= Introduction =
Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. The figures contained in the file are used for the 'This period' display accessed from the '''View''' menu and to build any graphs based on daily values.
* Cumulus is frequently reading observations from your weather station, but these don't affect the daily log "dayfile.txt" as it is only updated once a day.▼
* There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options when selected from the main Cumulus 1 screen.▼
*
* Cumulus will not mind you accessing the daily log outside its software, except when it needs write access for processing end of day.
* If you do need to correct some rogue data in the log file, first take a copy and work on that copy, because any edits you do could muck up the specific format that Cumulus 1 or MX needs, there is a section on dealing with rogue data below. Only when you are absolutely sure that your edited copy meets all the constraints listed later, should you replace the original.
===
*
▲*Cumulus is frequently reading observations from your weather station.
*Back ups of all the log files mentioned here (not all the log files) are copied to the 'cumulus\backup\daily' folder, a total of 9 sub-folders are retained.▼
▲*It tracks the highs and lows in those observations by comparing read values against those it has stored in [[Today.ini]], updating that file as required.
* Some people take a copy of the local file, after it has been updated and store it on (or file transfer it to) their web server. One way of doing this is [[Upload_Dayfile| described here]].
=== When Cumulus is restarted after a break in running ===
▲*When Cumulus processes the end of the (meteorological) day, the highs, lows and other information in 'today.ini' are used to append a new line into dayfile.txt, then a new 'today.ini' is created initialised with the observations just read from the weather station.
*
▲*When Cumulus is restarted after a break in running,
* Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
▲*#it uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see [[FAQ#Where_does_Cumulus_get_its_this_month_and_this_year_rainfall_totals_from.3F|FAQ]])
* Back ups of all the log files mentioned here (not all the log files) including dayfile.txt are copied to the 'cumulus\backup' folder, the last 8 only are retained.
▲*Back ups of all the files mentioned here are copied to the 'cumulus\backup\daily' folder.
▲*There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options when selected from the main Cumulus 1 screen.
== How you can use the daily log ==
*If you want to run scripts that use the daily log, it is best if you take a copy first, I take a copy that is put onto my web server by using the '''Daily''' box in the bottom left of the ''Sites/Options'' frame within the ''Internet'' options screen from the '''Configuration''' menu to safely take a copy of 'dayfile.txt' after it is updated. See Cumulus '''Help''' for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from my "T:\Cumulus\daily_batch_all.cmd" in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure. My Cumulus 1 batch command also runs PHP to create SQL to update a database table version of the log, but my database table version also contains some additional information from the daily backup '[[today.ini]]' log.
=== Editing the file or other Manipulation outside Cumulus ===
The file is <tt>data\dayfile.txt</tt> within the directory holding the Cumulus
'''Tip''': Take a copy of the file before you
'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end). (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page).
'''Important Rules''':
These notes were written for Cumulus 1,
* The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
*
* All rows must ''start with date'' and include some of the parameters listed ''in correct sequence''.
* The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
* The (meteorological) date format uses ''two digits for the year''.
* (Remember the month must be the middle figure in the date, USA convention cannot apply within logfiles). The separator between the three parts of the date
** If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent. Equally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines. Some third-party tools have to be told what separator you use for dates.
* The fields are separated using the Windows (or whatever operating system you are using for MX) list separator (e.g. a comma or semi-colon)[[File:Open office (editing cumulus log files).png]] If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it.
* Rows can vary in length but only by missing off ''fields at the end''.
* Shorter lines can have multiple field separators added at end of row (during editing after all available valid parameters inserted, extra field separators may be added at end of shorter lines inserted by 'Create Missing' or by a spreadsheet as these make all lines end up with same number of fields)▼
* Most value fields are in ''real number format x.y'' using your system decimal notation, a few (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]). Whilst an integer can be used for a real number field, decimals are not allowed in an integer field.
* If you insert a ''lowest or highest value'' for a new day, where there was no record before, insert a ''time-stamp'' too, as a dayfile.txt row is only accepted by the Cumulus editor if each value has any related time-stamp. (Use a time-stamp of your rollovertime 00:00, 09:00 or 10:00 if you have not looked up the precise time). If you are using a 9am or 10am rollover time, create missing in Cumulus inserts 00:00 for null time-stamps, but normally Cumulus uses the rollover time for null time-stamps.
* Times appearing for some of the fields must always be in ''format HH:mm'' i.e. 2 digit hour, followed by a colon, then 2 digit minutes (Be aware you will have problems if you, or your editing software, add seconds). Except for wind gust (start of line), each time field will immediately follow the value for that parameter.
* Nulls (2 field separators without something between them ',,') are not allowed within the part of the line with values and time-stamps, so if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).▼
▲* Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool. (during editing after all available valid parameters inserted, extra field separators may be added at end of shorter lines inserted by 'Create Missing' or by a spreadsheet as these make all lines end up with same number of fields)
▲* Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
**Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
*** Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
**Some third-party scripts read the file to calculate averages or other statistics, and their authors suggest you remove rogue values (creating the ',,' that Cumulus objects to). You must never do this in the dayfile.txt that Cumulus processes. My suggestion is use the 'External Program' facility to create a copy of ''dayfile.txt''
* The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character.▼
*** Alternatively, do as I do, clean your data as you upload the file into a database. I use a script with validation code checking for rogue values and recognising the -999.9 etc., in all cases my script will store a '''NULL''' value as default in the database table if that validation finds any obviously invalid figure.
* Make sure that any editing does not create any ''blank lines'' in the file.▼
▲* The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. Similar rules apply for terminators used by other operating systems, so be careful if you edit the file on a different device to that running Cumulus.
▲* Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing.
* Don't add a header line to the file, Cumulus expects all lines to be data lines.
=== Editing daily summary in Cumulus ===
| |||
