Cumulus Wiki

Dayfile.txt: Difference between revisions

Page Discussion

Dayfile.txt (view source)

Revision as of 13:47, 26 January 2021

23,349 bytes removed ,  26 January 2021
m
Updated categories, removed external editing
m (Text replacement - "[[Diags" to "[[Diags_folder")
m (Updated categories, removed external editing)
 
* The [[ImportCumulusFile|article here]] describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have bee edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using.
 
 
 
 
UPDATE BELOW
 
 
 
*There is a section at [[Cumulus_MX#Optional_Sections|Cumulus MX]] to explain how MX includes the ability to generate SQL for creating the database table, for updating it with past data, and to add a new row at the end of the day for the standard database table version of this daily summary log.
 
 
 
 
 
 
 
==== Using that table ====
*Normally if you use any third-party packages like for example "Cumulusutils", the separator used in first line is assumed to be true for all lines.
*Some third-party tools have to be told what separator you use for dates, before they can read your dayfile.txt.
 
= Safely editing the daily summary log =
 
Both Cumulus 1 and 3 (2 did not) now provide editors where you can see what is in your dayfile.txt log, and if you click on a particular line you can edit the fields in that line, or indeed delete that line.
 
If you use these provided editors, then you don't need to worry about the editing rules in the next section. The in-built editors always insert a full line including all fields available in the version of Cumulus you are currently using, so you cannot accidentally insert blank lines. The Cumulus 1 editor also does some validation to ensure the right entry format is used for each field. There is no validation in the MX editor (it was set up relatively quickly in version 3.4.5 as the first of 3 log file editors to plug a gap in MX functionality in earlier versions), but all lines are passed back via an application programme interface to the MX engine which might raise an error when it can't replace the line in the logging file.
 
== Important Rules when editing dayfile.txt ==
 
If you are editing dayfile.txt outside the Cumulus 1 or MX software, there is the danger of changing something that prevents Cumulus from understanding the file when it next tries to use it.
 
=== General Editing Rules ===
 
# Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
# Take another copy and use that for your editing, don't edit the actual file being used by the software.
#*This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
#*It also means that you can go back to the last working copy, you can't upset your "revert" copy.
#The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
#* It is recommended that you use either a specialised "Comma Separated Value" file editor or a text editor, both of these can be easily used.
#** These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
#*You can use a spreadsheet tool, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
#**If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
#*Don't remove any figures from fields where figures currently exist, only change one entry into another entry in same format.
#Cumulus does not accept the concept of nulls, there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted.
#All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
# Make sure that any editing does not create any ''blank lines'' in the file. Cumulus assumes an empty line means end of processing.
# Don't add a header line to the file, Cumulus expects all lines to be data lines.
 
=== File specific Editing Rules ===
 
[[File:Open office (editing cumulus log files).png| right]]
# The file should be saved without "Byte Order Mark", specialised text editors will include a menu where you select the encoding and can select not to include BOM.
# All rows must ''start with date'' and include at least 14 further fields ''in correct sequence''.
# The (meteorological) date format uses ''two digits for the year''.
#*This is one reason why you need to edit this file using an editor that treats all fields as text (a text editor, a CSV editor, or a spreadsheet program that can be instructed ''not'' to recognise special field types).
#*For spreadsheet tools (e.g. '''Calc''' in Libre Office, or on Microsoft Windows '''Excel''') avoid using default of recognising formats, ensure that such recognition is turned off (see image), as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to change it to four figure years, and then Cumulus will no longer be able to use the log file.
#Remember the month must be the middle figure in the date, USA convention cannot apply within this logfile.
#The separator between the three parts of the date should be a '-' hyphen or a '/' slash, it cannot be a space.
#*Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus.
#*Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
#* If you move your software to a new device, or you change from Cumulus 1 to Cumulus MX (or back), then you must ensure your dates still use the same separator, so all lines are consistent.
# Each of the fields from date to the end of the line are separated using the list separator (e.g. a comma or semi-colon) defined for your device. After your editing it must still match what your existing dayfile.txt uses.
#* If you wish to use Excel, or to use "Calc" in 'Apache Open Office', "Libre Office", or similar, you may on opening the file need to pre-select the field separator that is being used now (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" (or whatever similar feature name your tool uses) unselected. Again third party packages processing dayfile.txt will need to recognise your field separator, and some may need to specify it. Don't forget to also select it when you save the edited file (you probably need to select "save as" or the equivalent in your tool to see the option).
# Rows can vary in length but only by missing off ''fields at the end''. The minimum number of fields after the date is 14, the maximum varies between different versions.
#Each field has a pre-defined format, and the same format must always be used in that field position.
#No fields will accept letters.
#*Some fields (e.g. bearings, solar, humidity) are ''integers'' (see [[#List_of_fields_in_the_file]]) only take integers. Decimals are not allowed in an integer field, so no comma or full-stop can be within these fields.
#* Most value fields are in ''real number format'' using your system/locale decimal notation ("x.y" or "x,y"). Trailing zeroes are not required, so you can put an integer in a real number field, you don't have to have a decimal comma or decimal point.
#Although only the date and 14 other fields are mandatory, you cannot skip some fields defaulting them to null is not allowed, so you cannot add fields at the end, without adding all earlier fields.
#when you do add fields beyond the 14, or however many already exist, be aware that for most derivatives what you add will represent a ''lowest or highest value'' and that must be paired with a time-stamp in the next field.
#*Cumulus will only accept highest/lowest figures if each value has any related time-stamp.
#Time stamp fields must always be in ''format HH:mm'' i.e. 2 digit hour in 24-hour format, followed by a colon, then 2 digit minutes
#**Be aware you will have problems if you, or your editing software, add seconds.
#*If when editing, you don't know what time to quote, the convention is to use a time-stamp of your roll over time i.e. 00:00, 09:00, or 10:00, if you have not looked up the precise time.
#* Except for wind gust (start of line) where an extra field is fitted in, each time field will immediately follow the value field for that parameter.
# Shorter lines can have multiple field separators added at end of row added either when editing within Cumulus or when editing using a spreadsheet tool.
#* Nulls (2 field separators without something between them ',,') are thus allowed at end of line, but are not allowed within the part of the line with values and time-stamps.
#*If you are editing out rogue values and if you do not know the value for a particular field within the line, then type in a zero or 9999 for nulls in integer format and an extreme with opposite value (e.g. -999.9 for a signed decimal maximum, and 9999.9 for a decimal minimum) for nulls in decimal format (replace the full stops with your decimal separator).
#*Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
#Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and will repeat the last valid value if the station fails to send a value it should provide, so if a station fails to send a value for more than a day, dayfile.txt may show the same value as the previous day.
#* Note that Cumulus will stop if your station fails to send what it considers as a vital reading, like pressure or temperature, so the previous point does not apply in all cases.
# The row terminator for Windows is ''CR LF'', ensure any external editor does not change the two character terminator into a single character. Similar rules apply for single character terminators used by other operating systems, don't let windows change a single character terminator into its double character terminator.
#*Problems with terminating characters are intercepted by operating system, before it reaches the software, but may still stop the software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.
 
 
 
=== Dealing with errors identified by the software ===
 
If there is an error in ''dayfile.txt'', then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus 1 will illuminate its ''Error'' light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.
 
If you do have a 'duplicate' error, you need to decide which row to ''delete'', and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.
 
=== Editing externally ===
 
See earlier section on this page for the rules and later section for advice on dealing with rogue values.
 
=== Using the Cumulus 1 editing feature ===
 
{{Version badge 1}}
'''This section applies to Cumulus 1.x.y only'''. The last command in '''Edit''' [[Cumulus_Screenshots#File.2FEdit.2FHelp_Menu |menu]] is ''dayfile.txt''. ''This is how you view'' the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. Click the ''Help'' button for detailed instructions. Cumulus Help is comprehensive.
 
You can use this Cumulus 1 editor as follows:
*use '''insert''' key to add one or more missing rows (complete days) manually,
*correct individual values by over-typing,
*use '''delete''' key to remove an entire day (e.g. if you get a 'duplicate' error message),
*use '''Create missing''' button to insert missing rows (complete days) by reading from [[Standard_log_files]] and automatically calculating the best approximations for each field for those missing days.
 
==== Create Missing ====
 
The Cumulus 1 editor provides a "Create Missing" option where it will, for any dates for which a line does not exist, create a line if it can from reading the [[Monthly log files|detailed log file]] to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation for each dayfile.txt field, storing the time from the relevant other log file in any time-stamp field in dayfile.txt. If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an extra row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occurred at moments in-between those that were logged. For ''Create missing'' a list of inserted records is produced in [[dayfileeditlog.txt]]. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | work around]] as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the [[Standard_log_files]] if the whole day (a line starting with that date) is missing in ''dayfile.txt''. Although Cumulus does not recognise the concept of a sensor not being available, it will write solar information even if you don't have a solar sensor; it does have to cope with reading a monthly log file that might have fewer derivatives than it wants (when using Create Missing) and therefore it may not know what to write into dayfile.txt as the calculated value. Cumulus 1 can't write a null value, so it writes zero for values, and "00:00" for time stamps. If you are using a 9am or 10am rollover time, be aware that create missing in Cumulus 1 always inserts 00:00 for null time-stamps, but in normal running Cumulus uses the rollover time for null time-stamps.
 
==== Using Standard logs to deal with shorter (or incomplete) dayfile.txt records for particular dates ====
 
'''WORKAROUND if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''
 
Example: '''add extra fields to records created by an earlier version of Cumulus''' [to help you, the versions (not builds) at which fields were added are indicated below].
Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files.
 
''One method '''for Cumulus 1 only''' is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use ''create missing'' in the [[dayfile.txt]] selection in the '''Edit''' menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).''
 
'''Cumulus MX'''
Although there is a tentative plan for an executable to be added to the MX package that can read the standard logs and calculate all the fields needed for a line in dayfile.txt, this is only a gleam in the developer's eye!
 
The only method available at the time of writing involves multiple steps:
#*Either use '''ExportMySQL.exe monthly''' to create a database table called "monthly" (you can call it another name, but you must have set name in MX, before you run ExportMySQL.exe), and read all the standard logs into that database table
#*Or if you already have the table and have MX updating the standard logs into that database table
#Then, you can fairly easily update to add any missing values into existing columns in this monthly table, and from it update columns in a daily summary table. See [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] where I describe two PHP scripts designed for this type of task.
 
=== Importing data not recorded by Cumulus ===
 
As an alternative to manual line ''insert'', in the Cumulus 1 editor, you may wish to use a procedure for importing, and processing, pre-Cumulus observations into [[Standard_log_files]]. Once there is data for required days in monthly logs, ''Create missing'' can insert the new rows for those days previously missing in dayfile.txt.
 
== Cumulus 2 ==
 
Pick Edit, click that and an editing dialog pops up, that does not let you change the line number nor the date, but all other fields show their current contents and you can overtype as necessary. Scroll down to see 2 buttons (how they are labelled depends on which version you are using), the left button ignores any edits you have made (it is labelled 'Close' or "Cancel" and simply does same effect as clicking the "x" in the top right corner), and the right hand button saves your changes (even if it is labelled 'Edit' rather than "Save" in the version you are using).
 
=== Editing externally ===
 
See earlier section on this page for the rules and later section for advice on dealing with rogue values. Remember if you edit this file externally, either do not add any missing fields, or if you do then these must remain empty for every extra field in every line that was originally shorter!
 
MX does not have the capability to recalculate, the values and time-stamp for any entry in dayfile.txt, by reading the [[Monthly log files|detailed log file]] to extract all values relevant to that day and do the necessary minimum/maximum/total/average calculation. It can be done in a script if you are a programmer. I have done this, but it was not as easy as I thought, the log file you read from may have duplicate times potentially making the calculation harder, and you cannot edit the file as you go along, you basically rename the old file and create a new file with original name as you work through all days changing individual fields whenever you need to.
 
== Editing the file or other Manipulation outside Cumulus ==
 
{{Version badge 1}}Apart from bulk changes, Cumulus 1 made editing this log file quite easy. Although the create missing did not work where a line was present for the date, but an individual field needed to be corrected or populated, there was a [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates |work around]] for this.
 
[[File:Badge vMx.png]]The beta versions of MX did not include any feature to view or edit dayfile.txt, so any checking or editing had to take place outside the software. As mentioned above, MX does not provide any way to read, or calculate, values from any other file, nor will it do bulk edits. For any of these, you will need to take a copy of the Cumulus file and manipulate using a script or a spreadsheet processor. Do be careful not to change the encoding, the position of line feeds, the format of any field, the locale, or anything else that might stop MX being able to read the file.
 
 
The file is <tt>data\dayfile.txt</tt> within the directory holding the Cumulus executable, it can be viewed in a text editor, imported into various database systems, or imported into spreadsheets, to manipulate as you wish. Just remember that Cumulus reads it when it is restarted, and updates it as part of the rollover process, so never attempt to work on it either when Cumulus has just been restarted and is checking/updating (and possibly doing a rollover of logs), or around the midnight/9am/10am local rollover time when Cumulus is writing a new row.
 
'''Tip''': Take a copy of the file before you do any edit outside Cumulus, so you can revert to old file.
 
'''Note''': Since new versions/builds can add to number of fields, Cumulus will accept lines of 15 fields or more (including without the more recent fields at the end). (Additions by versions are indicated below, you can explore details of earlier versions via the official [Software software/download] page).
 
=== Dealing with rogue measurements ===
Cumulus provides via '''Configuration''' menu ''Calibration'' screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes.
 
If you discover a rogue measurement (perhaps the wind affected your tipping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see [[today.ini]] or a [[FAQ]] for further advice. In general, you need to stop Cumulus, edit the monthly log row containing the dodgy values, edit 'today.ini' and possibly other '.ini' files, looking up the logs indicated below that show the updates with previous high or low.
 
If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time. As your first step you should update the appropriate field in the row for the affected date in dayfile.txt. Once ''dayfile.txt is correct'' the Cumulus editors will allow you to:
* update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. See [[Alltimelog.txt]] for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it.
* update the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.
* update the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags_folder]] for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago.
* update the highs and lows month-by-month in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows.
* '' '''Note''' in each of above 4 editing screens you can:
*# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp.
*# load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and
*# optionally ''Copy'' (row by row) the logged values (and associated date/time information) into the relevant .ini file.
*# click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''.
*# store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel'').
(Each of these screens is a text editor, and works best when at full screen).''
* create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report.
Ideally, you will subsequently try to edit the rogue data for the particular time it was logged; see [[Standard_log_files#Correcting_any_logged_data_problems]], but correcting the daily summary in dayfile.txt must always be the priority.
 
== Using the daily summary log on your web-site ==
|colspan="4" style="background:lightblue;"|Above here represents the minimum length for every line, a count of 15 items
|-
|colspan="4" style="background:pink;"|([There is no record of which version added this next field. The Cumulus Support Forum, while it iswas almosthosted certainlyby oneSteve ofLoft, themoved versionsto innew forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the releasecontent historyof each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 thatas wasthese lostdo not appear in the release history issued by Steve Loft. The first mention of it in the new forum relateswas tonot auntil questionDecember re2008, thebut 1.8.2that releasewas confirmingnot about when it was presentreleased. byA Decemberweb 2008,tag butfor thosethis releasevariable noteswas don'tadded mentionin itBuild being978 introducedof in1.9.1 thatbeta, releasewhich was obviously long after it was first calculated.
 
Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment)]
|-
|16
</pre>
 
[[Category:Log Files]][[Category:Cumulus 1]][[Category:Cumulus MX]]
Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/7640"