Standard log files: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
m (Release 3.9.0 field 30 section corrected as change delayed)
 
(62 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{Template:Version badge Mx}}{{Version badge 1}}[[Category:Cumulus Files]][[Category:Files_with_Comma_Separated_Values]]
This page applies to all flavours.

= Introduction =
= Introduction =


Cumulus has ability to log data on a periodic basis, see [[Monthly log files]]:
This article is about the standard logging file that stores the full set of current (read and derived) values at whatever interval you have set for Cumulus to keep detailed logging. The default time between data logging lines is every 10 minutes, but you can change this:
* This page is about the standard logging file, that is always created, with a new file in each new month
*Cumulus 1: From Configuration menu, select the [[Cumulus_Screenshots#Station|station settings screen]], '''Data log interval''' box on extreme RH side.
* Steve Loft started work, for Cumulus 2 on a replacement for these files using a database, and planned [[Cumulusmx.db]] to hold such data for MX 3.0.0 beta, although his work on this was never completed.
*Cumulus MX: From settings tab, select the '''Station settings''' screen, and find '''Log interval''' after the ''Ecowitt GW1000 Settings'' section.
* Other files ('''[[Speciallog.txt]]''' , [[Extra_Sensor_Files]], and [[Air_Link_Log.txt]]) are optional, in that settings determine whether they are created or not.


In fact, in Cumulus MX, the processing of this extra sensor logging file shares a lot of the code with the processing of the standard logging file, as they both are updated at same interval, and they both take values processed following a read from your weather station. The applications programming interface (api) that sends values between the MX engine and the MX admin interface is also identical (with a parameter "Extra" that swaps between true and false depending on the log file involved).
These files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is often called <tt>Aug09log.txt</tt>), but the month can be in another language or a different abbreviation depending on your locale. There is no setting to change this format.


The logging file, [[speciallog.txt]], for detailed logging of internal temperature and humidity is not split into separate files for each month. Although this exists in Cumulus 1, it appears this functionality will remain missing from MX.
The standard logging file is in comma-separated format with one line per entry. Being comma separated does not mean the fields in this log are always separated by commas, they might use semi-colon (;) for example.


The logging file for [[Air Link Log.txt|air link]] sensors, is only available in MX.
There is an equivalent logging file for [[Extra_Sensor_Files|extra sensor]] values. In fact, in Cumulus MX, the processing of this extra sensor logging file shares a lot of the code with the processing of the standard logging file, as they both are updated at same interval, and they both take values processed following a read from your weather station. The applications programming interface (api) that sends values between the MX engine and the MX admin interface is also identical (with a parameter "Extra" that swaps between true and false depending on the log file involved).


=How often is data stored in the standard file?=
There is another logging file, [[speciallog.txt]], for detailed logging of internal temperature and humidity, but it is not split into separate files for each month. Although this exists in Cumulus 1, it appears this functionality is missing from MX.


These store the full set of current (read and derived) values at whatever interval you have set for Cumulus to keep detailed logging.
{{TOCright}}
* The interval has to be a fraction of an hour, so the least often storing is every half hour.
* For some weather station types, you MUST match the interval at which the weather station logs data internally
* The default time interval between adding data logging lines is every 10 minutes, but you can change this:
** Legacy Cumulus 1: On main screen, select '''Configuration''' menu, select the [[Cumulus_Screenshots#Station|'''station settings''' screen]], look for '''Data log interval''' box on extreme RH side.
** Cumulus MX: From [[MX Administrative Interface|Interface]], find settings menu, select the '''Station settings''' page, select '''General Settings''', and find '''Log interval''' where there is a drop down that offers the various timings that are permitted.

=What name is used for the standard file?=

These files are automatically created with names in the form <tt><Month><Year>log.txt</tt>.

In UK English locale, the file for August 2009 is called <code>Aug09log.txt</code>.

The month can be in another language, or a different abbreviation, depending on your locale.

The same naming format must be used for all files,if Cumulus is to be able to find them. For Cumulus MX only, any file lines for dates before the Cumulus start date will be ignored.


= File content =
= File content =
Line 20: Line 39:


All data is logged in the units as selected by the user:
All data is logged in the units as selected by the user:
*Cumulus 1: From Configuration menu, select the [[Cumulus_Screenshots#Station|station settings screen]], '''Units''' box is just below the centre of that screen.
*Cumulus 1: On main screen, select '''Configuration''' menu, select the [[Cumulus_Screenshots#Station|'''station settings''' screen]], '''Units''' box is just below the centre of that screen.
*Cumulus MX: From settings tab, select the '''Station settings''' screen, and find '''Units''' section at the start of that screen.
*Cumulus MX: From [[MX Administrative Interface|Interface]], find settings menu, select the '''Station settings''' page, and find '''Units''' section.
** Advanced settings in MX allow you to vary the number of decimal places for the various stored values



Each line has a number of fields separated by whatever symbol is defined as the list separator on your device, typically a comma (,) or a semi-colon (;).
Each line has a number of fields separated by whatever symbol is defined as the list separator on your device, typically a comma (,) or a semi-colon (;).
* The various fields are all listed later in this article, but they can be of various types:
* The various fields are all listed after this section, but they can be of various types:
** Date, this is always the first field, and it always uses a fixed format, ignoring the format defined in your locale.
** Date, this is always the first field, and it always uses a fixed format, ignoring the format defined in your locale.
** Time, this always the second field, and is always in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). However, if you use the feature in MX to upload the equivalent of this file to a database table, the automatic update upload will add ":00" to include seconds.
** Time, this always the second field, and is always in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). However, if you use the feature in MX to upload the equivalent of this file to a database table, the automatic update upload will add ":00" to include seconds.
***Note that it is perfectly possible to have two successive lines in a standard log file with same time, although this should not happen in normal running, it can result from a hiccup or other interference on your device. If you are asking MX to upload the contents of this file, a second line with same time is ignored, only the first line with a particular date and time is uploaded.
***Note that it is perfectly possible to have two successive lines in a standard log file with same time, although this should not happen in normal running, it can result from a hiccup or other interference on your device. If you are asking MX to upload the contents of this file, a second line with same time is ignored, only the first line with a particular date and time is uploaded.
** All Remaining fields are spot values, either what has just been read from your station, or a derived value calculated from a set of values read from your station. For these value fields, Cumulus may have converted from the unit used by your weather station to the unit that you have requested to use, and there will be some inaccuracy inevitable in such conversions. Where the value to be stored is a real number with decimal places, Cumulus will use a decimal comma or decimal point depending on which is defined in your locale, do be careful not to change that locale if you need to change your device or update it, as Cumulus can only read files where the same decimal separator is used everywhere.
** All Remaining fields are spot values, either what has just been read from your station, or a derived value calculated from a set of values read from your station. For these value fields, Cumulus may have converted from the unit used by your weather station to the unit that you have requested to use, and there will be some inaccuracy inevitable in such conversions. Where the value to be stored is a real number with decimal places, Cumulus will use a decimal comma or decimal point depending on which is defined in your locale, do be careful not to change that locale if you need to change your device or update it, as Cumulus can only read files where the same decimal separator is used everywhere.



For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from [[Software#Resources]] page in this Wiki.
For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from [[Software#Resources]] page in this Wiki.


=List of fields in the file=
= Accessing the Standard Log File =


The table is split by Cumulus version for all changes since version 1.7.x (Steve Loft did not retain his list of changes in 1.4, his '''changes.txt''' is rather sparse in details for many releases, and the support forum excludes any posts before 2 Jun 2008, so earlier changes are not now documented).
== Access to current log by other processes ==


The table shows:
As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus applies an exclusive lock, and (as explained further below) conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while Cumulus is running.
#The field number
#*When this list was first produced, only the original Cumulus software was available and the decision then was to start with date in field zero to be consistent with index used for arrays in programming languages like JavaScript.
Also note these log files do not include a header line, and should not be edited to include it.
#*In the data log editing page of the MX admin interface, and internally in its code, MX uses field zero for the line number, and therefore the table below has been edited so that date moves to field 1.

#* For each field the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc is also shown in first column.
The appropriate header file for the standard data logs can be found for any installed version of Cumulus by looking for "monthlyfileheader.txt", in the directory above the 'data' directory containing the standard log. The viewing screens for the two flavours, as described next, automatically include the header line above the tables they show.
#As a quick guide to which fields can take signs and which can not accept decimal places, the second column indicates formatting rules.

#*Any item labelled integer will not accept any decimal places.
A full discussion of the problems with conflicts of access to the standard log file can be found [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 in this support forum topic].
#*Any item labelled number can in Cumulus 1 take integer or decimal values, but in MX the decimal places are mandatory.

#*Any item labelled unsigned will not accept a minus sign. Those labelled signed don't need '''+''' for positive, but do need '''-''' for negative.
== Viewing the standard log file in Cumulus 1 ==
#The third column shows an example of a value accepted by Cumulus 1

#*The file content rules for MX changed in release 3.7.0, and it has not been verified that all Cumulus 1 examples are still accepted
In the '''View''' menu of Cumulus 1, select ''Data logs'', click '''Load...'''.
#* For Cumulus 1, where decimals are allowed, the stored value might be an integer as this depends on a number of factors, including your station type.

#*If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures
The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory.
#* In Cumulus 1 it is optional to use floating point for the others

#* In Cumulus MX add a ".0" if necessary because MX must see decimal places where these are allowed
Select the required month for which to display the standard log, click '''Open''' and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen. Cumulus 1 does not provide an editing facility for this file.
#* Note that the figure in a field varies depending on the units you select, and your weather station may not have all the sensors needed
#*In Cumulus 1 any fields defined in the version you are using for sensors you do not have will always contain zero.
#*Also, for Cumulus 1 any fields that are not defined at the version when the line was created can:
#**Either be missing (no fields seen after the last field at your version)
#**Or can be null (field separators only are seen after the last field at your version)
#* MX since version 3.7.0 is more fussy, but less consistent, certain fields contain a single space when not used, others contain zero, and others can be left empty
#The final column describes the observation often with a link to where there is more information.


==Number of fields per line varies by release==
=== Problems ===


The fields that are available in the release you are running are listed in [[Monthlyfileheader.txt]] (found in folder level above the data sub-folder).
If no logs seen, see [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]]. Also note, this '''Data logs''' viewer is only for viewing standard log files (not any other files in the same ''data'' directory), the column headings for standard (monthly) log files will always be shown at the top of the screen.


If you have some early files with fewer fields than the latest files, there are limited possibilities for calculating correct values for those missing fields


[[File:Badge v1.png]] For the legacy Cumulus software, the number of fields in the file increased as the software was developed. The Cumulus author, Steve Loft, did not provide any utility for populating fields missing from earlier versions of his software. A third party did produce a [[Calculate_Missing_Values#Using_a_PHP_script_on_your_web_server|PHP script]] that could:
=== Editing for Cumulus 1 users ===
* add Heat Index to lines created prior to 1.7.11
* add Apparent Temperature (not other fields, see below) to lines created before 1.9.1
* add Feels Like temperature (not other fields, see below) to lines created before 3.6.0
The fields that ''cannot'' be calculated by this script are (set them all to zero when editing file manually):
* UV Index
* Solar Radiation
* Evapotranspiration (spot value and annual cumulative total)
* Current theoretical max solar radiation
* Hours of sunshine so far today
* Current Wind bearing
* RG-11 rain today
* Total Rainfall since midnight (this is for 0900/1000 'rollover' users; normally same as rain today for 'midnight rollover' users)


[[File:Badge vMx.png]] As MX has developed, further fields have been added to the file at 3.6.0 and 3.6.12:
Cumulus 1 allows you to view any standard log file, but not to edit the contents. Consequently for Cumulus 1, you need to edit standard log files outside Cumulus 1, subject to the following rules:
* Feels Like temperature
* You can't edit the current log file while Cumulus 1 is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors.
* Canadian Humidity Index (Humidex)
* You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand.
The main MX developer, Mark Crossley, has written a [[Calculate_Missing_Values#CreateMissing.exe|Create Missing utility]], this is dependent on having a significantly recent MX release, but it can calculate the following derived values (coping with cases where any of these are missing:
* You can use either a specialised comma separated value file editor or a text editor.
* [[Wind chill|Wind Chill]]
* Text editors designed for programmers will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM).
* [[Apparent_temperature|Australian Apparent Temperature]]
* If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected. Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ...").
* [[Feels Like|Feels Like temperature]]
* If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer).
* [[Heat index|North American Heat Index]]
* Ensure no blank lines, and if you have added in the column headings then remove that heading line when saving ready for Cumulus.
* [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
You can see [[#Manipulation outside Cumulus]] below for more information.
* [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
* Rain since midnight)
''Please note the developer does not mention any of this at [https://github.com/cumulusmx/CreateMissing/blob/master/README.md his utility details explanation] so the author of this Wiki update cannot guarantee the above list is correct''


== The viewing and editing web page in admin interface of Cumulus MX ==


You will need to check in MX release announcements for any update, at time of typing this, the developer has written a routine within CumulusMX.exe to calculate Evapotranspiration, but that is not in the CreateMissing.exe Utility.
In the admin interface, there is a "Data Logs" tab, from it select ''Data logs''.
*On this screen,you see a box for selecting the log you want to display. The default month and year (shown on loading the web page) is taken from yesterday's date. No log is loaded at start up. You can '''either''' type in a period (one or two digit representing month, then hyphen, then four digits representing year) '''or''' select year then month in the data picker (this is produced by the bootstrap software MX uses, it is not the date picker provided by your browser), when you click on the month the box is updated with selected period.
* Next to it is a "Load" button. Press it and the Cumulus MX engine will generate an application programming interface (api) table that will appear on the web page using dataTables software. There is a "Refresh" button, for when you want that api to refresh the table (without reloading the whole page).


=== Pagination ===
===Before Version 1.7.10:===

The table generated is paginated, it shows ten lines at a time, and there are arrow icons to move to next, and back to previous page.

=== Deleting or Editing a line ===

==== The Edit Modal ====

When you select a line, you are in the dataTables software, but the AltEditor software now kicks in generating what it calls a '''Modal'''. For our purposes, it is probably more friendly to describe it as three buttons and a pop up dialog.

Before you select the line, only the '''Refresh''' button is enabled. That basically just reloads the dataTables part of the web page without a need to reload the whole web page. When you select a line, two extra buttons (for editing that line and for deleting that line) are enabled:
#'''Edit a line''' button - brings up a dialog with a HTML form
#* You will see you can change the contents of most individual fields as all values are shown in HTML form input fields.
#*Scroll down and in the footer are two buttons:
##''Save Edit'' will send the edited line back to the MX engine where the log file is then read into an array, and the relevant array element is replaced by the line received. After that the log file is overwritten from the amended array. After that the log file as a whole is converted back to dataTables format and returned by the api that delivered the table originally.
##''Close dialog''. You can close this dialog in 3 ways:
###clicking that Close dialog button
###Clicking the small '''x''' at the top right
### Clicking anywhere outside the dialog
#'''Delete a line''' button - brings up a dialog with the current contents of each field shown as a simple list
#*Scroll down to the footer and find two buttons:
##'''Delete''' - that confirms that you want the api to send back to the mX engine a message delete this line
##'''Close dialog'''. You can close this dialog in 3 ways:
###clicking that Close dialog button
###Clicking the small '''x''' at the top right
### Clicking anywhere outside the dialog

There is no need to say any more about deleting a line, but editing a line does deserve a full description in the following sections, although the interface is such you can probably work out what to do without reading what follows!

==== Derived Fields ====
It must be remembered that the standard log file contains some fields that are directly read from your weather station and some that are derived from those readings. Consequently, if you edit:
#Temperature
#Humidity
#Wind speed
#Rainfall so far

then there is a need to make consequent change to some of these:
*Dew point
*Wind chill
*Apparent temperature
*Heat index
*Humidex
*Rain counter
*Feels like temperature

==== First time you view a line to edit ====

*The first time you bring up the edit modal, it shows all the fields with exactly the same content as was in the line you selected.[[File:Edit stnd log line.png|right]]
*All the fields have the content converted to strings for display purposes. This allows the display of the contents whether a decimal point or a decimal comma is used for those fields where decimal places are permitted
*A few fields are read only:
*#The line number cannot be changed as this is part of how the MX engine recognises the line when the edit is saved and sent back via an application programming interface.
*#The date and time cannot be changed as they record when the readings were made. This editor is not designed to deal with clock problems, the time the line was created was taken from system time so there should not be a problem
*#The rain counter cannot be amended. It is only relevant when MX is actually processing the meteorological day with the rain counter, and you are not advised to change the rain counter for the current day using this web page. Instead got to '''Edit''' tab and choose ''Today's rain'' and that will edit the rain counter correctly.
*Some fields show a '''‡''' which links to a footnote at the bottom of the fields.
**These are source fields, and were listed earlier.
*Some fields have a line below the field box saying something like: <tt>Derived: heat index=12.1 (humidex=7.4)</tt>
**These derived fields were also listed earlier.
**This added line shows what is the calculated value for the box above (and in this particular case an alternative derived value)
**If there are decimal places, the way the value is shown should use a decimal comma if you use a decimal comma, and a decimal point otherwise.
**You can check whether this number (12.1 in this case) agrees with the value in the box.

==== If you change the value in one or more source field(s) ====

[[File:Edit stnd log line(field change).png|right]]
If you alter the number shown in the input box for one of the fields marked with a '''‡''', then any derived fields that are calculated using that source will be recalculated and you will see a message like the one shown in the screen shot: <tt>Humidity change=> 12.8</tt> which appears a few fields down from the top.

Where altering a particular source does not change a particular derived value, it will continue to say something like: <tt>Derived: 13.5 ºC</tt> as you can see much further down in the image.

These messages below the input boxes for derived values, will be seen to change again if you then edit another source field. Now you can understand why on this dialog, the values read in from the existing line stay in the input boxes where you have not made an edit. There is no point in making the updates to the derived value fields permanent until you have finished editing the source fields. Consequently, when you have finished your edits to the source fields, you will see a prompt at the bottom of the screen (see image below to left) suggesting you can manually change the derived values to the values shown below those input fields, before you click '''Save edit'''.

If you don't want to edit all the derived fields manually, there is an alternative approach also mentioned in the footnotes in image below left, and the next section describes that alternative.

==== Second time you edit same line ====

[[File:Edit stnd log line scrolldown.png|left|500px]]
The image on the left reflects when you have edited one or more source fields, but not edited all the derived field input boxes. As it says when you scroll down to click '''Save edit''', there is an OR option. Here are the instructions for that alternative option:
#Subsequently, click that '''Save edit''',
#wait for ''Success'' information message to appear,
#then check which line number is shown at top of dialog.
#Next close the dialog in one of the three ways described earlier.
#Finally, select that same line number and click '''Edit a row'''

You now see something like the image on the right. [[File:Edit stnd log line2ndTime.png|right|600px]]
*You will see that the values that were shown as derived values (under the input boxes) are now automatically placed in those input boxes.
*And the values that were shown previously in those input boxes (after any edits) have been moved below the boxes and shown like this: <tt>Previously: 13.6
Automatically updated to derived value</tt>
*If you look closely at the image you can see for some fields a difference between the values that were previously saved to the line and the values that are now to be saved when the '''Save edit''' button is clicked for second time.

=== Problems ===

Because the dataTables version of the log seen on this web page in the admin interface, is not the actual log being used by the MX "engine", there can be no conflicts on access and this way of editing log files is just as safe for the current log as for past logs.

== Additional ideas for how any Standard Log File can be used within Cumulus ==

=== Uploading current log to your web site ===

The current standard log can be updated at your usual web site update interval using ''<currentlogfile>'' as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced.

#*For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]).
#*For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column.
#You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. You will also need to specify a path in the remote column for MX or the 'ExtraRemote[0 to 99]=...' for Cumulus 1. You might of course want to specify a local path, and leave the FTP column unticked (or set to 0 for Cumulus 1). You can then use a separate FTP process to upload this copy.

*It is recommended that you do not try to FTP the actual standard log file (instead FTP a copy) because the FTP process is relatively slow, particularly as the file size increases as the month progresses, so there is an increasing chance of conflict with logging the next line of values - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for more information.
*An alternative approach is to upload the standard log file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus.

=== Cumulus 1: transferring the contents of the standard log file into a database table ===

*A script can be used to read the your copy of the standard log file into either a database table (see [[ImportCumulusFile]]), or into an array, for further processing.
*It is, as advised above, best to work with a copy of the file that Cumulus is actually updating, because there can be a conflict of the locks, Cumulus needs to have exclusive use when it does its normal update.
*But this script can safely be used with older standard log files, or a copy of the latest standard log file.

=== Cumulus MX: transferring the contents of the standard log file into a database table ===

Although the '''ImportCumulusFile''' script can be used with MX too, MX provides 2 other ways to transfer the contents of a Standard Log File into a database table.

#The first, uploads ALL past data, using '''ExportMySQL.exe''' found in the top level folder where ''CumulusMX.exe'' is found. It can be run using the same way as you run the main program, but you need to specify "monthly" as the first parameter (this indicates the table name that is to be updated, it does not work for any other table name).
#The second, which only uploads the line most recently added to the standard log file is found in the '''My SQL''' settings section of the admin interface. Find more information at [[Cumulus_MX#MySQL_settings|MySQL settings]]. In this option the default table name is "monthly", but it can be changed.

===After you have edited (or created any missing) Standard log files, you can:===

* HOW TO CORRECT HIGHS AND LOWS IN CUMULUS 1
** update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. See [[Alltimelog.txt]] for current and previous values.
** update (if the created/edited standard log is within the current year) the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu.
** update (if the created/edited standard log is for the current month) the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values.
** update (from version 1.9.3) the highs and lows in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows.
'' '''Note''' in each of above 4 editing screens you can:
# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp.
# load all standard log files, and dayfile.txt to view extremes calculated from those figures (as available) and
# optionally (in Cumulus 1 only) click ''Copy'' (row by row) to move the logged values (and associated date/time information) into the relevant .ini file.
#*In the MX editor, you need to manually type the numbers read in from the other logs into the output log column.
# (In cumulus 1 only) click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''.
# In cumulus 1, store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel'').
#*In MX there are tick and cross options for same effect.
(Each of these screens is a text editor, and works best when at full screen).''

*HOW TO USE CORRECTED INFORMATION
** (In cumulus 1 only) see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the standard log files and the daily summary log.
** (In cumulus 1 only) use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all standard log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant standard log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below).
** obtain (in Cumulus 1 by [[Standard_log_files#Viewing_standard_log_files | viewing the created/edited log file]]) figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]].
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''.
***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from values read from the standard log files) for period in question. Finally press ''Save'' button to store the new or amended report.
***In Cumulus MX, use the '''Reports''' tab, then choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. A date selector allows you to choose the exact report you want, click "Load" button next to it. You can alternatively click another button to generate a missing report, or to regenerate an existing report after any corrections you have made to any standard log file relevant to that report. Be aware that a regenerated report might not be quite as accurate as original report which when generated is using figures actually being processed by end of day for the current day.

'''NOAA Report Notes:'''
* for obsolete versions up to 1.9.4 build 1085 only: The average wind speed used for NOAA reports and 'This period' type screens was, by a bug, based on midnight to midnight days regardless of rollover time in use. (It is calculated on generation of report or screen from standard log file entries).
*From build 1086, the calculation is based on the rollover time being used.
* For all Cumulus 1 versions, and the MX versions below 3.4.4: the yearly average temperature is calculated from summing monthly maximum and monthly minimum values and dividing by number of values in calculation.
* From version 3.4.4 the annual average temperature is calculated using the integrated method based on every temperature reading processed (it is actually the average calculated from all daily means stored in daily summary log for this year).

==Correcting any logged data problems==

As already mentioned, it is only MX where the 'Data Logs' screen will allow editing of values in any standard log file, but in both flavours you can use their viewer to look and see if you are happy with what has been recorded.

The MX editor is in the admin interface where it has no access to the actual log file which is stored wherever the MX engine runs. When you ask to load a selected month in that editor, what happens is that the MX engine does a quick read of the log file and puts its contents into the format used by dataTables software. That is what you see on the web page. Subsequently you can select a line there, click the '''Edit a line''' button, and the AltEditor software will raise a pop up showing all the fields in the line. This is where you make the edits. When you click the '''Save edit''' button, the line is sent back to the MX engine. On arrival of an updated line, the MX engine reads the log file again, into a standard array, replaces the array element corresponding to the received line, writes the array back into log file (replacing previous contents) and regenerates the dataTables format to send back to web page.

Some Weather stations may occasionally supply corrupted or rogue values to Cumulus:
*Cumulus 1 provides via '''Configuration''' menu [[Cumulus.ini#Section:_Offsets|''Calibration'']] screen the ability to screen out spikes (i.e. abnormal differences between two successive readings from weather station) in data picked up from some weather stations, it does not work for all types. See Cumulus help screen if you decide to use that to cope with future spikes.
*Cumulus MX provides the same functionality, for the same restricted set of station types in its admin interface.

If a weather station for some reason does not make a new reading available for a particular parameter, then both flavours of Cumulus reuse the same value as before for up to 6 times of reading the other parameters successfully. After that, Cumulus may stop any further processing. By default, Cumulus (of any flavour) will stop any further processing if certain outdoor sensors stop sending information:
*pressure
*wind speed
*temperature
*humidity


Cumulus records in the standard log file various parameters that cannot be obtained from every weather station. Both flavours store zero if no value can be obtained for that parameter.

Some rogue values may be obviously invalid (e.g. a zero pressure), others (e.g. wind shake on a tipping bucket rain gauge on a fine day) may be within the accepted all-time range, but you know the value is not right for that particular time.

You may choose to just edit the rogue value into a reasonable value, or (if you are unable to decide on a better value) to delete that particular entire row in the standard log.

In Cumulus MX you can do both of these in the data logs screen, in Cumulus 1 you will need to use an external editor.

As explained above once any monthly log has been corrected, new highs and lows in other logs are easily rectified.

=== Spotting rogue values ===

Frequently, the graphical views ('''Charts''', ''Graphs'' or ''Trends'') are the best way to spot many rogue values and to help you guess by interpolation a more reasonable value.

Another possibility is to examine the relevant values in previous versions of the log files stored in the ''backup'' or ''backup/daily'' subdirectories of where your cumulus executable is stored.


===Manipulation outside Cumulus===

Any log files for ''previous'' months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The standard log file for the current month can also be edited outside Cumulus, but as it is being updated frequently you must stop Cumulus first (that explains why there was no edit option within Cumulus 1!).

'''Tips''' -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the ''not to be touched copy'' a filename of "<Month><Year>log.csv").
Edit the original file using an editor that treats all fields as text [use either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date and time) types]. [[File:Open office (editing cumulus log files).png | left]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected (as normal default for this software). Best not to use 'Microsoft Excel' as that normally has opposite default, and may change without any warning the format of all dates so Cumulus can no longer recognise the first two fields:

'''Important Rules'''
*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end).
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]].
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points.
*Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360.
*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can.
* Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers.

===Importing pre-Cumulus data into Cumulus 1===

This section was written for Cumulus 1. MX works differently, and all of its processing checks that the value it is processing is after the "start date", so it is not so easy to bring in earlier data. This is discussed further later within this section.

See [[FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station|FAQ: I've just installed Cumulus and it didn't download all the old data from my weather station]]

Given that standard log files are used as input for '''updating the all-time, month-by-month alltime (from version 1.9.3), and this year record extremes''', for '''creating missing dayfile.txt entries''', and for '''creating NOAA style reports''', you may have some observations from your weather station recorded manually or electronically for a period before you first starting using Cumulus 1 and want to create new standard log files to feed into Cumulus 1.

'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[Standard_log_files#List_of_fields_in_the_file|List of fields in the file]] below with the fields you have available in your source.
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number).
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings').
* Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions).
* Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]]
* Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures.
* Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated from the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] if unavailable from your station. The daily total in field 9 is since rollover time, and available weather station outputs may need some processing in your spreadsheet to derive this. Some stations output a count of total rainfall, others the total annual rainfall, either of these may be used for field 11. Rainfall processing issues are frequently discussed in the Support Forum, so you may find the solution there or ask there for assistance.

'''The date that Cumulus first started tracking all-time records''' (<#recordsbegandate> see [[Webtags#Records]]), '''does ''not'' need to be updated for Cumulus 1''' to recognise earlier monthly log files in its ''data'' subfolder and to use any monthly logs found in the various '''View''' and '''Edit''' options available from the Cumulus main screen.

'''Cumulus MX is different''', it only reads monthly log files from the existing ''records began date'', so that needs to be edited (''StartDate='' line in [[cumulus.ini]] with MX stopped, because it is not on any settings screen) to allow any earlier data to be read. You will also need to update all the other log files that record extremes manually using the editors provided in MX.

As <#recordsbegandate> is on the default Cumulus 1 or MX ''recordsT.htm'' template with the label "Records began on", inserting pre-Cumulus data implies you might want to amend that label wording on that template to indicate it is the date that Cumulus 1 was first operational, rather than the date of the earliest tracked highs and lows.

If you prefer to edit the start date and keep the ''recordsT.htm'' template label "Records began on" unchanged on the default in Cumulus 1 or MX, the method is to stop Cumulus 1, and amend the ''StartDate='' line in [[cumulus.ini]] within the main Cumulus folder before re-starting Cumulus, but (in case you make a mistake) back up everything first!

=== Cumulus Wind Speed Terminology ===

How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus:
#Gust - the highest wind speed over a particular period (this period defaults to 10 minutes),
#Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes),
#Latest - the most recent (wind speed or gust speed) measurement.
[[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Davis_stations.3F|FAQ: Wind speeds in Cumulus with Davis stations]], the ''EasyWeather'' differences in [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F|FAQ: Wind speeds in Cumulus with Fine Offset stations]] and [[FAQ#How_do_I_set_up_recording_wind_speeds_in_Cumulus.2C_with_Oregon_Scientific_and_La_Crosse_stations| FAQ: Wind speeds with Oregon Scientific and La Crosse stations]]

When Cumulus is running normally, it actually stores (in the fifth item ''after'' the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record (assuming you have a) kept default averaging interval to 10 minutes, and b) have also adjusted the logging interval from the default 15 minutes to be 10 minutes).

If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day.

Cumulus calculates the monthly/annual/period ''average wind speed'' (as used for 'This ...' screens in '''View''' menu and the NOAA report) by trawling through the relevant standard logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your standard log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval.

[[Windrun]] is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger. Again, if Cumulus is not left running the calculated value will be skewed.

===Using Standard logs to deal with shorter (or incomplete) dayfile.txt records for particular dates===

'''WORKAROUND if required dates are present in both the standard log and dayfile.txt, but not all fields for that date exist in dayfile.txt'''

Example: '''add extra fields to records created by an earlier version of Cumulus''' [to help you, the versions (not builds) at which fields were added are indicated below].
Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files.

''One method '''for Cumulus 1 only''' is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use ''create missing'' in the [[dayfile.txt]] selection in the '''Edit''' menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).''

'''Cumulus MX'''
Although there is a tentative plan for an executable to be added to the MX package that can read the standard logs and calculate all the fields needed for a line in dayfile.txt, this is only a gleam in the developer's eye!

The only method available at the time of writing involves multiple steps:
#*Either use '''ExportMySQL.exe monthly''' to create a database table called "monthly", and read all the standard logs into that database table
#*Or if you already have the table and have MX updating the standard logs into that database table
#Then, you can fairly easily update to add any missing values into existing columns in this monthly table, and from it update columns in a daily summary table. See [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] where I describe two PHP scripts designed for this type of task.

==List of fields in the file==

The table is split by Cumulus version for all changes since version 1.8.5; it shows:
#The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column.
#As a quick guide to which fields can take signs and which can not accept decimal places, the second column indicates formatting rules.
#*Any item labelled integer will not accept any decimal places. Any item labelled number will take integer or decimal values.
#*Any item labelled unsigned will not accept a minus sign. Those labelled signed don't need '''+''' for positive, but need '''-''' for negative.
#The third column shows an example, and the values shown are sometimes integers and sometimes floating point numbers, and this depends on a number of factors, including your station type. If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures, and floating point for the others (add a ".0" if necessary). Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero.
#The final column describes the observation often with a link to where there is more information.

===Version 1.8.5:===


{| border="1" class="wikitable"
{| border="1" class="wikitable"
Line 354: Line 124:
! style="width:450px" | Description
! style="width:450px" | Description
|-
|-
| 00(A)
| 1(A)
|Day-Month-Year (2 digits each)
|Day-Month-Year (2 digits each)
|22/04/11
|22/04/11
| Date as ''2 figure day [separator] 2 figure month [separator] 2 figure year'' - the separator is that set in the your locale for short date format (see [[setup]])
| Date as ''2 figure day [separator] 2 figure month [separator] 2 figure year'' - the separator is that set in the your locale for short date format (see [[setup]])
|-
|-
| 01(B)
| 2(B)
|Hour-Minute (2 digits each)
|Hour-Minute (2 digits each)
|10:25
|10:25
| Current time to nearest minute (no seconds allowed). Separator must be colon in MX, but can also be set to something else in Cumulus 1
| Current time to nearest minute (no seconds allowed). Separator must be colon in MX, but can also be set to something else in Cumulus 1
|-
|-
| 02(C)
| 3(C)
|Signed number
|Signed number
|8.1
|8.1
| Current temperature
| Current temperature
|-
|-
| 03(D)
| 4(D)
|unsigned integer
|unsigned integer
|96
|96
| Current [http://en.wikipedia.org/wiki/Relative_humidity relative humidity]
| Current [http://en.wikipedia.org/wiki/Relative_humidity relative humidity]
|-
|-
| 04(E)
| 5(E)
|Signed number
|Signed number
|7.5
|7.5
| Current [http://en.wikipedia.org/wiki/Dewpoint dew point]
| Current [http://en.wikipedia.org/wiki/Dewpoint dew point]
|-
|-
| 05(F)
| 6(F)
|unsigned number
|unsigned number
|13.5
|13.5
| Cumulus 'Average' wind speed over 10 minutes (or whatever period selected) (See [[#Cumulus_Wind_Speed_Terminology]])
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes), (See [[#Cumulus_Wind_Speed_Terminology]])
|-
|-
| 06(G)
| 7(G)
|Unsigned number
|Unsigned number
|20.3
|20.3
| Cumulus 'Gust' wind speed (See [[#Cumulus_Wind_Speed_Terminology]])
| Cumulus '''Gust''' wind speed - the highest wind speed over a particular period (this period defaults to 10 minutes), (See [[#Cumulus_Wind_Speed_Terminology]])
|-
|-
| 07(H)
| 8(H)
|integer
|integer
|138
|138
| Average wind bearing (in degrees)
| Average wind bearing (in degrees)
|-
|-
| 08(I)
| 9(I)
|unsigned number
|unsigned number
|7.2
|7.2
| [[FAQ#How_is_my_rain_rate_calculated.3F | Current rainfall rate]]
| [[FAQ#How_is_my_rain_rate_calculated.3F | Current rainfall rate]]
|-
|-
| 09(J)
| 10(J)
|unsigned number
|unsigned number
|5.4
|5.4
| Total rainfall today so far (i.e. resets to zero at daily rollover)
| Total rainfall today so far (i.e. resets to zero at daily rollover)
|-
|-
| 10(K)
| 11(K)
|unsigned number
|unsigned number
|30.10 or 998.2
|30.10 or 998.2
| Current sea level pressure
| Current sea level pressure
|-
|-
| 11(L)
| 12(L)
|unsigned number
|unsigned number
|215.2
|215.2
| Total rainfall counter (content depends on weather station type), might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]. This figure is not intended to be useful in itself, it may vary for no apparent reason. (It is used internally by Cumulus, but only within current meteorological day.)
| Total rainfall counter (content depends on weather station type), might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]. This figure is not intended to be useful in itself, it may vary for no apparent reason. (It is used internally by Cumulus, but only within current meteorological day.)
|-
|-
| 12(M)
| 13(M)
|signed number
|signed number
|20.3
|20.3
| Inside temperature
| Inside temperature This is not available as daily or longer period summary.
|-
|-
|-
| 13(N)
| 14(N)
|unsigned integer
|unsigned integer
|53
|53
| Inside humidity
| Inside humidity This is not available as daily or longer period summary.
|-
|-
|-
| 14(O)
| 15(O)
|unsigned number
|unsigned number
|17.6
|17.6
| Cumulus 'Latest' gust (See [[#Cumulus_Wind_Speed_Terminology]])
| Cumulus '''Latest'''' measured wind speed (may be a gust) (See [[#Cumulus_Wind_Speed_Terminology]]). This is not available as daily or longer period summary.
|-
|-
| 15(P)
| 16(P)
|Signed number
|Signed number
|4.8
|4.8
| [[wind chill]] (content depends on weather station type and Cumulus station settings)
| [[wind chill]] (content depends on weather station type and Cumulus station settings)
|}

===Added in 1.7.11===

{| border="1" class="wikitable"
|-
|-
! style="width:60px" | Field #
| 16(Q)
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 17(Q)
|signed number
|signed number
|8.1
|8.1
| [[Heat index]]
| [[Heat index]]
|-
|}

| 17(R)

===Added in 1.8.4===

{| border="1" class="wikitable"
|-
! style="width:60px" | Field #
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 18(R)
|unsigned number
|unsigned number
|0.5
|0.5
| UV Index (only valid if output by weather station)
| UV Index (only valid if output by weather station)
|-
|-
| 18(S)
| 19(S)
|unsigned integer
|unsigned integer
|197
|197
Line 458: Line 251:
! style="width:450px" | Description
! style="width:450px" | Description
|-
|-
| 19(T)
| 20(T)
|unsigned number
|unsigned number
|0.08
|0.08
| Evapotranspiration (only valid if output by weather station)
| Evapotranspiration (only valid if output by weather station)
|-
|-
| 20(U)
| 21(U)
|unsigned number
|unsigned number
|171.88
|171.88
| Annual Evapotranspiration (only valid if ET sensor on weather station)
| Annual Evapotranspiration (only valid if ET sensor on weather station)
|-
|-
| 21(V)
| 22(V)
|Unsigned number
|Unsigned number
|3.4
|3.4
| [[Apparent temperature]]
| [[Apparent temperature]]
|-
|-
| 22(W)
| 23(W)
|Unsigned integer
|Unsigned integer
|663
|663
| Current theoretical max solar radiation (see [[FAQ#What_is_this_.22Solar_Max.22_value_which_Cumulus_is_displaying.3F | the FAQ on Solar max]])
| Current theoretical max solar radiation (see [[FAQ#What_is_this_.22Solar_Max.22_value_which_Cumulus_is_displaying.3F | the FAQ on Solar max]])
|-
|-
| 23(X)
| 24(X)
|Unsigned number
|Unsigned number
|3.1
|3.1
Line 493: Line 286:
! style="width:450px" | Description
! style="width:450px" | Description
|-
|-
| 24(Y)
| 25(Y)
|Unsigned integer
|Unsigned integer
|158
|158
| Current Wind bearing (See 7)
| Current Wind bearing (See average wind bearing in field 8)
|}
|}
''For Cumulus MX editing a file created by Cumulus 1'', any of the preceding fields that are not present will be '''defaulted to an empty string''', but there will always be 50 fields (unless you are using MX version before 3.6.0, it which case MX cannot read the file.


===Added in 1.9.3===
===Added in 1.9.3===

''For Cumulus MX 3.6.x reading a line created by Cumulus 1'', all the remaining fields will be '''filled with exactly one space''', and editing that to an empty string will be rejected. Cumulus MX before version 3.6.0 cannot read a log file with all lines in that log file created by any Cumulus 1 version prior to 1.9.4.

{| border="1" class="wikitable"
{| border="1" class="wikitable"
|-
|-
Line 507: Line 304:
! style="width:450px" | Description
! style="width:450px" | Description
|-
|-
| 25(Z)
| 26(Z)
|Unsigned number
|Unsigned number
|6.0
|6.0
| RG-11 rain today (only valid if output by sensor)
| RG-11 rain today (only valid if output by sensor) This is not available as daily or longer period summary.
|-
|}
|}


Line 522: Line 320:
! style="width:450px" | Description
! style="width:450px" | Description
|-
|-
| 26(AA)
| 27(AA)
|Unsigned number
|Unsigned number
|5.7
|5.7
Line 537: Line 335:
! style="width:450px" | Description
! style="width:450px" | Description
|-
|-
| 27(AB)
| 28(AB)
|signed number (or space)
|13.7
| [[Feels Like]] temperature

NOTE: MX adds a space here, if a line (created before 3.6.0 without a valid value in this field) is edited.
|}

=== Added in 3.6.12 ===

Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following field, but until version 3.7.0 (build 3089) the extra field is empty.


{| border="1" class="wikitable"
|-
! style="width:60px" | Field #
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 29(AC)
|signed number
|signed number
|13.7
|13.7
| [[Feels Like]] temperature
| [[Humidex]]
|}
|}


==Example lines from the file==
=== Added in 3.nn?.0 ===

This new field makes it easier to work out a time in UTC, and identifies uniquely those lines created during a switch from Daylight Saving Time to Winter time, that would otherwise have duplicate times for an hour at time-zone changeover.

The following change was initially planned for release of MX 3.9.0 (b3095), but this section has been amended as the introduction of this new field has been postponed, because of implications of other changes planned at same time (see below table).

The developer has said it is still his plan to add this field during a major release in 2022, but he cannot yet say when it will actually be available to public.


{| border="1" class="wikitable"
|-
! style="width:60px" | Field #
! style="width:180px" | Rule
! style="width:80px" | Example
! style="width:450px" | Description
|-
| 30(AD)
|datetime integer
| 1579564800000
| UNIX time-stamp
|}

''The proposed "Updates.txt" entry read <pre>
- Major changes to how Cumulus MX handles it's data files
- Dayfile.txt
- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility)
- The decimal point is now fixed at "."
- The list separator is now fixed at ","
- Monthly log files
- The file names are fixed at "yyyyMMlog.txt"
- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility), any lines using "dd/MM/yy" are converted to new separator
- The time field format is now fixed at "HH:mm"
- An extra field is added that is the UTC date-time integer
- The decimal point is now fixed at "."
- The list separator is now fixed at ","
- Extra Data log files
- No change in file names
- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility), any lines using "dd/MM/yy" are converted to new separator
- The time field format is now fixed at "HH:mm"
- An extra field is added that is the UTC date-time integer
- The decimal point is now fixed at "."
- The list separator is now fixed at ","
</pre>''

= Accessing the Standard Log File =

Please see [[Calculate Missing Values]] and [[Correcting Extremes]] for full details on relevant functionality in Cumulus software.

== Access to current log by other processes ==

As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus 1 applies an exclusive lock, and [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 conflicts] can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while Cumulus is running. Whilst MX does not apply such locks, it would be unwise to externally edit a file that is being frequently updated while MX is running.
Also note these log files do not include a header line, and should not be edited to include it.

== Additional ideas for how any Standard Log File can be used within Cumulus ==

=== Uploading current log to your web site ===

The current standard log can be updated at your usual web site update interval using ''<currentlogfile>'' as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced.

#*For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini_(Cumulus_1)#Section:_FTP_Site]]).
#*For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column.
#You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. You will also need to specify a path in the remote column for MX or the 'ExtraRemote[0 to 99]=...' for Cumulus 1. You might of course want to specify a local path, and leave the FTP column unticked (or set to 0 for Cumulus 1). You can then use a separate FTP process to upload this copy.

*It is recommended that you do not try to FTP the actual standard log file (instead FTP a copy) because the FTP process is relatively slow, particularly as the file size increases as the month progresses, so there is an increasing chance of conflict with logging the next line of values - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for more information.
*An alternative approach is to upload the standard log file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus.

=== Cumulus 1: transferring the contents of the standard log file into a database table ===

*A script can be used to read the your copy of the standard log file into either a database table (see [[ImportCumulusFile]]), or into an array, for further processing.
*It is, as advised above, best to work with a copy of the file that Cumulus is actually updating, because there can be a conflict of the locks, Cumulus needs to have exclusive use when it does its normal update.
*But this script can safely be used with older standard log files, or a copy of the latest standard log file.

=== Cumulus MX: transferring the contents of the standard log file into a database table ===

{{Template:WorkInProgressBanner}}

PLEASE UPDATE THIS SECTION - '''ExportMySQL.exe''' HAS BEEN REPLACED BY [https://github.com/cumulusmx/ExportToMySQL ExportToMySQL] - see [https://cumulus.hosiene.co.uk/viewtopic.php?p=154206#p154206 forum] for an example

Although the '''ImportCumulusFile''' script can be used with MX too, MX provides 2 other ways to transfer the contents of a Standard Log File into a database table.

#The first, uploads ALL past data, using '''ExportMySQL.exe''' found in the top level folder where ''CumulusMX.exe'' is found. It can be run using the same way as you run the main program, but you need to specify "monthly" as the first parameter (this indicates the table name that is to be updated, it does not work for any other table name).
#The second, which only uploads the line most recently added to the standard log file is found in the '''My SQL''' settings section of the admin interface. Find more information at [[Cumulus_MX#MySQL_settings|MySQL settings]]. In this option the default table name is "monthly", but it can be changed.

===Additional Functionality in Cumulus===

*Additional derived values in original Cumulus software
** You can see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the standard log files and the daily summary log.
** You can use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu.
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''.
***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from values read from the standard log files) for period in question. Finally press ''Save'' button to store the new or amended report.

*The last of these is also in MX
**In Cumulus MX, use the '''Reports''' tab, then choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. A date selector allows you to choose the exact report you want, click "Load" button next to it. You can alternatively click another button to generate a missing report, or to regenerate an existing report after any corrections you have made to any standard log file relevant to that report.

=Rules to follow if editing a standard data log=

==Important Rules for Cumulus 1==

*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end).
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]].
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points.
*Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360.
*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can.
* Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers.

==Equivalent rules for MX ==

*All fields in all lines must contain the same character for separating one field from next and it must be consistent with whatever locale you use when you run MX (it might be set using a parameter to the command running MX, it might be set in Mono, or in Windows control panel).
* Each line must contain all the fields '''defined in the version of MX that you are using''' in correct sequence
* ''Feels like'' was added onto the end of this log file in version 3.5.4. Any line in any log file produced before you started using MX version 3.5.4 or later will end with ''Rain since midnight''. However, the way that MX was modified for version 3.5.4 means that those earlier lines will be modified by the application process interface used to transfer log lines between the MX engine and the admin interface to include the full number of fields per row and to contain a space character for that additional field. Consequently, any external edit also has to add a field separator and a space to every line where the original lacked feel like.
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files'').
* Times in these files are in the form ''hh:mm'' using a colon as separator, the 24 hour clock and local time (system time).
* If you are editing a log file that was produced by Cumulus 1 and it used a separator other than a colon (:) then the old separator must be replaced by colon.
* For MX the rules for fields containing values are different to those for Cumulus 1, which makes it difficult to successfully edit any pre MX log files. For MX if a field can take a decimal place, it must contain a decimal figure, and it must use the correct decimal separator as defined in locale. If the field only takes integers (see [[#List_of_fields_in_the_file]]) then you cannot edit in any decimal places.


==Importing pre-Cumulus data==

If you were recording weather data manually, or with different weather station software, you might create files, using the standard data log files format, holding that pre-Cumulus data. Cumulus can treat these files just like any where it created the data log entries, and therefore you can apply any of the techniques explained in [[Calculate Missing Values]] and [[Correcting Extremes]].

In the configuration file, Cumulus.ini, there is a parameter '''StartDate=''', this is assigned a value when you first start using Cumulus of the current date, in the format specified by the '''local''' you are using.
* The original Cumulus 1 software takes no notice of this date, other than to use it to populate the <#recordsbegandate> see [[Webtags#Records]], so it is quite happy to process standard data logs with lines with earlier dates.
* The '''ExportMySQL.exe''' which was written by Steve Loft and included with the beta releases of MX, and subsequent early MX release distributions from Mark Crossley, also ignores the date in the configuration file.
* All releases containing '''CumulusMX.exe''', will ignore any data log lines dated before the date in the configuration file.
* The other Mark Crossley utility packages linked from [[Software#Other_utilities|Software page utilities section]] will also generally ignore any data log lines dated before the date in the configuration file.
** [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] will use the earliest date in [[dayfile.txt]] if that is earlier than the date in the configuration file.
** '''ExportToMySQL.exe''' (not currently documented in this Wiki) will normally ignore any data log lines dated before the date in the configuration file.

Don't worry if you are using MX and your pre-cumulus data does not have all the fields listed at [[#List_of_fields_in_the_file]]. [[Calculate_Missing_Values#CreateMissing.exe|CreateMissing.exe]] can be used to calculate (at lease some of, maybe all of) those missing fields. That utility will also create [[dayfile.txt]] entries for the days in the pre-Cumulus data.

If you are not using MX, you may still be able to populate missing fields (see [[Calculate_Missing_Values|calculate missing values]] page), and create lines in dayfile.txt (see [[Amending dayfile]] page).

===Potential issues when trying to match formats===

'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[#List_of_fields_in_the_file|List of fields in the file]] below with the fields you have available in your source.
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number).
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings').
* Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions).
* Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]]
* Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[Standard_log_files#Using_Standard_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures.
* Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated by following advice in [[Calculate Missing Values]].

==== Cumulus Wind Speed Terminology ====

'''This section was written for the original Cumulus software'''. There are some differences in how MX reads information from weather stations, particularly those by Davis, but that should not affect content below.

How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus:
#Gust - the highest wind speed over a particular period (this period defaults to 10 minutes),
#Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes),
#Latest - the most recent (wind speed or gust speed) measurement.
[[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Davis_stations.3F|FAQ: Wind speeds in Cumulus with Davis stations]], the ''EasyWeather'' differences in [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F|FAQ: Wind speeds in Cumulus with Fine Offset stations]] and [[FAQ#How_do_I_set_up_recording_wind_speeds_in_Cumulus.2C_with_Oregon_Scientific_and_La_Crosse_stations| FAQ: Wind speeds with Oregon Scientific and La Crosse stations]]

When Cumulus is running normally, it actually stores (in the fifth item ''after'' the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record (assuming you have a) kept default averaging interval to 10 minutes, and b) have also adjusted the logging interval from the default 15 minutes to be 10 minutes).

If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day.

Cumulus calculates the monthly/annual/period ''average wind speed'' (as used for 'This ...' screens in '''View''' menu and the NOAA report) by trawling through the relevant standard logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your standard log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval.

[[Windrun]] is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger. Again, if Cumulus is not left running the calculated value will be skewed.


=Example lines from the file=


Note that your field delimiters may be different, and your date delimiter too.
Note that your field delimiters may be different, and your date delimiter too.
Line 566: Line 547:
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0
[[Category:Cumulus MX]]



this is an example of a file from 1.9.1, with solar data:
this is an example of a file from 1.9.1, with solar data:
Line 577: Line 560:
and from version 1.9.2:
and from version 1.9.2:


Here logging is at the default interval, and the delimiters have their default settings for UK systems.
Here logging is at the default interval (10 minutes), and the delimiters have their default settings for UK systems.


01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158
01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158
Line 583: Line 566:
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180


and from version 3.6.0:

Here logging is at the default interval, and the delimiters have their default settings for UK systems.


01/06/20,14:11,24.9,31,6.6,10.8,13.1,45,0,0,1026,205.8,122.5,38,13.1,24.9,24.9,0,0,0,0,20.7,831,0,45,0,0,22
[[Category:Log Files]]
01/06/20,14:21,24.3,31,6.1,8.4,13.1,90,0,0,1025.8,205.8,122.6,38,9.3,24.3,24.3,0,0,0,0,20.8,822,0,90,0,0,22

Latest revision as of 12:15, 10 April 2022

Cumulus Version MX SpecificCumulus Version 1 Specific This page applies to all flavours.

Introduction

Cumulus has ability to log data on a periodic basis, see Monthly log files:

  • This page is about the standard logging file, that is always created, with a new file in each new month
  • Steve Loft started work, for Cumulus 2 on a replacement for these files using a database, and planned Cumulusmx.db to hold such data for MX 3.0.0 beta, although his work on this was never completed.
  • Other files (Speciallog.txt , Extra_Sensor_Files, and Air_Link_Log.txt) are optional, in that settings determine whether they are created or not.

In fact, in Cumulus MX, the processing of this extra sensor logging file shares a lot of the code with the processing of the standard logging file, as they both are updated at same interval, and they both take values processed following a read from your weather station. The applications programming interface (api) that sends values between the MX engine and the MX admin interface is also identical (with a parameter "Extra" that swaps between true and false depending on the log file involved).

The logging file, speciallog.txt, for detailed logging of internal temperature and humidity is not split into separate files for each month. Although this exists in Cumulus 1, it appears this functionality will remain missing from MX.

The logging file for air link sensors, is only available in MX.

How often is data stored in the standard file?

These store the full set of current (read and derived) values at whatever interval you have set for Cumulus to keep detailed logging.

  • The interval has to be a fraction of an hour, so the least often storing is every half hour.
  • For some weather station types, you MUST match the interval at which the weather station logs data internally
  • The default time interval between adding data logging lines is every 10 minutes, but you can change this:
    • Legacy Cumulus 1: On main screen, select Configuration menu, select the station settings screen, look for Data log interval box on extreme RH side.
    • Cumulus MX: From Interface, find settings menu, select the Station settings page, select General Settings, and find Log interval where there is a drop down that offers the various timings that are permitted.

What name is used for the standard file?

These files are automatically created with names in the form <Month><Year>log.txt.

In UK English locale, the file for August 2009 is called Aug09log.txt.

The month can be in another language, or a different abbreviation, depending on your locale.

The same naming format must be used for all files,if Cumulus is to be able to find them. For Cumulus MX only, any file lines for dates before the Cumulus start date will be ignored.

File content

This file has one line added each time the Cumulus timer says the logging interval has passed.

All data is logged in the units as selected by the user:

  • Cumulus 1: On main screen, select Configuration menu, select the station settings screen, Units box is just below the centre of that screen.
  • Cumulus MX: From Interface, find settings menu, select the Station settings page, and find Units section.
    • Advanced settings in MX allow you to vary the number of decimal places for the various stored values

Each line has a number of fields separated by whatever symbol is defined as the list separator on your device, typically a comma (,) or a semi-colon (;).

  • The various fields are all listed after this section, but they can be of various types:
    • Date, this is always the first field, and it always uses a fixed format, ignoring the format defined in your locale.
    • Time, this always the second field, and is always in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). However, if you use the feature in MX to upload the equivalent of this file to a database table, the automatic update upload will add ":00" to include seconds.
      • Note that it is perfectly possible to have two successive lines in a standard log file with same time, although this should not happen in normal running, it can result from a hiccup or other interference on your device. If you are asking MX to upload the contents of this file, a second line with same time is ignored, only the first line with a particular date and time is uploaded.
    • All Remaining fields are spot values, either what has just been read from your station, or a derived value calculated from a set of values read from your station. For these value fields, Cumulus may have converted from the unit used by your weather station to the unit that you have requested to use, and there will be some inaccuracy inevitable in such conversions. Where the value to be stored is a real number with decimal places, Cumulus will use a decimal comma or decimal point depending on which is defined in your locale, do be careful not to change that locale if you need to change your device or update it, as Cumulus can only read files where the same decimal separator is used everywhere.

For more information on these files see in the Cumulus help file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from Software#Resources page in this Wiki.

List of fields in the file

The table is split by Cumulus version for all changes since version 1.7.x (Steve Loft did not retain his list of changes in 1.4, his changes.txt is rather sparse in details for many releases, and the support forum excludes any posts before 2 Jun 2008, so earlier changes are not now documented).

The table shows:

  1. The field number
    • When this list was first produced, only the original Cumulus software was available and the decision then was to start with date in field zero to be consistent with index used for arrays in programming languages like JavaScript.
    • In the data log editing page of the MX admin interface, and internally in its code, MX uses field zero for the line number, and therefore the table below has been edited so that date moves to field 1.
    • For each field the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc is also shown in first column.
  2. As a quick guide to which fields can take signs and which can not accept decimal places, the second column indicates formatting rules.
    • Any item labelled integer will not accept any decimal places.
    • Any item labelled number can in Cumulus 1 take integer or decimal values, but in MX the decimal places are mandatory.
    • Any item labelled unsigned will not accept a minus sign. Those labelled signed don't need + for positive, but do need - for negative.
  3. The third column shows an example of a value accepted by Cumulus 1
    • The file content rules for MX changed in release 3.7.0, and it has not been verified that all Cumulus 1 examples are still accepted
    • For Cumulus 1, where decimals are allowed, the stored value might be an integer as this depends on a number of factors, including your station type.
    • If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures
    • In Cumulus 1 it is optional to use floating point for the others
    • In Cumulus MX add a ".0" if necessary because MX must see decimal places where these are allowed
    • Note that the figure in a field varies depending on the units you select, and your weather station may not have all the sensors needed
    • In Cumulus 1 any fields defined in the version you are using for sensors you do not have will always contain zero.
    • Also, for Cumulus 1 any fields that are not defined at the version when the line was created can:
      • Either be missing (no fields seen after the last field at your version)
      • Or can be null (field separators only are seen after the last field at your version)
    • MX since version 3.7.0 is more fussy, but less consistent, certain fields contain a single space when not used, others contain zero, and others can be left empty
  4. The final column describes the observation often with a link to where there is more information.

Number of fields per line varies by release

The fields that are available in the release you are running are listed in Monthlyfileheader.txt (found in folder level above the data sub-folder).

If you have some early files with fewer fields than the latest files, there are limited possibilities for calculating correct values for those missing fields

Badge v1.png For the legacy Cumulus software, the number of fields in the file increased as the software was developed. The Cumulus author, Steve Loft, did not provide any utility for populating fields missing from earlier versions of his software. A third party did produce a PHP script that could:

  • add Heat Index to lines created prior to 1.7.11
  • add Apparent Temperature (not other fields, see below) to lines created before 1.9.1
  • add Feels Like temperature (not other fields, see below) to lines created before 3.6.0

The fields that cannot be calculated by this script are (set them all to zero when editing file manually):

  • UV Index
  • Solar Radiation
  • Evapotranspiration (spot value and annual cumulative total)
  • Current theoretical max solar radiation
  • Hours of sunshine so far today
  • Current Wind bearing
  • RG-11 rain today
  • Total Rainfall since midnight (this is for 0900/1000 'rollover' users; normally same as rain today for 'midnight rollover' users)

Badge vMx.png As MX has developed, further fields have been added to the file at 3.6.0 and 3.6.12:

  • Feels Like temperature
  • Canadian Humidity Index (Humidex)

The main MX developer, Mark Crossley, has written a Create Missing utility, this is dependent on having a significantly recent MX release, but it can calculate the following derived values (coping with cases where any of these are missing:

Please note the developer does not mention any of this at his utility details explanation so the author of this Wiki update cannot guarantee the above list is correct


You will need to check in MX release announcements for any update, at time of typing this, the developer has written a routine within CumulusMX.exe to calculate Evapotranspiration, but that is not in the CreateMissing.exe Utility.

Before Version 1.7.10:

Field # Rule Example Description
1(A) Day-Month-Year (2 digits each) 22/04/11 Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the your locale for short date format (see setup)
2(B) Hour-Minute (2 digits each) 10:25 Current time to nearest minute (no seconds allowed). Separator must be colon in MX, but can also be set to something else in Cumulus 1
3(C) Signed number 8.1 Current temperature
4(D) unsigned integer 96 Current relative humidity
5(E) Signed number 7.5 Current dew point
6(F) unsigned number 13.5 Cumulus moving 'Average' of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes), (See #Cumulus_Wind_Speed_Terminology)
7(G) Unsigned number 20.3 Cumulus Gust wind speed - the highest wind speed over a particular period (this period defaults to 10 minutes), (See #Cumulus_Wind_Speed_Terminology)
8(H) integer 138 Average wind bearing (in degrees)
9(I) unsigned number 7.2 Current rainfall rate
10(J) unsigned number 5.4 Total rainfall today so far (i.e. resets to zero at daily rollover)
11(K) unsigned number 30.10 or 998.2 Current sea level pressure
12(L) unsigned number 215.2 Total rainfall counter (content depends on weather station type), might be rainfall total so far this year. This figure is not intended to be useful in itself, it may vary for no apparent reason. (It is used internally by Cumulus, but only within current meteorological day.)
13(M) signed number 20.3 Inside temperature This is not available as daily or longer period summary.
14(N) unsigned integer 53 Inside humidity This is not available as daily or longer period summary.
15(O) unsigned number 17.6 Cumulus Latest' measured wind speed (may be a gust) (See #Cumulus_Wind_Speed_Terminology). This is not available as daily or longer period summary.
16(P) Signed number 4.8 wind chill (content depends on weather station type and Cumulus station settings)

Added in 1.7.11

Field # Rule Example Description
17(Q) signed number 8.1 Heat index


Added in 1.8.4

Field # Rule Example Description
18(R) unsigned number 0.5 UV Index (only valid if output by weather station)
19(S) unsigned integer 197 Solar Radiation (only valid if solar sensor on weather station)

Added in 1.9.1

Field # Rule Example Description
20(T) unsigned number 0.08 Evapotranspiration (only valid if output by weather station)
21(U) unsigned number 171.88 Annual Evapotranspiration (only valid if ET sensor on weather station)
22(V) Unsigned number 3.4 Apparent temperature
23(W) Unsigned integer 663 Current theoretical max solar radiation (see the FAQ on Solar max)
24(X) Unsigned number 3.1 Hours of sunshine so far today (only valid if solar sensor on weather station)

Added in 1.9.2

Field # Rule Example Description
25(Y) Unsigned integer 158 Current Wind bearing (See average wind bearing in field 8)

For Cumulus MX editing a file created by Cumulus 1, any of the preceding fields that are not present will be defaulted to an empty string, but there will always be 50 fields (unless you are using MX version before 3.6.0, it which case MX cannot read the file.

Added in 1.9.3

For Cumulus MX 3.6.x reading a line created by Cumulus 1, all the remaining fields will be filled with exactly one space, and editing that to an empty string will be rejected. Cumulus MX before version 3.6.0 cannot read a log file with all lines in that log file created by any Cumulus 1 version prior to 1.9.4.

Field # Rule Example Description
26(Z) Unsigned number 6.0 RG-11 rain today (only valid if output by sensor) This is not available as daily or longer period summary.

Added in 1.9.4

Field # Rule Example Description
27(AA) Unsigned number 5.7 Total Rainfall since midnight (this is for 0900/1000 'rollover' users; normally same as rain today for 'midnight rollover' users)

Added in 3.6.0

Field # Rule Example Description
28(AB) signed number (or space) 13.7 Feels Like temperature

NOTE: MX adds a space here, if a line (created before 3.6.0 without a valid value in this field) is edited.

Added in 3.6.12

Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following field, but until version 3.7.0 (build 3089) the extra field is empty.


Field # Rule Example Description
29(AC) signed number 13.7 Humidex

Added in 3.nn?.0

This new field makes it easier to work out a time in UTC, and identifies uniquely those lines created during a switch from Daylight Saving Time to Winter time, that would otherwise have duplicate times for an hour at time-zone changeover.

The following change was initially planned for release of MX 3.9.0 (b3095), but this section has been amended as the introduction of this new field has been postponed, because of implications of other changes planned at same time (see below table).

The developer has said it is still his plan to add this field during a major release in 2022, but he cannot yet say when it will actually be available to public.


Field # Rule Example Description
30(AD) datetime integer 1579564800000 UNIX time-stamp

The proposed "Updates.txt" entry read 

- Major changes to how Cumulus MX handles it's data files
	- Dayfile.txt
		- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility)
		- The decimal point is now fixed at "."
		- The list separator is now fixed at ","
	- Monthly log files
		- The file names are fixed at "yyyyMMlog.txt"
		- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility), any lines using "dd/MM/yy" are converted to new separator
		- The time field format is now fixed at "HH:mm"
		- An extra field is added that is the UTC date-time integer
		- The decimal point is now fixed at "."
		- The list separator is now fixed at ","
	- Extra Data log files
		- No change in file names
		- The date field format is now fixed at "dd-MM-yy" (old order is retained for third party compatibility), any lines using "dd/MM/yy" are converted to new separator
		- The time field format is now fixed at "HH:mm"
		- An extra field is added that is the UTC date-time integer
		- The decimal point is now fixed at "."
		- The list separator is now fixed at ","

Accessing the Standard Log File

Please see Calculate Missing Values and Correcting Extremes for full details on relevant functionality in Cumulus software.

Access to current log by other processes

As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus 1 applies an exclusive lock, and conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while Cumulus is running. Whilst MX does not apply such locks, it would be unwise to externally edit a file that is being frequently updated while MX is running.

Also note these log files do not include a header line, and should not be edited to include it.

Additional ideas for how any Standard Log File can be used within Cumulus

Uploading current log to your web site

The current standard log can be updated at your usual web site update interval using <currentlogfile> as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced.

    • For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in Cumulus.ini_(Cumulus_1)#Section:_FTP_Site).
    • For Cumulus MX, use the Extra web files option within the "Settings" tab, and enter "<currentlogfile>" in the local column.
  2. You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus Help. You will also need to specify a path in the remote column for MX or the 'ExtraRemote[0 to 99]=...' for Cumulus 1. You might of course want to specify a local path, and leave the FTP column unticked (or set to 0 for Cumulus 1). You can then use a separate FTP process to upload this copy.
  • It is recommended that you do not try to FTP the actual standard log file (instead FTP a copy) because the FTP process is relatively slow, particularly as the file size increases as the month progresses, so there is an increasing chance of conflict with logging the next line of values - see support forum here for more information.
  • An alternative approach is to upload the standard log file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus.

Cumulus 1: transferring the contents of the standard log file into a database table

  • A script can be used to read the your copy of the standard log file into either a database table (see ImportCumulusFile), or into an array, for further processing.
  • It is, as advised above, best to work with a copy of the file that Cumulus is actually updating, because there can be a conflict of the locks, Cumulus needs to have exclusive use when it does its normal update.
  • But this script can safely be used with older standard log files, or a copy of the latest standard log file.

Cumulus MX: transferring the contents of the standard log file into a database table

Crystal Clear info.png This document is 'Work In Progress' so content may not be complete.

Request for help from Wiki Readers

  • Do you understand how MX works?
  • Do you use hardware, or MX functionality, that is not yet documented? Can you begin that documenting?
  • Can you contribute simple text for novice users, examples of what you have done, correction of typing or factual errors, or supply missing details?
  • Will you make this page more useful by bringing content up-to-date as new releases change some information written for older releases?
  • Does any page need a section for novices, so they don't need to read more technical information further down that page?
  • Is there some information on this page, that should be on a separate page? Can you create the new page and move the less relevant information off this page, don't forget this page needs a link to the new page so people who expect to find it here know where it has moved to?

If you plan on contributing to the Wiki, then you will need an account.

  • Please use the Request Account form to apply for an account. Note that the Wiki is currently undergoing restructuring and is largely locked for editing, but please apply for an account if you wish to contribute in the future.
  • You will find help on how to contribute to this wiki at How to Edit.
  • If you need to consult others, please use the Cumulus Wiki suggestions forum.

Please be aware that information on this page may be incorrect.

PLEASE UPDATE THIS SECTION - ExportMySQL.exe HAS BEEN REPLACED BY ExportToMySQL - see forum for an example

Although the ImportCumulusFile script can be used with MX too, MX provides 2 other ways to transfer the contents of a Standard Log File into a database table.

  1. The first, uploads ALL past data, using ExportMySQL.exe found in the top level folder where CumulusMX.exe is found. It can be run using the same way as you run the main program, but you need to specify "monthly" as the first parameter (this indicates the table name that is to be updated, it does not work for any other table name).
  2. The second, which only uploads the line most recently added to the standard log file is found in the My SQL settings section of the admin interface. Find more information at MySQL settings. In this option the default table name is "monthly", but it can be changed.

Additional Functionality in Cumulus

  • Additional derived values in original Cumulus software
    • You can see Mean temperatures, Air frosts, Gale days, Rain days, Dry days, and Total Wind run using View menu displays for This month, This period, and This year (These figures are not stored anywhere, nor available as web tags), but Cumulus 1 can calculate these from what it finds in the standard log files and the daily summary log.
    • You can use create missing button in the dayfile.txt selection in the Edit menu.
    • create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report.
      • In Cumulus 1 do this from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from values read from the standard log files) for period in question. Finally press Save button to store the new or amended report.
  • The last of these is also in MX
    • In Cumulus MX, use the Reports tab, then choosing NOAA Monthly Report or NOAA Annual Report. A date selector allows you to choose the exact report you want, click "Load" button next to it. You can alternatively click another button to generate a missing report, or to regenerate an existing report after any corrections you have made to any standard log file relevant to that report.

Rules to follow if editing a standard data log

Important Rules for Cumulus 1

  • Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end).
  • The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within Log_Files).
  • Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). See FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F.
  • See #List_of_fields_in_the_file to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points.
  • Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360.
  • Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of #List_of_fields_in_the_file and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can.
  • Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers.

Equivalent rules for MX

  • All fields in all lines must contain the same character for separating one field from next and it must be consistent with whatever locale you use when you run MX (it might be set using a parameter to the command running MX, it might be set in Mono, or in Windows control panel).
  • Each line must contain all the fields defined in the version of MX that you are using in correct sequence
  • Feels like was added onto the end of this log file in version 3.5.4. Any line in any log file produced before you started using MX version 3.5.4 or later will end with Rain since midnight. However, the way that MX was modified for version 3.5.4 means that those earlier lines will be modified by the application process interface used to transfer log lines between the MX engine and the admin interface to include the full number of fields per row and to contain a space character for that additional field. Consequently, any external edit also has to add a field separator and a space to every line where the original lacked feel like.
  • The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within Log_Files).
  • Times in these files are in the form hh:mm using a colon as separator, the 24 hour clock and local time (system time).
  • If you are editing a log file that was produced by Cumulus 1 and it used a separator other than a colon (:) then the old separator must be replaced by colon.
  • For MX the rules for fields containing values are different to those for Cumulus 1, which makes it difficult to successfully edit any pre MX log files. For MX if a field can take a decimal place, it must contain a decimal figure, and it must use the correct decimal separator as defined in locale. If the field only takes integers (see #List_of_fields_in_the_file) then you cannot edit in any decimal places.


Importing pre-Cumulus data

If you were recording weather data manually, or with different weather station software, you might create files, using the standard data log files format, holding that pre-Cumulus data. Cumulus can treat these files just like any where it created the data log entries, and therefore you can apply any of the techniques explained in Calculate Missing Values and Correcting Extremes.

In the configuration file, Cumulus.ini, there is a parameter StartDate=, this is assigned a value when you first start using Cumulus of the current date, in the format specified by the local you are using.

  • The original Cumulus 1 software takes no notice of this date, other than to use it to populate the <#recordsbegandate> see Webtags#Records, so it is quite happy to process standard data logs with lines with earlier dates.
  • The ExportMySQL.exe which was written by Steve Loft and included with the beta releases of MX, and subsequent early MX release distributions from Mark Crossley, also ignores the date in the configuration file.
  • All releases containing CumulusMX.exe, will ignore any data log lines dated before the date in the configuration file.
  • The other Mark Crossley utility packages linked from Software page utilities section will also generally ignore any data log lines dated before the date in the configuration file.
    • CreateMissing.exe will use the earliest date in dayfile.txt if that is earlier than the date in the configuration file.
    • ExportToMySQL.exe (not currently documented in this Wiki) will normally ignore any data log lines dated before the date in the configuration file.

Don't worry if you are using MX and your pre-cumulus data does not have all the fields listed at #List_of_fields_in_the_file. CreateMissing.exe can be used to calculate (at lease some of, maybe all of) those missing fields. That utility will also create dayfile.txt entries for the days in the pre-Cumulus data.

If you are not using MX, you may still be able to populate missing fields (see calculate missing values page), and create lines in dayfile.txt (see Amending dayfile page).

Potential issues when trying to match formats

Here are some issues to consider. There are some postings in the Support Forum about importing past data. Essentially match the fields listed in List of fields in the file below with the fields you have available in your source.

  • Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to concatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number).
  • Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings').
  • Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide look up functionality to do such conversions).
  • Wind speed fields may not directly match to weather station outputs. See #Cumulus_Wind_Speed_Terminology
  • Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the workaround described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures.
  • Rainfall fields may not directly match to weather station outputs. Rain rate in field 8 may be generated by following advice in Calculate Missing Values.

Cumulus Wind Speed Terminology

This section was written for the original Cumulus software. There are some differences in how MX reads information from weather stations, particularly those by Davis, but that should not affect content below.

How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus:

  1. Gust - the highest wind speed over a particular period (this period defaults to 10 minutes),
  2. Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes),
  3. Latest - the most recent (wind speed or gust speed) measurement.

FAQ: Wind speeds in Cumulus with Davis stations, the EasyWeather differences in FAQ: Wind speeds in Cumulus with Fine Offset stations and FAQ: Wind speeds with Oregon Scientific and La Crosse stations

When Cumulus is running normally, it actually stores (in the fifth item after the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record (assuming you have a) kept default averaging interval to 10 minutes, and b) have also adjusted the logging interval from the default 15 minutes to be 10 minutes).

If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day.

Cumulus calculates the monthly/annual/period average wind speed (as used for 'This ...' screens in View menu and the NOAA report) by trawling through the relevant standard logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your standard log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval.

Windrun is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger. Again, if Cumulus is not left running the calculated value will be skewed.


Example lines from the file

Note that your field delimiters may be different, and your date delimiter too.


An extract of a few lines of the file (v.1.9.0)

Here logging is quarter hourly, the field delimiter is a semicolon, the decimal separator is a comma, and the date separator a dash. 

30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0 
30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0
30-09-10;19:30;16,2;94;15,2;7,9;15,8;355;7,2;25,8;994,3;52,5;25,7;62;12,2;16,2;16,2;0,0;0
30-09-10;19:45;16,0;94;15,0;9,9;19,4;7;7,2;27,9;993,3;54,6;25,7;62;14,8;15,8;16,0;0,0;0
30-09-10;20:00;15,9;94;14,9;12,4;20,9;354;7,2;30,0;993,0;56,7;25,7;62;19,4;15,4;15,9;0,0;0
30-09-10;20:15;15,8;94;14,8;8,4;15,8;349;14,4;32,7;993,4;59,4;25,8;61;12,2;15,8;15,8;0,0;0
30-09-10;20:30;15,4;94;14,4;13,8;33,1;317;28,8;40,5;993,7;67,2;25,8;61;23,4;14,7;15,4;0,0;0
30-09-10;20:45;15,1;94;14,1;20,3;34,2;356;7,2;43,8;992,3;70,5;25,8;60;29,5;13,8;15,1;0,0;0
30-09-10;21:00;15,3;94;14,3;20,2;35,6;358;10,8;46,8;991,0;73,5;25,8;60;28,1;14,0;15,3;0,0;0
30-09-10;21:15;15,3;95;14,5;16,6;31,7;358;10,8;49,5;991,4;76,2;25,8;60;20,9;14,3;15,3;0,0;0
30-09-10;21:30;15,3;94;14,3;14,0;27,0;324;18,0;54,3;992,3;81,0;25,8;60;15,8;14,5;15,3;0,0;0
30-09-10;21:45;15,2;94;14,2;13,0;25,6;323;10,8;57,9;992,3;84,6;25,8;59;24,5;14,5;15,2;0,0;0
30-09-10;22:00;15,0;94;14,0;16,7;31,7;312;10,8;60,6;993,0;87,3;25,8;59;23,4;13,9;15,0;0,0;0
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0


this is an example of a file from 1.9.1, with solar data:

Here logging is increased to every five minutes, the field delimiter is a comma, the decimal separator is a full stop, and the date separator is also a full stop. 

22.04.11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0
22.04.11,10:30,8.1,96,7.5,13,20,142,0.0,0.0,1013.28,215.2,20.2,53,11,4.8,8.1,0.0,216,0.08,171.88,3.3,673,0.0
22.04.11,10:35,8.1,96,7.5,12,18,142,0.0,0.0,1013.31,215.2,20.2,53,11,5.1,8.1,0.0,227,0.08,171.88,3.8,682,0.0

and from version 1.9.2:

Here logging is at the default interval (10 minutes), and the delimiters have their default settings for UK systems. 

01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158
01/10/11,04:50,17.9,75,13.4,4.5,9.5,169,0.0,0.0,1021.4,885.3,24.4,61,2.0,17.9,17.9,0.0,0,0.00,0.00,17.5,0,0.0,180
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180

and from version 3.6.0:

Here logging is at the default interval, and the delimiters have their default settings for UK systems. 

01/06/20,14:11,24.9,31,6.6,10.8,13.1,45,0,0,1026,205.8,122.5,38,13.1,24.9,24.9,0,0,0,0,20.7,831,0,45,0,0,22
01/06/20,14:21,24.3,31,6.1,8.4,13.1,90,0,0,1025.8,205.8,122.6,38,9.3,24.3,24.3,0,0,0,0,20.8,822,0,90,0,0,22
Retrieved from "https://www.cumuluswiki.org/index.php?title=Standard_log_files&oldid=10320"