Cumulus Wiki

Monthly log files

Revision as of 08:40, 17 December 2013 by Sfws (talk | contribs) (→‎After you have edited (or created any missing) Monthly log files, you can:: added note about change in build 1086)

Files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is called Aug09log.txt); an entry is made to the current month's file every ten (default value; you can change this on the station settings screen) minutes, recording the current sensor values. The file is in comma-separated format with one line per entry.

Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). All data is logged in the units which which have been selected by the user on the station configuration screen. Note that 'comma-separated' does not necessarily mean that a comma is used to separate the values! If your PC uses a semicolon for the list separator, that will be used in these files.

For more information on these files see in the Cumulus help file, in the section “The Data log file”.

Viewing monthly log files

In the View menu, select Data logs, click Load.... The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. Select the required monthly log, click Open and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen.

If no logs seen, see FAQ: I can't find my data files. Also note, this Data logs viewer is only for viewing monthly log files (not any other files in the same data directory), the column headings for standard monthly log files will always be shown at the top of the screen.

Viewing summary figures for a month

In the View menu, select This month, and by default a summary for the current calendar month will be calculated mostly from the daily summary log (therefore not including current meteorological day). Some of the values are not in that log and are derived from the monthly logs or the weather diary. Note that many parameters are different to those available as web tags, especially if you use 9am/10am rollover.

  • For example, where a day number is shown on screen, it is the meteorological day (changing at rollover), but when a timestamp (with time and date) is given in a web tag the date is a calendar date (always changing at midnight).
  • As an example applying regardless of rollover time, the screen shows total number of dry or wet days in the month, the web tags report longest dry or wet period in the month.

To view for a different month (or if you use 9am/10am rollover, if current local time on the 1st day of a month is before rollover time so the log for the current calendar month has not been started), select the month and year required using the options at bottom left, click Update Display and the revised summary will be calculated. (Alternatively you can select This period and select a start date and end date then click Update Display to get the equivalent calculation for part of a month or any other period). By default, without selecting different dates, This period shows yesterday. (Above paragraphs originated 2 June 2013)

POSTSCRIPT (17 Nov 2013): The answer at this forum thread says that the View menu screens reports average wind speed midnight to midnight (even when rollover is 9am or 10am) deriving it from just looking at the calendar date on the monthly log file. So there is a period mismatch between some parameters on the screen.

How can Monthly log files be used within Cumulus?

Uploading current log to your website

The current monthly log can be updated at your usual website update interval by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' see Cumulus.ini#Section:_FTP_Site. A script can then read the file into an array or database for further processing.

After you have edited (or created any missing) Monthly log files, you can:

  • update the highs and lows in Alltime.ini by choosing all time records from the Edit menu. See Alltimelog.txt for current and previous values.
  • update (if the created/edited monthly log is within the current year) the highs and lows in year.ini by choosing This year's records from the Edit menu.
  • update (if the created/edited monthly log is for the current month) the highs and lows in month.ini by choosing This month's records from the Edit menu. See Diags for current and previous values.
  • update (from version 1.9.3) the highs and lows in monthlyalltime.ini by choosing Monthly records from the Edit menu. Click the Help button for specific instructions on using Reset and the two Copy column header buttons in this Monthly Records (Highs and Lows) Editor to action all rows.
  Note in each of above 4 editing screens you can:
  1. see the currently stored extremes, and optionally Reset (row by row) to pre-editing value and timestamp.
  2. load the monthly log(s) and dayfile.txt to view extremes calculated from those figures (as available) and
  3. optionally Copy (row by row) the logged values (and associated date/time information) into the relevant .ini file.
  4. click the Help button for detailed instructions on using The Records (Highs and Lows) Editors.
  5. store your revised figures by clicking OK (or abandon all your edits by clicking Cancel).
 (Each of these screens is a text editor, and works best when at full screen).
  • see Mean temperatures, Air frosts, Gale days, Rain days, Dry days, and Total Wind run using View menu displays for This month, This period, and This year (These figures are not stored anywhere, nor available as webtags).
  • obtain (by viewing the created/edited log file figures needed for correcting rogue figures in dayfile.txt. See for example FAQ re correcting rainfall.
  • use create missing button in the dayfile.txt selection in the Edit menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any missing dates in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called dayfileeditlog.txt which contains the entries it created). Click the Help button for more information. In past and current versions (including 1.9.3), create missing will not affect any records that are incomplete or contain some rogue values (see workaround below).
  • create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed (always based on midnight to midnight days) and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report.


Note for obsolete versions up to 1.9.4 build 1085 only: The average wind speed used for NOAA reports and 'This period' type screens was, by a bug, based on midnight to midnight days regardless of rollover time in use. (It is calculated on generation of report or screen from monthly log entries). From build 1086, the calculation is based on the rollover time being used.

Correcting any logged data problems

Some Weather stations may occasionally supply corrupted or rogue values to Cumulus. Cumulus provides via Configuration menu Calibration screen the ability to screen out spikes (i.e. abnormal differences between two successive readings from weather station) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with future spikes. Some rogue values may be obviously invalid (e.g. a zero pressure), others (e.g. wind shake on a tipping bucket rain gauge on a fine day) may be within the accepted all-time range (so cannot be prevented by spike settings), but you know the value is not right for that particular time.

Frequently, the graphical views (View menu, Graphs or Select-A-Graph screens) are the best way to spot many rogue values and to help you guess by interpolation a more reasonable value. Another possiblity is to examine the relevant values in previous versions of the log files stored in the backup or backup/daily subdirectories of where your cumulus executable is stored. You may choose to just edit the rogue value into a reasonable value, or (if you are unable to decide on a better value) to delete that particular entire row in the monthly log. As explained above once the log is corrected, new highs and lows in other logs are easily rectified. The screen within Cumulus for viewing the monthly log does not allow editing of values.

Manipulation in Cumulus Toolbox

 

The add-on Cumulus Toolbox contains a file editor that can manipulate monthly log files using a display rather like this screen (in this figure it is actually dayfile.txt that is being edited).

Manipulation outside Cumulus

Any log files for previous months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The current month log file can also be edited outside Cumulus, but you must stop Cumulus first.

Tips -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the not to be touched copy a filename of "<Month><Year>log.csv").

Edit the original file using an editor that treats all fields as text [either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed not to recognise special field (like date) types].

Important Rules

  • Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept lines of various lengths without the more recent fields at the end).
  • The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within logfiles).
  • Times in these files are in the form hh:mm using the 24 hour clock and local time (system time). See FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F.
  • See #List_of_fields_in_the_file to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points.
  • Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360.
  • Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or -999.9 (check the format in the middle column of #List_of_fields_in_the_file and think about the range of numbers allowed for particular fields as per examples in previous bullet).
  • Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers.

Importing pre-Cumulus data

See FAQ: I've just installed Cumulus and it didn't download all the old data from my weather station

Given that monthly log files are used as input for updating the all-time, month-by-month alltime (from version 1.9.3), and this year record extremes, for creating missing dayfile.txt entries, and for creating NOAA style reports, you may have some observations from your weather station recorded manually or electronically for a period before you first starting using Cumulus and want to create new monthly logs to feed into Cumulus.

Here are some issues to consider. There are some postings in the Support Forum about importing past data. Essentially match the fields listed in Monthly log files: List of fields in the file below with the fields you have available in your source.

  • Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to cocatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number).
  • Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings').
  • Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide look up functionality to do such conversions).
  • Wind speed fields may not directly match to weather station outputs. See #Cumulus_Wind_Speed_Terminology
  • Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the workaround described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures.
  • Rainfall fields may not directly match to weather station outputs. Rain rate in field 8 may be generated from the workaround if unavailable from your station. The daily total in field 9 is since rollover time, and available weather station outputs may need some processing in your spreadsheet to derive this. Some stations output a count of total rainfall, others the total annual rainfall, either of these may be used for field 11. Rainfall processing issues are frequently discussed in the Support Forum, so you may find the solution there or ask there for assistance.

The date that Cumulus first started tracking all-time records (<#recordsbegandate> see Webtags#Records), does not need to be updated for Cumulus to recognise earlier monthly log files in its data subfolder and to use any monthly logs found in the various View and Edit options available from the Cumulus main screen. As <#recordsbegandate> is on the default Cumulus recordsT.htm template with the label "Records began on", inserting pre-Cumulus data implies you might want to amend that label wording on that template to indicate it is the date that Cumulus was first operational, rather than the date of the earliest tracked highs and lows.

If you prefer to edit the start date and keep the recordsT.htm template label "Records began on" unchanged, the method is to stop Cumulus, and amend the StartDate= line in cumulus.ini within the main Cumulus folder before re-starting Cumulus, but (in case you make a mistake) back up everything first!

Cumulus Wind Speed Terminology

How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus:

  1. Gust - the highest wind speed over a particular period (this period defaults to 10 minutes),
  2. Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes),
  3. Latest - the most recent (wind speed or gust speed) measurement.

FAQ: Wind speeds in Cumulus with Davis stations, the EasyWeather differences in FAQ: Wind speeds in Cumulus with Fine Offset stations and FAQ: Wind speeds with Oregon Scientific and La Crosse stations

When Cumulus is running normally, it actually stores (in the fifth item after the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record. If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day. Cumulus calculates the monthly/annual/period average wind speed (as used for 'This ...' screens in View menu and the NOAA report) by trawling through the monthly logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your monthly log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval.

Windrun is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger.

Using Monthly logs to deal with shorter (or incomplete) dayfile.txt records for particular dates

WORKAROUND if dates are present in monthly log and dayfile.txt but not all fields for that date exist in dayfile.txt

Example: records created by earlier version of Cumulus [to help you, the versions (not builds) at which fields were added are indicated below]. Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files.

One method is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use create missing in the dayfile.txt selection in the Edit menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).

List of fields in the file

The table is split by Cumulus version for all changes since version 1.8.5; it shows:

  • The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column.
  • The second column shows an example, for the observations note whether format is integer or real number, because any edits must match this. Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero.
  • The third column describes the observation often with a link to where there is more information.

Version 1.8.5:

Field # Example Description
00(A) 22/04/11 Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup)
01(B) 10:25 Current time
02(C) 8.1 Current temperature
03(D) 96 Current relative humidity
04(E) 7.5 Current dewpoint
05(F) 13 Cumulus 'Average' wind speed (See #Cumulus_Wind_Speed_Terminology)
06(G) 20 Cumulus 'Gust' wind speed
07(H) 138 Average wind bearing (integer degrees)
08(I) 7.2 Current rainfall rate
09(J) 5.4 Total rainfall today so far (i.e. resets to zero at rollover)+
10(K) 30.10 Current sea level pressure
11(L) 215.2 Total rainfall counter (content depends on weather station type, might be rainfall total so far this year)
12(M) 20.3 Inside temperature
13(N) 53 Inside humidity
14(O) 17 Cumulus 'Latest' gust (See #Cumulus_Wind_Speed_Terminology)
15(P) 4.8 wind chill (content depends on weather station type and Cumulus station settings)
16(Q) 8.1 Heat index
17(R) 0.5 UV Index (only valid if output by weather station)
18(S) 197 Solar Radiation (only valid if solar sensor on weather station)

Added in 1.9.1

Field # Example Description
19(T) 0.08 Evapotranspiration (only valid if output by weather station)
20(U) 171.88 Annual Evapotranspiration (only valid if ET sensor on weather station)
21(V) 3.4 Apparent temperature
22(W) 663 Current theoretical max solar radiation (only valid if solar sensor on weather station)
23(X) 3.1 Hours of sunshine so far today (only valid if solar sensor on weather station)

Added in 1.9.2

Field # Example Description
24(Y) 158 Current Wind bearing (See 7)

Added in 1.9.3

Field # Example Description
25(Z) 6.0 RG-11 rain today (only valid if output by sensor)

Added in 1.9.4

Field # Example Description
26(AA) 5.7 Total Rainfall since midnight (this is for 0900/1000 'rollover' users; normally same as rain today for 'midnight rollover' users)

Example lines from the file

Note that your field delimiters may be different, and your date delimiter too.


An extract of a few lines of the file (v.1.9.0)

Here logging is quarter hourly, the field delimiter is a semicolon, the decimal separator is a comma, and the date separator a dash. 

30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0 
30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0
30-09-10;19:30;16,2;94;15,2;7,9;15,8;355;7,2;25,8;994,3;52,5;25,7;62;12,2;16,2;16,2;0,0;0
30-09-10;19:45;16,0;94;15,0;9,9;19,4;7;7,2;27,9;993,3;54,6;25,7;62;14,8;15,8;16,0;0,0;0
30-09-10;20:00;15,9;94;14,9;12,4;20,9;354;7,2;30,0;993,0;56,7;25,7;62;19,4;15,4;15,9;0,0;0
30-09-10;20:15;15,8;94;14,8;8,4;15,8;349;14,4;32,7;993,4;59,4;25,8;61;12,2;15,8;15,8;0,0;0
30-09-10;20:30;15,4;94;14,4;13,8;33,1;317;28,8;40,5;993,7;67,2;25,8;61;23,4;14,7;15,4;0,0;0
30-09-10;20:45;15,1;94;14,1;20,3;34,2;356;7,2;43,8;992,3;70,5;25,8;60;29,5;13,8;15,1;0,0;0
30-09-10;21:00;15,3;94;14,3;20,2;35,6;358;10,8;46,8;991,0;73,5;25,8;60;28,1;14,0;15,3;0,0;0
30-09-10;21:15;15,3;95;14,5;16,6;31,7;358;10,8;49,5;991,4;76,2;25,8;60;20,9;14,3;15,3;0,0;0
30-09-10;21:30;15,3;94;14,3;14,0;27,0;324;18,0;54,3;992,3;81,0;25,8;60;15,8;14,5;15,3;0,0;0
30-09-10;21:45;15,2;94;14,2;13,0;25,6;323;10,8;57,9;992,3;84,6;25,8;59;24,5;14,5;15,2;0,0;0
30-09-10;22:00;15,0;94;14,0;16,7;31,7;312;10,8;60,6;993,0;87,3;25,8;59;23,4;13,9;15,0;0,0;0
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0

this is an example of a file from 1.9.1, with solar data:

Here logging is increased to every five minutes, the field delimiter is a comma, the decimal separator is a full stop, and the date separator is also a full stop. 

22.04.11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0
22.04.11,10:30,8.1,96,7.5,13,20,142,0.0,0.0,1013.28,215.2,20.2,53,11,4.8,8.1,0.0,216,0.08,171.88,3.3,673,0.0
22.04.11,10:35,8.1,96,7.5,12,18,142,0.0,0.0,1013.31,215.2,20.2,53,11,5.1,8.1,0.0,227,0.08,171.88,3.8,682,0.0

and from version 1.9.2:

Here logging is at the default interval, and the delimiters have their default settings for UK systems. 

01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158
01/10/11,04:50,17.9,75,13.4,4.5,9.5,169,0.0,0.0,1021.4,885.3,24.4,61,2.0,17.9,17.9,0.0,0,0.00,0.00,17.5,0,0.0,180
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180
Retrieved from "https://www.cumuluswiki.org/index.php?title=Monthly_log_files&oldid=3222"