Daily Summary

DAILY SUMMARY

Introduction

1. This article updates Annual Data Weather article contributed by David Jamieson in March 2011.
2. A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is here; note that example uses a different carrier page (see below) from the Weather Blue set of Alternative Templates available from the French only site Météo du Québec and the international site Weather by_You.
3. This article is called Daily Summary as it summarises scripts that various authors have contributed that take as their source daily ststistics output from Cumulus
   • Typically these scripts allow the maximimum, minimum, average, or total of a statistic calculated for an individual day to be seen in a table that shows such values for many days often spanning at least one year.
4. The scripts mentioned below range quite widely:

• Some are direct descendants of David Jamieson's original idea, in that they read the daily summary log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another
  • Others are variants in that the columns might represent years, or you may be able to scroll the columns so although they are months, they don't have to start with January.
  • Some use purely HTML, CSS, and JavaScript (although they use a JavaScript library called jQuery)
  • Others add some PHP Hypertext Preprocessor (PHP) script so that some of the work is done on the web server before the HTML page reaches the browser
  • I have included cross-references to various forum threads that explore variations to the original idea by adding enhancements such as colour shading, day naming (e.g. 'Wednesday'), seasonal summaries, or in some other way offer a different look, but are still generally based on reading the log file.

• However, there are also scripts that are written in PHP and use database queries (typically MySQL) to retrieve the required statistics from a database Table containing these statistics with one table row for each day
  1. The output from these scripts can be a calendar type table, similar to David's concept, that only shows a single weather criterion
  2. The output can be a subset of the database table, in that a number of selected database columns are shown, and there is one row for each day
  3. The output can be for a single criterion, and for each day of the month (or year), figures shown for every year available; this allows instant comparison between tha same day in all years

Update of David Jamieson PHP script by BCJKiwi

Here is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines. They also include a couple of updates from DAJ version as advised by posts in the Forum.

Errors rectified by this version

• Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row').
• In version 1.1 of this tool, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight; from jQuery version 1.6 that was deprecated and the '.prop' object is now specified for the same purpose.
• Version 1.0 and 1.1 default to showing a table for viewer's local current calendar year. Remember the dayfile.txt (even if updated each day) only contains records up to the day ending at the last rollover time using the timezone local to the weather station. Timezone differences when combined with rollover time variations can lead to a period of 2 days discrepancy. At New Year a blank table will be displayed during this period by default. For example, 30 or 31 (depending on whether rollover has occured) in December of the last year could be the latest available for a site in USA whilst the original version of the code is showing a blank table of the next year during early morning on 2 January in Austrailia. Even when station and viewer are in same timezone, there will be a blank table until the first rollover on 2 January.
• Version 1.2 defaults to showing the year found in the last row of dayfile.txt.
• The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote fdata.split("\r\n"); i.e. defines that carriage return then line feed (newline) used to split daily observations in dayfile. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code.

Configuration Variables

By default in 'readDayfile.php':

• Assume the dayfile.txt is read from a subfolder called 'data'.
  • Look for the line 'dayfile='/data/dayfile.txt';' around line 15 and change this to point to your dayfile.txt on your webserver.

for PHP version the path should start from the / (root)

• Assume the date format in the dayfile is dd/mm/yy
  • You may need to edit the file if you do not use '/' between the parts of the date (the month is always in the middle for all Cumulus log files), look in file for date_delimiter' - the symbol separating your date format. See setup.
• Assume the data in the dayfile is separated with a comma
  • Look for the line mentioning field_delimiter - the symbol separating each of your fields in the dayfile.txt. For most people this is a comma but (if you use comma to separate integer and decimal parts of real numbers) it could be a semi-colon (;) or other symbol.

Localization / Language

The script has been designed to be easily translated to your language of choice.

As in Instructions below, edit the readDayfile.php

• variable mn is a list of the 12 months of the year, in an abbreviated format. You may change these as necessary, but try to keep it to an abbreviation as there is limited space.
• 'label_items' is a list of all possible data sets to be displayed. Again you can change these to suit if you are prepared to wade through all the script and make changes elsewhere too. However, the format is a little more involved and you should take some care. Each row represents one data set, with 4 columns of settings for that data set; so the default list is an array of four by six.

Example of a row,....

['maxtemp','Max Temp','Maximum Temperature',true]

• The first element, in this case 'maxtemp' -- is a system variable to identify the data set, used for the URL parameter option, used by the Switch coding (so any change here needs to be reflected there), and also tested elsewhere in the code (in the Rainfall and Windgust data sets a particular style is applied to their zero values).

However, if you add a new data set, then your new first element should be a unique identifier.

• Second, Max Temp is the text to be displayed in the button at the top of the table. You may change this to your own language

If you add a new data set, remember to consider how many buttons can be fitted across the top of the table, you may wish to split into multiple rows.

• Third, Maximum Temperature is the text shown at the top of the table to describe the current data set; again you may change this
• Finally, 'true' will display this button at the top; 'false' will hide it. Therefore, if you do not wish to allow users to jump to the 'Rainfall data set' change the 'true' to 'false' in the 'rainfall' element of the variable. (true/false MUST be lowercase)

NOTE: Sfws 12:22, 31 December 2012 (UTC)

Having made any change to the array as DAJ says above, you also need to change what is in a switch section and you will find that is below where DAJ says '// Nothing to edit below here'.

The switch ($whatdata) { ... } part of the coding identifies the columns in dayfile.txt for the values to be displayed. Any change you make in the array, needs a corresponding change within this part of the script. A typical case here reads

case 'mintemp'	: $tablelayout .= $label_items[1][2]; $dayfilecol=4; break;

so to add a new one, copy that line and paste it in again but in this new line replace 'mintemp' with whatever you have added as the first element in your new array row, replace '[1]' with square brackets round the number of your new array row (the first row is number zero) and finally replace '4' with the field number where the parameter you have added is found in dayfile.txt.

Instructions

The instructions in David's original article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant.

• Download the following zip file...AnnualDataSummary_12.zip, the zip contains 3 files:
  1. readDayfile.php - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style
  2. datasummary.php - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) Settings.php and common.php. You can use your own page instead of this file - see carrier page customisation below.
  3. datasummary.css - styling sheet for the table structure that displays selected statistics in a calendar style
• Edit the file 'readDayfile.php' in a good text editor (for example, notepad++) and consider the configuration variables described above that appear in the top section of the script.

• After any necessary editing, copy datasummary.php and readDayfile.php files to your web site into a subfolder, or the root -- your choice
• Copy datasummary.css (for possible tailoring see styling instructions below) to your web site into the same folder

Usage

• Open datasummary.php from your website in a browser

Along the top will be a menu of the (in original version) six data sets available and on the top left of the table is the year currently being shown. Change either the year, or click one of the top buttons to change the data set. Depending on the speed of your internet connection, and your browser, the new web page may take a few seconds to be processed as it involves: re-reading the dayfile.txt file, redrawing the table, and inserting the values.

If you mouse over a particular value, it will be highlighted together with the corresponding day (row heading) and month (column heading).

Opening a specific data set when the page is first loaded

By default (in the original version) the table created will show the available daily values of the maximum temperature for the current local calendar year, however you can start with any data set and any year (assuming you have those values in the dayfile.txt) by adding a 'Query String' to the end of your URL in the browser.

• The parameter 'data' (in the original version) takes mintemp, maxtemp, avgtemp, minmaxt, rainfall, windgust. (You can add further data sets or change the language of these values - see Localization). Note the specified data will be displayed even if it is assigned 'false' and does not appear as a button.
• You can supply both data set and year parameters, one only, or none

examples....,

• myserver.com/datasummary.php?year=2010

This will open the default (maximum temperature) data set but initially showing year 2010

• myserver.com/datasummary.php?data=rainfall&year=2009

This will open the rainfall data set for the year 2009.

Understanding the Basics

The original 'tool' relies on three files for successful operation:

1. the jQuery library routines that allow it to work in any user agent (browser)
2. datasummary.css (the style sheet) and
3. readDayfile.php (the PHP version of the script).

Please note that the table needs a good amount of space to show a full year of data (at least 900 pixels unless you start reducing the font size!)

In addition you need a carrier page to show the data.

• Note that BCJKiwi does not use a web page based on those that are supplied with Cumulus, as mentioned above his provided datasummary.php requires two extra scripts not provided. If you create two empty files with the specified names, or download files with those names from Ken True's web site, then the provided carrier page will load on your site.
• Alternatively don't bother to upload the provided datasummary.php, and instead create your own carrier page (use the same name or call it whatever suits you) folowing instructions below.

Creating your own Carrier Page

To create your own page, follow another page you are using successfully on your site and modify it by removing any content from <body> section you don't want and including in the <head> section:

• the CSS styling:

(HTML best practice is to include CSS declations as early as possible).....

<link rel="stylesheet" type="text/css" media="screen" href="datasummary.css"  />

• the script library (this example shows version 3.1, you can load whatever is the most recent version suitable for your browser):

This loads a jQuery library from Google servers rather than having the file on your server, (note many versions of jQuery are not supported by old versions of Internet Explorer, use the very old version '1.5' used by DAJ instead of the current version where '3.1' appears if you need to);

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1/jquery.js" type="text/javascript">

• Next you must load the script to build the table:

In the <body> section, use the code

<?php include('readDayfile.php');?>

(or simply copy everything that was in that file and paste it in this section directly) to include the script instructions that create all the HTML elements required to build and display the table

Styling

The table styling is completely configurable using the included Cascading Style Sheet datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see support forum thread about colouring with php

In the 'datasummary.css' file:

CSS Element	Description
#table_container .highlight	Used to highlight the mouse position within the table, and also the row and column header of the data cell.
#table_container .smallfont	Adjusting the font size to something smaller when showing both Max & Min temp on one data cell
#table_container .zerovalue	In the Rainfall and Windgust data sets this style is applied to any values of 0. By default, the colour is set to a lighter grey but you could add 'display:none;' to hide zero values completely
#table_container table th	Colouring for the top header (Month names) and left header (day numbers)
#table_container table td,table th	The width of each data cell in the table. Be careful adjusting this as making it too small will stop the data being displayed completely
#table_menu li	The styling for the buttons at the top of the page (those for changing the data set)
#table_container	Set the overall font size and style used in the table, as well as the text colour; table positioning and maximum width

Enhancement of JavaScript script

This section only is by Sfws to mention script originated 24 Nov 2012 Update of text here 00:53, 18 May 2013 (PDT) Finalised version of script in forum 7:08 am 15 Jan 2014, minor update 8:51 pm 15 Nov 2014

I originally created the alternative Javascript version in December 2011 by editing the DAJ original (with similar aims to BCJKiwi with his PHP version 1.2) to offer easier customisation plus standards compliance, i.e. addressing all the issues with his version.

It really is much easier to add new data sets if you use this alternative JavaScript version in the forum. That includes more explanation about the original coding, and an alternative simpler way to specify and adjust the columns in a revised version of the array specified for the PHP version above.

The advantages of my script are fully described in a file within the package. They include standards compliance, better integration with standard Cumulus pages, statistical additions to the bottom of each month's column, and easier tailoring to your preferences on wording, language, and text colouring. During 2012-2013, my alternative JavaScript version was rewritten from scratch (losing the original DAJ code), a number of additional features were added, whilst attempting to preserve ease of customisation to suit any requirements you have e.g. on the language and level of diagnostics output. One new feature is ability to scroll sideways, so calendar look can start with any month (any year) but always with 12 columns showing month headings.

Find my package in alternative Javascript/jQuery solution within the 3rd party tools sub forum of the Cumulus forum.

One site using in January 2013 a variant based on an early version of my Javascript/jQuery alternative code, with its own look to the page, is found here. Whilst a number of other people have tried later versions of this alternative code, and you can see some commenting in the forum thread, remember this was designed for easy tailoring, so is not an off the shelf package.

At the end of 2014, I rewrote this again, by adding use of PHP script for some of the actions previously done by JavaScript, retaining JavaScript for handling the selection of a change of criterion, and for handling the change of months to be shown (either by scrolling one month at a time or by picking one of the predefined periods on buttons). The resulting script was to prove very useful to me, and still remains on my own personal web server, but with the mix of languages (and the way it was developed) I found it is not easy to understand and hard to improve so I abandoned any idea of a version to share.

Variants of the original Daily Summary discussed in the support forum

Look in the Website Development section of the Cumulus support forum - Web general sub forum:

• 'Annual Data Summary - Coloured Values' - An extensively coloured version with some useful discussion on the trade-off between testing values and efficient code here.
• 'Annual data Summary (DAJ script) - show day-of-week' - This is just one of several places where further ideas are explored by Mark Crossley and beteljuice.
• 'Yet Another Dayfile Reader (PHP)' - This is the ultimate/latest PHP version, it incorporates all the ideas discussed before, amongst many features the highlighting makes day of week appear in row headers, you can swap between daily and seasonal views and there is a table summary including averages. See that forum url for links to sites that use this version.

Displaying daily statistics from a database table

Schema

1. The retrieval script needs to have the following defined for it to use (it is suggested that these variables are defined in a separate PHP script that is in a directory that the PHP parser can access, but is not in the part of the web server that is visible to a client):
   • The server host name, or w.x.y.z format Internet Protocol (=IP) address, running your MySQL database
   • The user name used to log-in to your database server
   • The password used to log-in to your database server
     • (It is good practice to have separate authentication for read only access, and for read and update access, alternatively use an additional key variable as David does in his update script described below)
   • The name of the (MySQL or whatever) database with the table
   • The name for the daily summary table
2. The retrieval script needs to know the names of the columns it is to read from the database.
   • Database designers talk about the schema meaning the specification that describes the design of the table.
   • A schema lists the column names together with what type and size of variables they hold.
   • For the daily summary database table, the columns can copy the fields in the daily summary log file, and this is what David Jamieson does in his approach a PHP script to read the contents of the dayfile.txt where the columns are generated in the same order as the corresponding fields in the log file, so David's script uses the field numbers to identify the columns by Position (rather than by Column Name).
   • Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39).

Loading and Updating the database table

Obviously if you are using a script that reads statistics from a table, you also need to be running a script that ensures the table is updated at the end of each day, and therefore always up to date.

• Functionality is included in Cumulus MX to do this update, it uses the schema devised by Mark Crossley by default, but Cumulus MX can also handle your own schema; for full details of the functionality see Cumulus MX Announcements - PLEASE READ THESE!.
• Cumulus 1 users need to use a script to create a database table, populate and update it from the daily summary log file. The most widely used script is an update by Mark Crossley of the script mentioned above by David Jamieson; and it is described in this Wiki article.
  • It is however worth also reading the forum post a PHP script to read the contents of the dayfile.txt as that explains the date delimiter, how a Comma Separated Variable file may use other punctuation to separate the fields, how to specify the location of your dayfile.txt log to the script and the reason the key variable is used.

Some example Scripts

Look in the Website Development section of the Cumulus support forum - Web general sub forum:

• SQL dayfile data Viewer - This script by Kocher shows selected statistics, you can scroll up and down through all available days and for those days (or all days) see some summary statistics at the foot of each column.
• MySQL code help - This shows a snippet of a PHP script by SFWS for creating a table that for a particular criterion shows the value with all the years as column headers and all the days of a year (1 Jan to 31 Dec) as rows. The provided script uses standard HTML to show/hide each month.
• Top 10 Records Page- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. ). The latest source php can be seen using a query-string on Mark Crossley's web site and there have been several updates since the forum thread was created.
• What to do with data from MySQL - This posting lists a number of web pages on PaulMy's web site where he presents ststistics from the daily summary table of his database in different formats.