MX Administrative Interface: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
Line 16: Line 16:


This same api can be used to get weather data into another device, although you may prefer to use MQTT instead as that is more easily customised to just share the few weather values you might want.
This same api can be used to get weather data into another device, although you may prefer to use MQTT instead as that is more easily customised to just share the few weather values you might want.
===Technical aside===
When reading the support forum (or articles in this Wiki), you may see mention of AJAX and WebSockets. They are both methods of exchanging information between a web page and the data source.  In the Cumulus MX context, they represent two ways of updating the data shown on an admin interface page like '''now.html''' or ''todayyest.html''.
Asynchronous JavaScript and XML (AJAX) is the widely used name for what Microsoft invented as '''XMLHttpRequest''' (late 1999, Internet Explorer 5.0) now it is implemented in all browsers and no longer purely Microsoft feature.  Put as simply as I can, it lets code written into JavaScript ask for updated information from the Cumulus MX engine to update the numbers on the web page.  In the standard Cumulus 1 trends.htm web page, AJAX was used to load an updated graph into the web page each time the client pressed a button requesting a graph.
WebSockets is an improvement on AJAX. While AJAX needs the client to make a request before an update can be made, WebSockets adds the option that the source can generate the request and do a remote update.  Thus when you open ''todayyest.html'', the MX engine gets a request from that web page to insert the various sub-tables into that web page. Subsequently, when WebSocket use is enabled, the MX engine sends updates with new figures, until you close the web page and a '''Connection Closed''' message is sent instead.


== The MQTT interface ==
== The MQTT interface ==

Revision as of 09:13, 7 August 2020


  1. Type the URL which is displayed (when MX starts running) into your browser (replace the * with the IP address, or use 'localhost') and the admin interface should appear. You will be viewing your admin interface on the same network as the the MX engine is using, so what you type might be something like http://192.168.1.x:y/ and you may need to look at your hub or router to see what to use for x and y as they represent the device where MX is running.
    • If this is a 'clean' installation, i.e. you don't have an existing Cumulus.ini file defining station type and units to be used, the first thing you will need to do is to go to the settings screen.
  2. When that page is displayed it always shows the options to set the station type and units.
  3. Before you leave this page, you can make any other configuration settings by picking a section and clicking it to show the selections within that section.
  4. Now go to bottom of this page and click the 'Save' button.

Having set the station type, and other settings, you will need stop Cumulus MX and start it again.

The API interface

The current data is transferred from the MX engine to the Admin interface as a JSON string accessed via a Application Program Interface (api). To see the full content type into your browser the same IP reference as for the admin interface and add a few more items i.e. something like http://192.168.1.x:y/api/data/currentdata and you may need to look at your hub or router to see what to use for x and y as they represent the device where MX is running.

Many browsers (e.g Firefox) allow you to specify the type of a file you want to view. So if you specify json as the type this will make the browser show it in a long but fairly readable format. This api is how the current data (now.html) page in the admin interface gets its data. Each table cell contains a span element and each span element is given an id attribute whose value matches one of the items in the json stream coming via the api.

This same api can be used to get weather data into another device, although you may prefer to use MQTT instead as that is more easily customised to just share the few weather values you might want.

Technical aside

When reading the support forum (or articles in this Wiki), you may see mention of AJAX and WebSockets. They are both methods of exchanging information between a web page and the data source. In the Cumulus MX context, they represent two ways of updating the data shown on an admin interface page like now.html or todayyest.html.

Asynchronous JavaScript and XML (AJAX) is the widely used name for what Microsoft invented as XMLHttpRequest (late 1999, Internet Explorer 5.0) now it is implemented in all browsers and no longer purely Microsoft feature. Put as simply as I can, it lets code written into JavaScript ask for updated information from the Cumulus MX engine to update the numbers on the web page. In the standard Cumulus 1 trends.htm web page, AJAX was used to load an updated graph into the web page each time the client pressed a button requesting a graph.

WebSockets is an improvement on AJAX. While AJAX needs the client to make a request before an update can be made, WebSockets adds the option that the source can generate the request and do a remote update. Thus when you open todayyest.html, the MX engine gets a request from that web page to insert the various sub-tables into that web page. Subsequently, when WebSocket use is enabled, the MX engine sends updates with new figures, until you close the web page and a Connection Closed message is sent instead.

The MQTT interface

Until this section is written, please see Cumulus.ini#MQTT for the information you can put into settings, and where to enter it.

The Data Log Viewing and Editing interface

The Data logs tab in the admin interface, gives access to screens that can display the contents of 3 types of log files:

  1. The standard log file; there is one of these for each month - choose menu item Data logs (yes same name as tab)
  2. The Extra Sensor log File; there is one of these for each month - choose menu item Extra Data Logs
  3. The Daily Summary log file; there is only one of these - choose menu item Dayfile

General points common to web pages for editing standard and extra sensor logs

For the first two web pages, but not the daily summary log web page:

  • On this screen,you see a box for selecting the log you want to display. The default month and year (shown on loading the web page) is taken from yesterday's date. No log is loaded at start up. You can either type in a period (one or two digit representing month, then hyphen, then four digits representing year) or select year then click month in the data picker (this is produced by the bootstrap software MX uses, it is not the date picker provided by your browser), when you click on the month the box is updated with selected period.
  • Next to it is a "Load" button. Press it and the Cumulus MX engine will generate an application programming interface (api) table containing a maximum of 10 lines taken from the log file selected and it is sent in dataTables software format to the web page.

Web page for editing daily summary log only

There is no need to select this log, the Cumulus MX engine will generate an application programming interface (api) table containing just 10 (at most) lines from log file and it is sent in dataTables software format to the web page.

General points common to all 3 web pages for editing all logs

For all 3 web pages

The application programming interface only takes a copy of just 10 (or less if not all ten available) lines from log file and sends those to the admin interface.

The Refresh button will get the 10 lines currently displayed from the MX engine again. Useful if the log file is being updated by another process, or you wisely took a back up before you started editing, you have mucked up the editing, so you replace the original file with the backup and this button gets you access to the ten lines from that replaced copy. For the dayfile.txt, of course MX does a backup for you every day, so you could use that to replace a mucked up edit.

The dataTables software that shows the log on the web page includes pagination. The full (up to) ten lines received from the Cumulus MX engine are shown at a time.

  • There is an icon to move to first page
  • An icon to move to previous page (if there is an earlier page)
  • A set of numbers, click a number (not all shown) to go to that page
  • An icon to move to next page (if there is a later page)
  • There is an icon to move to last page

When you choose another page, the api has to fetch the (up to) 10 lines for that page. These replace the dataTables insert you saw before


  • When you select a line, two buttons are enabled:
    1. Edit - rather obviously brings up a dialog (known as modal by the altEditor software that generates it) showing all fields and letting you change the contents of most individual fields. Scroll down and in the footer are two buttons:
      1. Save will send the edited line back to the MX engine where the log file is then read into an array, and the relevant array element is replaced by the line received. After that the log file is overwritten from the amended array. After that the log file as a whole is converted back to dataTables format and returned by the api that delivered the table originally.
    2. Close. You can close this dialog in 3 ways:
      1. clicking that Close button
      2. Clicking the small x at the top right
      3. Clicking anywhere outside the dialog
    3. Delete - rather obviously brings up a dialog (known as modal by the altEditor software that generates it) showing all fields. Scroll down and in the footer are two buttons:
      1. The second Delete button on the modal will send the existing line back to the MX engine where the log file is then read into an array, and next MX deletes the relevant array element whose date matches the line received. At present, MX software in the engine does not check the times match! After that the log file is overwritten from the amended array. After that the log file as a whole is converted back to dataTables format and returned by the api that delivered the table originally.
    4. Close. You can close this dialog in 3 ways:
      1. clicking that Close button
      2. Clicking the small x at the top right
      3. Clicking anywhere outside the dialog

Be aware

  • The api expects the log file to contain all the fields defined in the version of MX that is being used.
  • Lines in the log files created by earlier versions of Cumulus (Cumulus 1 or MX) may have fewer fields
  • The way that MX has been written is inconsistent in the way it deals with fields that MX now expects but at the time that log line was created did not exist. When adding the missing fields, some are added with empty string as content and some are added with a single space as content. When the editing modal is opened it will display placeholder text where a missing field is empty, but if you don't see that, there is a space that you can't see, but may well muck up any number you want to enter there.
  • If you do edit a line, and then Save, then that particular line (but no others in the log file) will change its length to that used by the current version of MX and you will see additional trailing field separators (commas, semicolons, or whatever defined for your locale) in that line making it seem different from its neighbours. Again you may spot that inconsistency in content as some gaps may appear between delimiters.

Changing Settings

All settings are stored in Cumulus.ini, so when you stop and restart MX, it can continue without you entering settings again.

Adjusting the majority of them is best done using the MX "admin interface" screens, you will see that Settings is the penultimate option in the navigation bar, and it has a drop down for the various settings screens that are now described.

Settings Menu.png

For most screens a HTML form element is used, there is a Save button to send the contents of that form for processing, and that must be clicked (and an acknowledgement displayed that the form processing has completed) before the edited settings are accepted. To make it easy to change settings, such setting screens uses tick boxes, radio buttons, and drop down selection boxes, so the choices available are clearly laid out. These form based MX settings screens do some validation, and if you make an error the contents change to a red colour (red text and red boxes), that means invalid data is present which must be corrected before the form can be sent by pressing Save button. One or more items on the page will have an error message added by the form processing in those validation fail cases.

If you attempt to set these settings by directly editing the file where they are stored, there is a danger of either making a typo or of choosing an illegal value for a particular attribute.

There are however some settings that can not be found in any of the setting screens, for these you need to edit the Cumulus.ini file directly, and the referenced Wiki page gives details of which settings can only be adjusted by adding parameters in the file, and tells you what values are accepted for those attributes, and also explains some of the differences in the settings available between Cumulus 1 and MX.

Note that if you change settings, that some settings do not take effect while MX is running, while other settings do take effect instantly (I have not found a definitive list anywhere) - anyway, you may need to restart Cumulus MX to get the new setting picked up. When you exit MX, it saves the settings in Cumulus.ini, and when you restart it it uses the settings it reads from that file.


(Aside: The "Success" message (with all possible alternatives) is defined in CumulusMX\interface\lib\datatableseditor\dataTables.altEditor.free.js, I am unsure whether this is used with the settings pages, but it is possible you can edit that script to make messages like "Success" appear in a different language).

Station Settings

Each setting has a hint beside it (with a small 'i' for information before each hint). If you have used Cumulus 1, the layout and section headings will be familiar. No settings take effect until Save button pressed.

Like all the settings pages, the headings are collapsed and need to be clicked to see the items under them.

The table here for the [Station] section will explain how MX stores the selections you make here, and give a bit more detail about each item and the values it can take. Please note that, in the Options section, there are two settings that take effect immediately Save is pressed, but are not written to Cumulus.ini and will be reset when MX is restarted; these two exceptions are to use Debugging logging and to use Data Logging. The first adds extra output to the log file created in "CumulusMX/MXDiags" folder reporting in detail each output task MX does. The second adds extra output to the log file created in "CumulusMX/MXDiags" folder reporting in detail each input received from the weather station.

Station Type

Select the one that matches your weather station. If you are unsure, look at Supported Devices article because it explains about alternative brands.

Units

One feature of Cumulus is that it does not matter what units the weather station uses, every reading is converted to units you select here, and generally every thing Cumulus does is based on those units. There is one exception, which only affects those in USA; internally Cumulus uses Celsius for more calculations, if you have requested Fahrenheit, it is only used for entries in log files, web tags, and NOAA reports.

Davis Connection

As you guessed, here is where you enter settings for how Cumulus interacts with your Davis station, see hints by selections.

Davis WeatherLink Live

There are several sub-sections, see hints by inputs.

Ecowitt GW1000 Settings

For equivalent brands please see Supported Devices. See hints by selections.

Log interval

This is time selected from a drop-down between times when MX logs current values in the Standard log files and the Extra Sensor Files.

  • A shorter time interval here has advantage of a greater chance of logging daily maximum and minimum values, but leads to bigger files. Because the log only records spot values (not as one enhancement requested maximum and minimum since last log) there remains a good chance of missing the extremes when logging.
  • Conversely, choosing a longer time here increases chance of not recording extremes, but reduces the storage needs on the device running MX.
  • For some weather station types, this interval should match the interval at which the logger associated with the station logs readings, for some types it does not matter, and some station types don't have a separate logger.
    • Davis stations in particular can struggle to do a correct catch-up when MX is restarted if the intervals do not match, and that catch-up download can be of every log in the logger and take a long time.
    • Calculations like average temperature and wind run are based on every measurement processed by MX. If you don't run MX all the time, it can only process what is in the station logger for the times when MX is not running. An average will be skewed if some of the inputs to that average are based on rapid sampling while MX is running and some on infrequent logs when it is not running.
    • If the station logger interval and this log interval are set to same number, then entries in the log files will be at same interval when MX is running and when the Cumulus log entries are populated from station log entries. If they disagree, the Cumulus log will store lines at the station logging interval during catch-up on restart and at the interval set here when MX is running normally; you might not want your Cumulus logs to have such a mixture.
    • If the station log interval is too short, the station logger might not hold enough entries if the Cumulus user is not able to keep MX running continuously or if MX fails for some reason. This tends to be the reason why some users want their station logger interval longer; they need to be able to retain say whatever length of time they might be on holiday. So some people will set that station interval to longer than this Cumulus interval (for normal running, they wish to maximise the values that are logged).
Davis users

From version 3.7.0 if you use a Davis station, a warning will be output if the log interval set here does not match the Davis logger interval. The developer says:

This is just a warning. Its best to keep the logger intervals the same to avoid excessively long start-ups, but it isn't mandatory.

I have put the warning in because lots of people aren't aware they have the issue.

Some people may choose to run with a MX interval of say 5 minutes and a station logger of 30 minutes because they say need to cope with extended offline periods. That is a conscious decision though, and they can ignore the warning.

To change the logging interval on a Davis to match the Cumulus logging interval:

  • You can use the WeatherLink software if you have it,
  • or use a terminal emulator connected to the logger and use the "SETPER xx" command where xx is the log interval in minutes.

Log rollover

Cumulus 1 was originally designed to allow you to match meteorological practice of rollover at 9 am in winter and 10 am in summer (so all days are exactly 24 hours long even when clocks change), but other options can be set.

Location

In this section enter:

  1. Latitude
  2. Longitude
    • The above two are entered as degrees, minutes, and seconds, as this is most accurate way to specify an angular position (internally MX uses decimal degrees with 18 decimal places as 17 of those are needed for one arc second)
    • You may need to look these up, don't forget there are 3 numbers and a hemisphere option to select correctly for each
    • The longitude and latitude must be right to get sun rise/set calculated correctly.
  3. altitude (not used in standard Cumulus functionality, but must be right if you send an alternative pressure reading to an external site)
  4. Name
  5. Description
    • The name and description are combined with other fixed words on the standard web pages, so must work in that context, and you are advised to look at the generated web page at CumulusMX/web/index.htm to check those added words.

People often struggle with these settings. However, these are not settings you have to get right first time, so a mistake is a learning experience.

Options

You will probably find it hard to find this section as it is some way down on the Station settings web page, but it is an important set of settings.

Some offer choice from two options summarised in the table. The others are described below table.


Label Not ticked Ticked
Use bearing zero when calm Reports bearing from weather station Takes into account if wind is blowing, if it is not, the resting bearing is invalid
Calculate average wind speed Use wind speed from weather station Use rolling average of next selection to smooth out individual spikes
Use speed for average calculation Use each 'latest' reading from weather station be it a gust or not for above calculation Use the average wind speed output from weather station as input to rolling average above
Use 100% for 98% humidity Report the humidity reading from station, even if station never reports 100% Assume if station says 98 or 99, MX should report 100%
Calculate dew point Use dew point from weather station (it can be calculated using lots of different formulae), if it is available Use the formula MX has selected for calculating dew point (not same as Cumulus 1), select if weather station does not calculate dew point
Calculate wind chill Use wind chill calculated by weather station (different formulas can be used) Use the formula MX has selected for calculating wind chill (not same as Cumulus 1), select if weather station does not make its calculation available to MX
Synchronise station clock (DAVIS and INSTROMET only) Don't worry if weather station clock gets out of sync with MX time. Once a day try to ensure that weather station clock and MX time differ by less than 1 minute
Cumulus pressure trend names Use what weather station reports as the pressure tendency Let MX work out whether pressure is steady, rising/falling slowly, or rising/falling quickly, select if weather station does not make pressure trend information available
Davis VP 1-minute barometer update Accept if weather station only updates pressure reading and trend information at quarter hour intervals Force Davis station to update pressure reading and trend information every minute
Round wind speeds Use wind speeds to the accuracy implied by the number of decimal places resulting from the conversion of reading from station to your chosen units Express wind speeds in integers for your chosen units, rounding away decimal places resulting from unit conversion
Ignore LACROSSE clock Use the time supplied by weather station when writing to standard log Use the time supplied by device MX is running on when writing to standard log

Extra Sensors - any sensors whose readings are logged in the standard log file are not extra. You tick here if you want Cumulus to log sensors using the extra sensors log file. Both log files are known as monthly logs in that a new log is created for each new month. (This is nothing to do with the special log that Cumulus 1 maintained for multiple temperature and humidity readings, as it is not available in MX).

Debug logging is used to switch on extra debugging information (about what Cumulus does in its processing) in the MXDiags log file. It only remains on until either it is turned off here or Cumulus is closed. Use a parameter -debug for adding debugging when you start Cumulus if you want the debugging to persist.

Data logging is used to switch on extra debugging information (about interaction between weather station and Cumulus) in the MXDiags log file. It only remains on until either it is turned off here or Cumulus is closed. Use a parameter -logging for adding data logging when you start Cumulus if you want the data logging to persist.

Stop second instance Invokes code that stops you having 2 copies of MX running at same time and conflicting.

Forecast

See tips by each setting.

Solar

See tips by each setting.

Annual Rainfall

This allows you to report seasonal rainfall, i.e. what is reported as this year's rain starts on the first of September say, rather than 1 January. If, you don't choose January, there is a difficulty with the year-to-date and year to which it applies, because these are always applied January to December and will therefore cover parts of two different seasons! The advice is to review these settings every month and check whether you want them to still apply or need to revise them.

Graphs

See tips by each setting.

Internet Settings

This has a lot of similarities with the Cumulus 1 settings, except that this screen only covers what was on the main tab in Cumulus 1. Again there are hints, MX has more options than Cumulus 1 had, and some defaults are different in the two flavours. No settings take effect until Save button pressed.

Include graph data files.PNG

In Cumulus 1, you had one setting to upload the standard files and that included web pages, moon image, and the graphs. MX has separate selection settings for the uploading the standard files (web pages), the include graph files (JSON files used by charts), for generating the moon image, and for uploading the moon image. See image where the graph file option is arrowed.

All settings entered here are stored in Cumulus.ini.

All retain their settings when MX is restarted. No settings take effect until Save button pressed. This includes FTP logging in Web/FTP settings section, (it is not in this image as it is further down), which is used to add additional information to MXDiags about standard File Transfer Process actions; it will persist across stopping and restarting MX, so you must switch it off here (or with MX stopped within Cumulus.ini FTP site section.

Each section for an external web site has a number of parameters to set, once set MX will automatically upload to that site the weather values in the correct format. Windy has been added to the external web sites that can be automatically updated. No settings take effect until Save button pressed.

The main new feature within this settings page is a Custom Http section. Within here you can define commands to be executed either at some multiple of seconds interval, and/or at a selected intervals in minutes, and/or at end of day (in EOD sequence shown below, the Custom HTTP is run before external programs are run, and that is before upload of Extra Files at EOD). In each of these you can use web tags to supply values for parameters to the command. Typically this would be used to send information to a remote web server. Here is a Custom HTTP example

https://the_URL_here/your_api_here?winddir=<#avgbearing>&windspeedmph=<#wspeed>&windgustmph=<#wgust>&tempf=<#temp>&rainin=<#rhour>&baromin=<#press>&dewptf=<#dew>&humidity=<#hum>&uv=<#UV>"

No settings take effect until Save button pressed. You need to turn on enhanced debug logging to see any confirmation that the http has run:

2018-07-21 16:05:00.821 Custom HTTP Minutes update
2018-07-21 16:05:01.037 Custom HTTP Minutes response: OK: ok! (24.2)

Extra Web Files

This is an extension of the Cumulus 1 facility on the "Files" tab of its Internet Settings. How to use these settings is explained for both Cumulus flavours on this wiki page, MX has an extra "end of day" (EOD) option, but otherwise you fill it out exactly the same way. Settings in one table cell are stored when you click in another table cell. Although, there is no Save button as clicking in another cell stores previous edit, the "Enter" button on many keyboards can optionally be used to be sure you have saved all edits made before you leave the screen.

Although to tick both real-time and EOD seems nonsensical, MX will let you for any selected file(s) do the processes and uploads at both intervals. I don't see why you should do that for normal running, but you might tick both to test a template without waiting for EOD, and after it has been processed once, remove the unwanted real-time tick, so from then onwards it just happens at EOD.

If you have moved from Cumulus 1, and are therefore using an existing Cumulus.ini, these screens may be partly pre-populated, despite that you might need to:

  1. change some paths in local column, as you may be referencing some files moved when you installed MX
  2. untick one column, and tick another, now that EOD is an option for the timing as well as real-time and normal logging/ftp interval.
  3. edit some templates (local files) where the process column is ticked because of Web tags differences meaning that some output modifiers are interpreted differently.
  4. you should not need to change remote file names, providing that you have not changed any directories on your web site as (like in Cumulus 1), for extra files, the remote path/file name required ignores any directories specified on Internet Settings screen for FTP settings.

Calibration settings

This is identical to Cumulus 1 screen functionality, already explained in Cumulus 1 FAQ here and "Dealing with rogue measurements" in this wiki.

NOAA report settings

NOAA settings.png

This is identical to Cumulus 1 functionality, the various settings available on this screen are already explained for the settings file here.

Just a quick reminder here that while Cumulus 1 is case insensitive for the code for the different ways to specify a month, MX only accepts upper case ('MM' for digits, 'MMM' for 3 letter month etc.), read more about the naming here.

MySQL settings

Cumulus MX includes functionality not in Cumulus 1, and this is one example of a new feature. It is designed to automate updating of MySQL databases whose schema has each table based on one of the Cumulus log files. This MX feature was developed from this script for Cumulus 1.

IMPORTANT

  • The MX automatic standard updates described below only work if the database table they are updating has columns named exactly as defined in the standard update.
    • The order of those columns does not matter, both CumulusMx.exe and ExportMySQL.exe use column names in their INSERT IGNORE statements.
  • All the MX updates will work even if you have defined additional columns in the standard tables and update those in a separate process.
  • But when extra columns are added as at versions 3.6.0 and 3.7.0, the MX updates will NOT work unless you have added the extra columns introduced in those releases with correct names.
    • For those releases, the release announcement includes a script to run to add the extra columns needed for those (and subsequent) releases.

Brief background on SQL

It was IBM who first invented the concept of Relational Databases in the 1970s and they needed a language for all aspects of interaction with the new database and they called it Structured (English) Query Language. The brackets indicate the word that was later removed. You may find when SQL is talked about, it is either pronounced "sequel" (as if there is still a "E" after the "S") or the 3 letters are simply spelt out.

  • SQL is not Structured in the modern computing usage of that word concerning languages that can implement decision trees; instead the word structured here comes from the ability to make a query from a main query and a sub-query, although for some people it is a useful reminder that the key words must appear in a particular order.
  • SQL is not just for running Queries, it can give and revoke access permissions, create and drop tables, and do many more tasks than just query a database to get results.
  • SQL is a Language as it does have a set vocabulary, a defined sequence in which key words must appear, and it is used for describing tasks to a database.

A relational database has to satisfy a number of conditions, but the basic one is that all data appears in a table with rows and columns. The columns have a particular order, but there is no control over the order of rows, so you can't specify a row number, you either specify a primary key that identifies a particular row, or you specify a sort by the value in any column in a particular order and SQL then returns your selected rows in that sorted order.

Like happened with video recorders and browsers, there was a relational database war, and thus division in language adoption, in the 1980s between 2 big players IBM's SQL and Ingres' QUEL, each gaining popularity in different ways, but the latter lost out, with newcomer Oracle taking the lion share of the commercial usage of SQL soon so very widely adopted, it achieved world dominance. There were a number of minor players who implemented their own relational databases, and initially their own languages, but SQL obtained a ISO definition in 1985 and was then widely adopted, surviving the invention of the internet, and the move from mainframes to small devices.

MySQL is one of the rivals, but all versions of SQL are related and the dialect differences are comparatively minor compared to the commonality of the majority of the language. SQL is designed to be largely independent of how data is stored, so from 2000 as per newer standard for SQL, it now works with XML as well as relational databases. That said, because it can define creation of columns, some SQL code is dependent on how data is stored in a particular database system.

Implementation

Mono and .Net implement SQL capability, and this is utilised by Cumulus MX in the queries it generates. However, for SQL to work on your web site, you need to have a relational database available on your web server that accepts the MySQL that MX generates. It is worth saying that one difference between MySQL and the SQL standard is the former can insert multiple rows in one MySQL statement, but in the standard for SQL the specification says only one row can be inserted by one statement. Having made that clear, the SQL that is generated by the MX engine does only insert one row at a time, so the SQL MX generates is standard SQL.

MX does not include a database to install on your web site. Note, this does not mean your web server must have a MySQL database, as other products will understand the updating SQL (because of the dialect, commonality, and standard-meeting points I have just made), so the automatic updating should always work. One common difference between products is which data types available, so it is just possible that you might have a database that does not understand the column definitions in the MX option to create a table for you using MySQL dialect, so do be prepared in case you have to create each table yourself using a different method, the SQL to create a table is fairly simple, it even starts "CREATE TABLE table_name", but it is the list of columns with their data types and sizes that follows that is tricky if you don't know what is allowed on the database you are using.

Equally, the separate application, ExportMySQL.exe mentioned in its context below, includes SQL to create a table (with MYSQL data types) and add multiple rows in one instruction, so it is only able to work with databases that use MySQL (the clue is in the name of the application), or databases where the tables have already been set up so it is just issuing Insert Ignore SQL.

Mandatory section

  • Server Details - expand this drop down as it is used for essential information for any access to the database on your web server:
    1. Enter your host name or a IPv4 address for your web server. If you host your own server, it might be something like 127.0.0.1. It will be the same as you enter for host in the "internet settings" screen.
    2. Enter the port for communicating with database server e.g. 33106. (Your web server provider should tell you whether the port is this or another number)
    3. Enter the User Name for updating your database. (You may be able to set up user names on your database with different permissions, you need to state here one with updating permissions)
    4. Enter the password for updating your database. (This will be set up for each user name)
    5. Enter the name of the database that holds the tables you wish to update. (You probably have set this up, ask your provider if you need help)

Optional Sections

The remaining 6 drop-down sections are optional, you choose which you want to use, they appear on the screen in a different order to how I explain them below.

  • Some of the optional settings described below, allow you to choose which log file to use for such automatic updates and what to call the table uploaded to (uploads will not work before the required table has been created).
  • Alternatively, you can devise your own schema, create that table, and then write the SQL to update your table using web tags to supply all the values.
  • You need to turn on enhanced debug logging to see any confirmation that the standard or custom SQL has run. With enhanced logging you will see messages like:
2020-04-09 10:00:01.047 MySQL: 2 rows were affected.
Standard Daily Summary Table
  • 1.Dayfile.txt upload
    • This section is about uploading to a database table that contains one row for each day.
    • This feature takes the set of values that MX has just used for the line added to this log file at the end of the day, and soon afterwards inserts those same values into a new row (with columns named as per SQL example below) in a database table.
  1. If you don't have a table in your database for this upload (skip to step after SQL if you do), first
    • Choose Table name - the default table name is "Dayfile", but you can choose any other name
    • Now move down the screen and click the Save button, and wait for MX to pop up Settings Saved message
  2. Now find "Create database table" section below the Save button and click Create Dayfile.
    • MX will confirm when table has been created.
    • This will create the table using the following SQL (here using default table name) (the feels like columns were added in MX version 3.6.0):
CREATE TABLE Dayfile (LogDate date NOT NULL ,HighWindGust decimal(4,1) NOT NULL,HWindGBear varchar(3) NOT NULL,THWindG varchar(5) NOT NULL,MinTemp decimal(5,1) NOT NULL,TMinTemp varchar(5) NOT NULL,MaxTemp decimal(5,1) NOT NULL,TMaxTemp varchar(5) NOT NULL,MinPress decimal(6,1) NOT NULL,TMinPress varchar(5) NOT NULL,MaxPress decimal(6,1) NOT NULL,TMaxPress varchar(5) NOT NULL,MaxRainRate decimal(4,1) NOT NULL,TMaxRR varchar(5) NOT NULL,TotRainFall decimal(6,1) NOT NULL,AvgTemp decimal(4,1) NOT NULL,TotWindRun decimal(5,1) NOT NULL,HighAvgWSpeed decimal(3,1),THAvgWSpeed varchar(5),LowHum decimal(4,0),TLowHum varchar(5),HighHum decimal(4,0),THighHum varchar(5),TotalEvap decimal(5,1),HoursSun decimal(3,1),HighHeatInd decimal(4,1),THighHeatInd varchar(5),HighAppTemp decimal(4,1),THighAppTemp varchar(5),LowAppTemp decimal(4,1),TLowAppTemp varchar(5),HighHourRain decimal(4,1),THighHourRain varchar(5),LowWindChill decimal(4,1),TLowWindChill varchar(5),HighDewPoint decimal(4,1),THighDewPoint varchar(5),LowDewPoint decimal(4,1),TLowDewPoint varchar(5),DomWindDir varchar(3),HeatDegDays decimal(4,1),CoolDegDays decimal(4,1),HighSolarRad decimal(5,1),THighSolarRad varchar(5),HighUV decimal(3,1),THighUV varchar(5),HWindGBearSym varchar(3),DomWindDirSym varchar(3),PRIMARY KEY(LogDate, 
'MaxFeelsLike',   'decimal(4,1)' ,'TMaxFeelsLike',  'varchar(5)','MinFeelsLike',   'decimal(4,1)','TMinFeelsLike',  'varchar(5)' ) COMMENT = "Dayfile from Cumulus"
  1. With the table existing, all you need to do is:
    • Enable - tick here when you are ready for this action [using the schema (set of column names) in the SQL quoted above] to happen at end of day
    • Now move down the screen and click the Save button, and wait for MX to pop up Settings Saved message.
  2. Now, to populate your table with past rows:
    • If you are using Windows, open a command window in the folder where "CumulusMX.exe" is found. Type ExportMySql.exe dayfile
    • If you are using another operating system, send the following instruction to the terminal in the folder where you installed MX. sudo mono ExportMySql.exe dayfile
  3. At the end of the meteorological day, MX will now automatically run the SQL to add a new row with the daily summary values as mentioned at the start of this section.
Another daily summary table
  • 2. Custom upload - at rollover
    • In the previous option, you have no ability to vary the schema, it will update a column for Total Evaporation even if your weather station cannot calculate that. It will update columns for total hours of sunshine, highest solar radiation level, and the maximum UV in the day even if you cannot measure these. It will not record whether snow was falling or lying, or the depth of snow if you wanted to be recording those.
    • MX provides this alternative option, again doing an upload as part of roll over to next day (sequence shown below, the Custom EOD SQL is run after the day reset to new date, but before the dayfile.txt update with existing values and so before today.ini to yesterday.ini processing).
    • In this section you can specify the schema, and say which columns are to be updated with three selections:
      1. Save - a button after all option sections, until you click it any changes you make in this section have no effect
      2. A tick box to enable or disable this upload (so you can leave the SQL recorded, but stop running it when you like.
      3. The SQL you want to run, what you type in this small text box should include INSERT IGNORE (or REPLACE) to insert a row, or include UPDATE to change columns in a row that already exists, like any SQL it must include the name of the table, the columns to be updated, and the values you want to insert into the columns are either expressed as web tags, as SQL functions on web tags, or as a sub-query reading the values from somewhere else.
        • Here is an example of a suitable query that MX can process for you [note I have had to include some yesterday tags e.g. for primary key (<#metdateyesterday format=yyyy-MM-dd>, and I have used the SUBSTRING function at one point, but I don't have a sub-query in this example):
INSERT IGNORE INTO `test_daily_summary` (`MaxRainRate`, `TMaxRainRate`, `HighHourRain`, `THighHourRain`, `TotRainFall`, `SnowFalling`, `SnowLying`, `SnowDepth`,  `CumChillHours`, `LogDate`, `RollOver`,  `MinTemp`, `TMinTemp`, `HeatDegDays`, `AvgTemp`, `MaxTemp`, `TMaxTemp`, `CoolDegDays`, `LowDewPoint`, `TLowDewPoint`, `LowHum`, `TLowHum`, `HighHum`, `THighHum`, `HighDewPoint`, `THighDewPoint`, `GreatWindChill`, `TGreatWindChill`, `LowAppTemp`, `TLowAppTemp`, `HighAppTemp`, `THighAppTemp`, `HighHeatInd`, `THighHeatInd`, `MinPress`, `TMinPress`, `MaxPress`, `TMaxPress`, `HighAvgWSpeed`, `THighAvgWSpeed`, `StrongestWindGust`, `TStrongestWindGust`, `BearStrongestWindGust`,   `BearStrongestWindGustSym`,`BearDomWind`, `BearDomWindSym`,  `TotWindRun`) VALUES ('<#rrateTM>', '<#TrrateTM>', '<#hourlyrainTH>', '<#ThourlyrainTH>', '<#rfall> ', '<#snowfalling>', '<#snowlying>', '<#snowdepth>', '<#chillhours>', '<#metdateyesterday format=yyyy-MM-dd>',  '(1 * SUBSTRING(<#rollovertime>,0,2))', '<#tempYL>', '<#TtempYL> ', '<#heatdegdays> ', '<#avgtemp>', '<#tempTH>', '<#TtempTH> ', '<#cooldegdays> ', '<#dewpointTL>', '<#TdewpointTL>', '<#humTL>', '<#ThumTL>', '<#humTH>', '<#ThumTH>', '<#dewpointTH>', '<#TdewpointTH>', '<#wchillTL>', '<#TwchillTL>', '<#apptempTL>', '<#TapptempTL>', '<#apptempTH>', '<#TapptempTH>', '<#heatindexTH>', '<#TheatindexTH>', '<#pressTL>', '<#TpressTL>', '<#pressTH>', '<#TpressTH>', '<#windTM>', '<#TwindTM>', '<#wgustTM>', '<#TwgustTM>', '<#bearingTM>', '<#directionTM>',  '<#domwindbearing>', '<#domwinddir>', '<#windrun>');
    • Again before you enable this option, there is a facility lower down this setting page (under the heading Create database table) where you can type some SQL to be run immediately, that can create the table you want this option to update, (although it could even populate any table with historic data, it is only intended for a small query). I am using a table that already exists as I have used it for testing changes to my PHP scripts, so I did not need to create a table before I enabled the query shown above.


Standard Monthly table
  • 3.Standard log file upload
    • Cumulus starts a new log file for each new month, that is why this is called the monthly log.
    • This database table has one row for each line in any monthly log and therefore the same table contains all months.
    • This feature takes the set of values that MX has just added to the monthly log file, and soon afterwards if this feature is enabled, MX inserts those same values into a new row in a database table.
  1. If you don't have a table in your database for this upload (skip to step after SQL if you do), first
    • Choose Table name - the default table name is "Monthly", but you can choose any other name
    • Now move down the screen and click the Save button, and wait for MX to pop up Settings Saved message
  2. Now find "Create database table" section below the Save button and click Create Monthly.
    • MX will confirm when table has been created.
    • The SQL used to create the table is (columns marked NOT NULL have been in use for all Cumulus versions, the other columns have been added from various different versions)
      CREATE TABLE Monthly ('LogDateTime',        'DATETIME NOT NULL', 'Temp',               'decimal(4,1) NOT NULL', 'Humidity',           'decimal(4,1) NOT NULL',

'Dewpoint', 'decimal(4,1) NOT NULL', 'Windspeed', 'decimal(4,1) NOT NULL', 'Windgust', 'decimal(4,1) NOT NULL','Windbearing' 'smallint(3) unsigned zerofill NOT NULL', 'RainRate', 'decimal(4,$rainDec) NOT NULL', 'TodayRainSoFar', 'decimal(4,$rainDec) NOT NULL', 'Pressure', 'decimal(6,2) NOT NULL', 'Raincounter', 'decimal(6,2) NOT NULL', 'InsideTemp', 'decimal(4,1) NOT NULL', 'InsideHumidity', 'decimal(4,1) NOT NULL', 'LatestWindGust', 'decimal(5,1) NOT NULL', 'WindChill', 'decimal(4,1) NOT NULL', 'HeatIndex', 'decimal(4,1) NOT NULL', 'UVindex', 'decimal(4,1)', 'SolarRad', 'decimal(5,1)', 'Evapotrans', 'decimal(4,1)', 'AnnualEvapTran', 'decimal(5,1)', 'ApparentTemp', 'decimal(4,1)', 'MaxSolarRad', 'decimal(5,1)', 'HrsSunShine', 'decimal(3,1)', 'CurrWindBearing', 'varchar(3)', 'RG11rain', 'decimal(4,1)', 'RainSinceMidnight', 'decimal(4,1)', 'WindbearingSym', 'varchar(3)','CurrWindBearingSym', 'varchar(3)', 'FeelsLike', 'decimal(4,1)') COMMENT = "Monthly logs from Cumulus";

  1. With the table existing, all you need to do is:
    • Enable - tick here when you are ready for this action [using the schema (set of column names) in the SQL quoted above] to happen automatically.
    • Now move down the screen and click the Save button, and wait for MX to pop up Settings Saved message.
    • The upload you select here will happen every time MX creates a new line in the monthly log file, it might be every 10 minutes, but you may have configured a different interval.
  2. Separately, to populate your table with past rows:
    • If you are using Windows, open a command window in the folder where "CumulusMX.exe" is found. Type ExportMySql.exe monthly
    • If you are using another operating system, send the following instruction to the terminal in the folder where you installed MX. sudo mono ExportMySql.exe monthly


Table created from Standard and/or Extra Sensor Log files
  • 4. Custom upload - minutes interval
    • One way you could use this option, is to replace the monthly log file upload if you wanted to change the schema, by leaving out some columns if your weather station is not able to measure all the derivatives included in the standard schema.
    • This feature allows you to specify your own SQL for an upload to be repeated every NN minutes. Unlike the Monthly log file upload option you choose what schema (columns) are in the table that you are uploading a new row to and indeed exactly what SQL is used.
    • Apart from the need to press the Save button that follows all the options, there are 3 items just for this option:
      1. A tick box to enable or disable this upload (so you can leave the SQL recorded, but stop running it when you like.
      2. The SQL you want to run, it should include INSERT IGNORE (or REPLACE or UPDATE) to insert/replace/update a row, include (as all SQL needs) the name of the table, include the columns to be updated and include the values either expressed as web tags or derived from a sub-query.
      3. A drop down for the number of minutes between runs, the default is 10, but if your weather station updates less frequently, maybe you will choose 15, 20, 30, or 60 as the interval out of the 11 available in drop down.
Table updated at real time interval
  • 5. Realtime.txt upload
    • Cumulus MX can be set to recreate a file called Realtime.txt on a very frequent basis. The real time interval defines the time from the end of doing one real time update until the start of the next real time update. The file is recreated, in that unlike other log files, MX does not add new rows in each update, the file only ever contains a single line of values.
      • MX does not use this realtime.txt file in any of its supplied components, so that file by default is not available on your web server.
      • There is the ability elsewhere (Internet Settings screen) to upload this log file to your web server, should you wish to use it. The most common use is as a source for Ajax (JavaScript based) updating of web pages on the same very frequent basis.
    • The option being described here is an alternative to having that log file on your web server, instead MX in this option MX updates a database table, adding a new row at the same very frequent interval.
      • In this standard option, where you cannot specify which columns to include, MX will put into a database table row, the same set of values it would put into that log file on recreation.
      • It is important to stress, this database table has rows added, so it is not equivalent to the uploaded file that contains a single line.
  1. If you don't already have a table in your database with the right columns for this upload, first
    • Choose Table name - the default table name is "Realtime", but you can choose any other name
    • Now move down the screen and click the Save button, and wait for MX to pop up Settings Saved message
  2. Now find "Create database table" section below the Save button and click Create Realtime.
    • MX will confirm when table has been created.
  1. With the table existing, there are 3 steps left:
  2. Enable - tick here when you are ready for this action of creating new rows to happen automatically.
  3. Enter in the text box a retention string in format retainVal=NNN retainUnit=XXXX where NNN is a number from 1 to 3 digits long, and XXX is a time unit e.g. second, minute, hour, day, week, month, quarter, or year.
    • Because the updates are so frequent this database table grows very quickly, and you need to say when it should delete the older rows so the table never has too many rows. If you think about it, after a few days, you probably do not need to look at this very detailed level of values information within a day. In that case set retention to delete after a few days retainVal=3 retainUnit=day.
  4. Now move down the screen and click the Save button, and wait for MX to pop up Settings Saved message.
Another table unpdated at an interval measured in seconds
  • 6. Custom upload - seconds interval
    • This feature allows you to specify your own SQL for an upload to be repeated every NN seconds. This caters for when you want something like the values in "realtime.txt" but want to specify your own schema (set of column names) or your own interval between updates (independent of what has been selected for real-time interval). Like the other custom options, this might be because you have extra sensors or do not have sensors for all items in standard log file.
    • Apart from the Save button below all options, there are 3 items specifically for this option:
      1. A tick box to enable or disable this upload (so you can leave the SQL recorded in Cumulus.ini for when you want to use it again, but start/stop running it as and when you like).
      2. The SQL you want to run, it should include INSERT IGNORE (or REPLACE or UPDATE) to insert a row, the name of the table, the columns to be updated and the values you include in your SQL are expressed as web tags. You can have more than one SQL statement in this box (end each with semi-colon) so you might want to add a delete "DELETE FROM YourTableName WHERE LogDateTime < DATE_SUB(NOW(), INTERVAL 7 DAY);" after your update/insert command to replicate the retention option of the previous feature, in this case deleting rows over a week old.
      3. The number of seconds between runs, the default is 10, but if your weather station updates less frequently, maybe you will choose 40 or 60 as the interval. In theory the number of seconds specified here might represent anything between how frequently your weather station reports readings and several hours.

Alarms

This is identical to Cumulus 1 functionality, apart from using a new default location for the files "\CumulusMX\interface\sounds", the alarms available are already explained elsewhere in this wiki.

The alarms are shown at the bottom of the Dashboard page of the user interface. They also feed a set of Webtags.

FTP Now

This is similar to the option in the file menu of Cumulus 1 to do an update now. Depending on which build of MX you are using, the functionality varies. On latest build it does whatever updates are set up to happen at normal updating interval whether these are by FTP to your web site, or by copying files between local and remote filenames with path (although both could be on same device).

Editing the Admin Interface

Caution against editing Admin Interface

The general advice is do not change any files that are part of the MX package, they are a package and therefore there are interdependencies. Also updating to a newer release is more complicated if you have edited any files. The files as provided in the MX package are a compromise, for example they include reporting on solar measurements but not all weather stations include such measurements. Given that the admin interface is not shared with anyone else, it could be argued its look and content is not that important. In particular this interface is the only way to change settings, so do not change anything that stops those setting screens from working!

Finally, if you don't like the look of the admin interface, then why not look at your web pages, apart from settings, they should show you the same information, and you can edit your web pages to show information in whatever way suits you.

Caution when updating if you have edited Admin Interface

Remember, if you decide to download a new release to not overwrite any file(s) that you have edited, or your edit will be lost. It is less likely that a new release will change the interface files than other files, but some releases do change these files. Remember, each release zip contains all MX files, even those not changed since previous release. The release notice will usually give some idea of whether interface files have changed, but it may not list which interface files have been added, modified, or removed.

General points for editing

If you do decide to change any file, I suggest you maintain a back-up copy of the original elsewhere (so it can be gone back to) and you save the edited file under a new name (so you can't lose my edited file by installing a new release).

If you are editing files, use Notetab lite, notepad++ (for windows), or BB-edit on a Mac, i.e. use an editor designed for code, do not use a word processor, a Microsoft or Google editor or Dreamweaver or any other web editor. The encoding that should be used is UTF, if your editor does not mention encoding, it is the wrong sort of editor!

Changing the look

You need some understanding of Cascading Style Sheets (CSS) to do this, but all you need to do is to edit the relevant style sheet either in \CumulusMX\interface\css or in the relevant folder within the lib folder. You may feel that the default look of grey, black, and white, is either boring or does not offer sufficient contrast for you, perhaps you feel certain font sizes are too small, or you want to change the page background. Well, you can change the look, it is all defined in .css files. However, because MX makes use of standard libraries (bootstrap, datatables, alcapa etc.) there are a multitude of .css files used and it might not be easy to work out which one to edit. Each HTML page has links to a number of css files. You will probably make use of developer functions in your browser to inspect any element whose look you wish to change to see where its different properties are defined. It is better to make any such edits at a high level, rather than on any CSS just for that particular element. As always when editing, keep a copy of original so you can go back to it; keep a copy of your edited file, so installing a new release does not lose you edited file.

Removing Solar Figures

If your weather station does not have solar instrumentation you might wish to remove some of the display elements that relate to that. You need some understanding of Hyper-Text Markup Language to do this correctly, but here are simple examples.

  1. Navigate to \CumulusMX\interface folder.
  2. Open the file now.html in an editor designed for code (e.g. Notepad++ for Windows, Notetab Lite)
  3. Near the bottom of the file edit it by inserting HTML comment delimiters (opening after </thead>, closing before </table>) so it looks like this:
<table id="SolarTable" style="width:100%">
      <thead>
          <tr>
               <th> Solar</th>
               <th></th>
               <th></th>
         </tr>
      </thead>
  <!--
         <tr>
               <td>Solar Radiation</td>
               <td><span id="SolarRad">--</span></td>
               <td>W/m<sup>2</sup></td>
         </tr>
         <tr>
               <td>Sunshine Today</td>
               <td><span id="SunshineHours">--</span></td>
               <td>hrs</td>
         </tr>
         <tr>
                <td>UV</td>
                <td><span id="UVindex">--</span></td>
                <td></td>
         </tr>
   -->
</table> 

IMPORTANT NOTES:

  1. The above approach works on "now.html", but it does not work on other pages where table rows are dynamically created by an external script, so the existing rows in the table body are dummies whose content is ignored.
  • An alternative technique is to delete the whole table and any "<div> .. </div>" that surrounds only that table, that will work on all the HTML pages.

Adding derivatives not shown on the existing admin interface page

It is a JavaScript file \CumulusMX\interface\js\dashboard.js that reads the real-time file and inserts particular content into position indicated by values of the HTML attribute "id" on the admin interface screens. The standard \CumulusMX\interface\now.html does not include temperature trend for example, but because there is a temptrend: inp.TempTrend.toString() defined in the JavaScript file, you can easily add it to the "now" page by a simple insert of the middle row here:

<tr>
          <td>Outdoor Temperature</td>
          <td><span id="OutdoorTemp">--</span></td>
          <td><span class="TempUnit">--</span></td>
</tr>
<tr>
           <td>Trend</td>
           <td><span id="TempTrend">--</span></td>
           <td><span class="TempUnit">--</span> hour<sup>-1</sup></td>
</tr>
<tr>
           <td>Dew Point</td>
           <td><span id="OutdoorDewpoint">--</span></td>
           <td><span class="TempUnit">--</span></td>
</tr>

You can't add any derivatives into any table unless the value (for the derivative you want to add) is already defined in the related files.

There is a section of the support forum devoted to Cumulus MX interface customisation, so you can see what other people are doing. There is also another sub-forum for making suggestions on what you would like added to MX.