MX Basic info

This page contains some reference information about MX, that applies whatever operating system you run MX on.

It was correct at release 3.5.0.

This document was written for a past version of MX, so content may need updating to cover both old and current releases!

Library software

For most Cumulus users, this whole section can be skipped, but I have included it for those few users who have a technical slant and might want to understand more.

Cumulus MX uses library software (i.e. software written by others and made available by the provider and often also by other content delivery nodes or 'cdn') for a lot of the standard functionality. The library software for the admin interface and the separate library software for the standard web pages are both mostly included in the distribution zip, although some is used via a link to a cdn.

Many of the libraries included by MX are very obsolete. However, Mark Crossley, the current developer, said the following on 30 Sept 2019: "Chasing the latest versions of all the packages for the sake of it is a thankless task, and requires considerable effort to regression test each update. I am only updating packages when required to fix issues or for platform compatibility."

Just to mention the other side of this balance. It is difficult to code an addition to MX that works with these obsolete versions of libraries. All documentation provided by providers of the libraries relates to current versions of the packages (and what is documented to work now often will not work with obsolete versions). The documentation for the packages that are no longer supported is only available in archive sites if available at all.

Library Software for the MX engine

The distribution zip contains various .dll files and these are the libraries used by MX itself. The exact mix of libraries included has varied at various times, the list below is a snapshot of those included at the version that was investigated when this article was extended to include this section, and may not be right for the current MX version.

Devart

The two files used are both related to the database functionality of MX.

FluentFTP

As the name suggests, this is used by MX for controlling the file transfer functionality. This component was first introduced at version 3.0.0 build 3045 (adding more functionality than that available from System.Net.FtpClien, which was used at earlier MX builds), and FluentFTP.dll has been updated at some subsequent MX releases, see the announcements for details.

Linq

Language INtegrated Query is used to work with sequences of items and pick the ones that are needed, putting them into output format required. MX uses two files in connection with preparing output for Twitter. There is a third Linq file for other processing.

MQTT

When MX added capability to talk to other devices using the MQTT protocol, it added this component for that optional functionality.

Newtonsoft

This component is used for processing JSON strings. It is a very popular choice for developers, and used therefore very widely. However, SystemText has superseded it, so MX is using an obsolete method.

Renci SSH

This component is server connection software, it is what processes the host name, password, and so on.

SQLite3

This is used for all interactions with the weather diary.

HTTP

These files handle the optional HTTP functionality.

Unosquare

The EmbedIO file is open-source software that handles the web-sockets functionality of MX. The Swan file is open-source software that handles JSON formatting and threading of tasks in MX.

Library software for admin interface

As the following sections reveal, MX uses external libraries rather than writing its own code whenever possible.

• However, that does not mean MX is good at meeting development standards.
• MX only implements small parts of the functionality of most libraries, the minimum to make a feature work, not all the features available to make it work well
• MX does not use the latest versions of libraries
• MX does not attempt to obey guidance for good user interaction, and although validation is being added, many parts of MX do depend on user understanding what is valid
• MX does not make provision for screen readers and other accessibility aids.

Alpaca

1. Alpaca software is effectively a programming language extension to help people design forms like those MX uses for all its settings, and as a Cumulus user you really don't need to worry about it.
2. It is used for most settings screens. See http://www.alpacajs.org/ for more information. The latest version there is 1.5.27 released on 14 May 2019.
3. MX uses Alpaca Release 1.1.3 from https://github.com/gitana/alpaca which was first released on 15 May 2014. Although some individual components have been updated on that github URL, and elsewhere, MX has not incorporated these.

Bootstrap

1. Also known by some as Twitter Bootstrap which gives a clue as to its developer and to its origins as an internal tool for those building Twitter, that company still keep making updates as it is now the most popular styling library of all those available widely.

• The simplest way to think about this package is as a standard set of styling promoting easy responsive (means adapts to screen dimensions) web site design.
• To give just a few examples, it defines a standard way to represent buttons, form components, lists, navigation, and breadcrumbs.
• MX uses Bootstrap version 3.3.7, which is very restricted in what it offers
  • Bootstrap version 5 is available (http://getbootstrap.com), so MX is using an obsolete library
  • Bootstrap version 4.5.0 (Bootstrap 4 released as alpha in August 2015, beta in August 2017, and with fully working releases frequently from January 2018) was very widely praised for its improved functionality, and ability to work with latest jQuery and multiple modern devices/browsers.
• MX does not implement key features of Bootstrap like colouring text according to what it represents (primary, secondary, information, warning etc.)

dataTables

1. When MX sends out multiple lines of a log file to view or edit, the application programming interface (api) that transfers the information from the MX engine sends it in dataTables format for display on the web page in the admin interface.
2. Thus dataTables does all the work of providing the ability to present the data in a HTML table, the functionality to move between multiple pages needed (as MX sends only up to 10 lines of a log file at a time to the admin interface).
3. The free version of dataTables used by MX lacks the most useful functionality that needs a subscription licence. For example, its editing functionality requires a subscription.

altEditor

• This is an editing tool that can read what is in dataTables, create what it calls a modal (a pop-up dialog) where rows can be added, edited, or deleted individually.
  • MX when it added editing of log files at version 3.4.5 - Build 3069 (Friday 13 March 2020) adopted this software as it was free (although Mark Crossley said in his release notice: "The main thrust of this release is to add some log file editing capability to Cumulus MX. It works on all three log file types, but it is fairly basic at present. You can edit or delete lines in the files. The editing has to be done via pop-up dialog.
  • I only found two libraries that support JQuery dataTables editing, one is very comprehensive - but costs $$$ - the other is free. The free version does not currently support in-line editing of the table which is a shame.
  • If any web guru out there can come up with a better solution please post about it on the forum, or send a pull request.") (By the way, it is possible to provide in-line editing and make it work with the existing api interface, but making it compatible with the obsolete software used by MX is hard).
• The single line of fields that is result of an edit or deletion done on the modal is sent back via another api to the server (the MX engine in our case) and that then regenerates the dataTables in the state after whatever action was done, sending back again up to 10 lines for the same page as before.
• As it happens there is another JQuery dataTables editing tool, but it has not been maintained since 2012. It is found at https://github.com/NicolasCARPi/jquery_jeditable, but the documentation is now only available in an archive at https://web.archive.org/web/20200615000000*/https://appelsiini.net/projects/jeditable. It is designed for editing table cells, so it does not involve any pop-up dialog.

datepicker

1. Although modern browsers generally will generate a calendar type interface when they meet an entry field defined as a date, this date picker software ensures all MX users see exactly the same interface for date selection needed for both the standard log and the extra sensors log which are monthly log files (a new one is created each month). It is used for picking which standard (monthly) log or extra (monthly) log is to be viewed by selecting a month and year only.
2. It is also used for selecting individual days in the weather diary editor.

editable grid

• As the name perhaps suggests MX only uses this for the extra web files screens where you can make selections within a grid like interface.
• I suspect it could enhance some other functionality in the future.

handlebars

1. Put simply this is a simple HTML generator based on templates.
2. I have not found any file in the admin interface actually using this, but I am scared to delete it just in case it stops something working.

x-editable

• Put simply, this allows in-place editing of web pages using bootstrap.
• In MX it is used for the record editing screens where you adjust the extreme values.

jQuery

1. The admin interface uses version 1.9.1 of this JavaScript based library.
   • At the time of typing this, the current jQuery is version 3.5.1.
2. Of all the old versions of jQuery to choose, MX has picked the only version that the developers withdrew due to an error when they released it.
3. Version 1.9.1, has a serious error in its code, because the developers accidentally combined code from two significantly different versions when they created the release file.
   • This reveals itself in two ways:
     1. The error handling does not work.
     2. It tries to load another script that does not match.
4. Consequently, the developers quickly removed it, but it remained available from Contents Delivery Nodes, which is where MX has found it.
5. Not surprisingly, the authors of jQuery strongly advise all 1.9.1 users to move to a later version.
   • Unfortunately, there are interdependencies between all the library code used by MX, so you cannot simply update this component (see next library item).

Jquery Template

• This is also obsolete, and therefore will only work with old versions of jQuery (1.4.2 to 1.11.0).
• It was originally available from jQuery downloads, but they now offer jsRender.js for this functionality.
• It basically is used to bind the contents of objects (like array elements) into particular locations within HTML.

SteelSeries

MX uses a modified version of the steel series library made available by Mark Crossley for all the gauges (see dashboard and gauges pages of the admin interface) in MX.

Highstocks

The odd one out is Highstocks (that includes HighCharts)

• This is loaded from a Contents Distribution Node (CDN), but it is still pinned to obsolete versions of the basic script and its themes.
• This means that the Charts page in the admin interface will only work when there is an internet connection working to permit download of this software
• If you need to view your admin interface where an internet connection is not available:

Then you need to edit the interface file...

<CMX_Folder>/Interface/charts.html Change lines 20,21 from

<script src="https://code.highcharts.com/stock/8.0/h ... "></script>
<script src="https://code.highcharts.com/themes/grid.js"></script>

to

<script src="webfiles/lib/highstock/js/highstock.js"></script>
<script src="webfiles/lib/highstock/js/themes/grid.js"></script>

Library software for your web server

The webfiles/lib folder includes a number of software library items that are needed for the standard web pages included in the MX distribution.

This is fine if your web site is purely for the provided standard web pages, but if on your web server you also have web pages from third parties, or you have written your own web pages, then you may get conflict because all the library items used by MX are obsolete versions, and in one case MX uses a version of library software withdrawn by its originators due to compile error (so only available form some CDN who provide obsolete versions as its originators insist it must not be used)!

1. Highstock
   • At the moment, as hinted in previous section, there is an old version of Highstocks included in the webfiles/lib folder.
   • However, that is not used, instead (like admin interface) an old version is loaded from a CDN.
   • Be careful if you also load the current version for use on web pages not produced by MX, that the browser does not try to reuse your latest version and not recognise that MX wants an older version.
2. jQuery
   • Be aware that MX distribution is not consistent as different web pages use different obsolete versions of jQuery.
     1. gauges.htm includes <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script> to load version 1.8.2 from the Google Content Delivery Node
        • You will find at the end of this web template, if you bother to read it, some instructions about how you can change that CDN load, but the instruction is neither grammatically correct, nor understandable for me.
     2. trends.htm includes <script src="lib/jquery/jquery-latest.min.js"></script> to load version 1.9.1 of jQuery loaded from this badly named file, there is nothing latest about it.
        • Ironically, the old version of jQuery that has been selected by MX to be included in the webfiles/lib folder is the only old version that jQuery themselves withdrew, and warn you not to use because the error handling content is from a different jQuery version to the correct handling content making the two parts of the code incompatible. In other words, this version only works if everything is set up correctly and no error handling is invoked.
3. Steel Series this is treated as library software, as it is copies of that separate product by Mark Crossley.
   • Most of the files are exactly as that was in revision 0.14.13 made 30 January 2015.
   • gauges.js has one tweak for MX, it defaults to not having chart rollovers (for Cumulus 1, this file had default to showing the images of graphs that were uploaded by Cumulus 1).

The provided web pages

Setting up a web site is covered in this wiki page and the pages linked from there. I won't repeat that, but will try to explain below the MX context of the various files involved. MX will produce web pages locally even if you don't have a remote web site to display them on. You can view the web pages created in the web folder using a browser.

Cumulus MX provides a set of web templates, images, and json files, in \CumulusMX\web. The first of these are called templates (and have a 'T' at the end of the file name before the extension) because they include both text and web tags. MX will process these templates as it creates a web page (file name without a "T" in same web folder), during that processing, any text in the template is copied into new file without change, any web tags found in the template are parsed and the correct value is placed in that position in the output file. If you have a web server, then MX can process these files and upload them for you (by default using File Transfer Process) providing you specify the host, port, protocol, directory, username, and password, for the upload process to use. If you don't understand any of these terms, then this is not the place for explaining them, but generally if your web space is supplied by a provider, they will be able to tell you most of these settings, and you will choose the directory name. If you have set up a web server yourself, then you should know the required settings.

The web templates included are based on designs by Beth Loft used for Cumulus 1. Remember, Steve Loft (who wrote the original Cumulus software) said "They exist because they're our web pages, and they're really only included with Cumulus as examples of how the web tags work. It never occurred to me that most people would simply use the supplied examples instead of creating their own pages!"

The templates are written in fairly simple Hyper-Text Mark-up Language, designed to help people see how to write their own web templates. Here is a list of web templates provided:

• indexT .htm
• todayT.htm
• yesterdayT.htm
• thismonth.htm
• thisyear.htm
• recordT.htm
• monthlyrecordt.htm
• gaugesT.htm
• trendsT.htm

All these templates (except gaugesT.htm and trendsT.htm) include a table for showing values and styling that gives a graded background colour. The tables include a navigation row with links to the other pages in the set. That navigation line fixes the width of the table, and you will realise it was designed in the days when all monitors were a standard shape. Therefore the standard web pages as provided cannot adapt to the range of devices we use for viewing web pages nowadays. There are a selection of alternative web page sets available on the User_Contributions page, and some of these are responsive and adapt to the width of the device they are being viewed on.

The gaugesT.htm is a template similar to SteelSeries Gauges although the latter is designed to work with a range of software and the former is specific to MX. As supplied in MX if you mouse over the provided gauges appearing on your web site you will see a box with figures, not a graph as is seen with the general steel series gauges, but there are some other differences such as how the figures are supplied for the displays. The remaining template trendsT.htm, creates a structure that can display graphs. The data for all the graphs that can be displayed is contained in the various json files in \CumulusMX\web, these files are also processed by Cumulus so latest values are added, and then uploaded so the web page produced by this template can use them.

The image that is provided in \CumulusMX\web is MoonBaseImage.png, MX can be set to use that to generate (on MX start-up and on the hour) "moon.png" which it then can FTP to your web (also on the hour).

To set up your web server for the first time

When you first want to use Cumulus web pages on your web server, you need a number of static (unchanging) files to be put onto your web server. The web pages that MX uploads for you reference that static files and will not look right without them. The files that only have to be uploaded once are found in \CumulusMX\webfiles and its sub-folders. You don't create a folder called webfiles on your web server, but you put the files and sub-folders into position relative to where MX will upload the htm files.

To do this, you will must invoke a FTP process outside of Cumulus, MX does not include any functionality to do this one-off upload for you. The filezilla client is a popular choice as it has probably the most friendly graphical user interface, although other software to do this is also available. You may prefer a tool that lets you do the uploads from a command line without requiring working with a graphical interface.

1. The static files to be uploaded include the standard styling file \CumulusMX\webfiles\weatherstyle.css which you place in the directory specified for the uploads.
2. Next you have three sub-folders, each of those sub-folders need to be replicated within the directory specified for the uploads.
   • For example \CumulusMX\webfiles\images\picture.jpg will be stored in a "images" sub-directory of the upload directory and is used as the background image for web pages.
     • There is nothing to stop you creating your own "picture.jpg" (instead of uploading the supplied one) and then Cumulus web pages will use that for the background image on each page.
   • Similarly \CumulusMX\webfiles\js\cumuluscharts.js needs to be stored in a "js" sub-directory of your upload directory (this is the script that allows you to change the chart shown on the trends page and uses the appropriate json file to populate it with data).
3. The "lib" sub-folder contains further levels of sub-folders all to be replicated on your web site.
4. The trends.htm web page also loads some library software from an internet Content Delivery Network (cdn) to invoke the JavaScript based Highstocks library.

Operating a web site with uploads from MX engine

The standard web pages

• If you want to operate the 'standard' web site, then just the same as with Cumulus 1, you will need to upload the contents of the webfiles folder from the zip file (don't upload the containing webfiles folder itself).
  • Note that the MX web files are not the same as the ones for Cumulus 1, so make sure you upload the MX files if moving from Cumulus 1 to MX.
  • The standard gauges are now the SteelSeries gauges. The default versions do not display a graph when you hover over a gauge as happened when you added the stand-alone Steel Series gauges to Cumulus 1.
  • The trends web page in Cumulus 1 relied on that software generating graphs as images. In MX, the software generates files with time and value pairs, these are stored in json format, the trends page then uses a library package (Highstocks) to draw graphs from those data pairs.

Alternative ways to obtain web pages

You can choose to use some of the alternative web pages available from third parties and described on User Contributions page.

Using your own web pages

• Of course you can use your own web pages, instead of the standard ones. Assuming they need to include figures that are available as web tags, there are three alternative ways to implement this:
  1. MX can process template files with a HTML structure and those web tags in the structure where values are required just as it does with the standard templates, and MX can upload the resulting web pages at either the real-time interval, the standard interval, or after end of day. All of this is covered on the Customised templates page in this Wiki.
  2. MX can process a file with a string of web tags mirroring the realtime.txt option in MX, and upload the resulting file so your web pages can use JavaScript for a one-off insert of the values or an Ajax routine to update the web page at a fixed interval.
  3. Alternatively, you can use template scripts processed locally by MX that don't create web pages, but are uploaded by MX at either the real-time interval, the standard interval, or after end of day. These scripts simply initialise script variables with values obtained from web tags. You then independently have a set of web pages resident only on your web server (they don't exist where you run MX) using a combination of HTML and script content that bring in the script(s) with the variables by the appropriate syntax. All of this is covered on the PHP web tags page in this wiki. As it suggests there, you might therefore have several files processed by Cumulus MX at these different intervals, converting the web tags into script variables, and then use AJAX (JavaScript that may use json format to bring in the variables) or PHP (using 'require_once 'filename'; syntax) to put those variables into a web page.

You may find this wiki page useful for understanding more about the different script languages.

MX End of Day Process

I have added this section, because this process has given me some headaches.

My version

If you write custom SQL, or have a template being processed at end of day, then what I find strange is that web tags related to system date report the new date, but other web tags report weather derivatives from the old day. Put another way, all the date related tags change at start of rollover, but all the weather related web tags change at end of rollover.

However, it is not quite as simple as that, the month and year are reset before the extra web files are processed (so they cannot use monthly web tags at end of month, nor yearly web tags at end of year). The complication is that in the extra web files you can use <currentlogfile> (and from build 3087 <currentextralogfile>) and these pick up the old month/year. Now you see why I found it hard to digest, and why I wanted to write it here to make it easier for others.

If you use Custom SQL and therefore have to quote web tags, the SQL should use monthly and yearly web tags related to previous day, but all the weather tags it uses must be those for current conditions or today. Yes it is confusing.

As part of the so called "end of day" process, MX (just like Cumulus 1) creates a start of day back up in the daily sub-folder. So on first day of a new month (and new year sometimes), the backed standard log files (and extra sensor log files from build 3087) are for the month that has just started, there is no back-up of the old month.

The official version

Mark Crossley says the MX day reset does this (at version 3.5.x)...

   Reset midnight rain
   Entering Day Reset (message about current day of month, at this stage web tag <#metdate> changes to new date)
   Day Reset (message about date ending, time shown as 00:00:00 because time not defined, not because it is midnight, it might be 9am or 10am)
   Run EOD custom SQL
   Save dayfile entry (uses what is still in today.ini that includes old date, i.e. what is now in web tag <#metdateyesterday>)
   Write monthly & yearly file entries
   Write any new daily extreme records
   if day of month = 1 then: copy month.ini to saved file, reset monthly figures
   if day of month = 1 and month = 1 then: copy year.ini to saved file, reset yearly figures
   Copy todays high/lows to yesterdays
   Reset todays high/lows to current
   Write today.ini & yesterday.ini
   Create NOAA reports
   Execute user daily external program
   Process Extra EOD files 

But independent of above EOD thread that occurs on the rollover hour, the normal interval and hourly processes thread is seeking to run at same time, whether that happens at same time depends on processing capability and whether it can process multiple threads.

What actually happens in above list depends on your settings, and if your FTP interval is synchronised with the logging interval.