Cumulus Wiki

Migrating from Cumulus 1 to MX

Revision as of 10:54, 21 June 2020 by Sfws (talk | contribs) (Created page with " This article compares functionality between Cumulus 1 and MX. This article also covers moving from Cumulus 1 to MX. = Comparing Cumulus 1 and MX = (The next bit retains w...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

This article compares functionality between Cumulus 1 and MX.

This article also covers moving from Cumulus 1 to MX.


Comparing Cumulus 1 and MX

(The next bit retains wording by Steve Loft, but this is a new location for it)

Cumulus MX aims to be as compatible with Cumulus 1 as possible with 3 exceptions: 

   Like Cumulus 2, MX separates:
       the engine (reads weather station, calculates derivatives, creates web server for user interface, and sends updates to various external web sites), and the
       administrative (shortened to admin hereafter) interface (displays basic information, allows you to vary settings, contains editors for highs and lows and for log files).
   Like Cumulus 2, MX runs on Linux and OS X as well as (like Cumulus 1) Windows.
   Like Cumulus 2, MX learns from some of the mistakes made in early part of design of Cumulus 1, that limited further development.

Because the development environment for Cumulus 1 is no longer available, it cannot have any extra functionality added. MX uses standard language C# (pronounced "C Sharp"), an object-oriented programming language created by Microsoft for use with the .NET Framework. It also works with Mono an open source implementation of Microsoft's .NET Framework based on the ECMA standards for C# and the Common Language Runtime.

Features and functionality

Initially MX, as written by Steve Loft, lacked a lot of features that were available in Cumulus 1, but subsequent developments headed by Mark Crossley have now added the majority of the missing features. I am trying to get a list of Cumulus 1 features that are not in MX (or are implemented differently) in the forum at Features in C1 that are not replicated in MX, so anyone contemplating the move knows what they will lose out on. Although MX now generates a Moon image, it does not generate all the chart images (replaced by graph drawing from json data) nor the wind images that Cumulus 1 generated.


There are also many features that have been added to MX that were either on the now lost list of enhancements for Cumulus 1 that never got implemented, or are extra functionality to reflect recent changes in weather station features. All quotes below are from the release notes issued by Steve or Mark. Only significant changes to functionality are noted below (for fixes see support forum for full release announcements). There is also version change information here for MX versions.


Version 3.0.x

   build 3023 - you can now control the output format of <#tomorrowdaylength> using an entry in strings.ini like this example:

[Solar] MoreDaylightTomorrow=Il y aura {0} minutes {1} secondes plus la lumière du jour demain LessDaylightTomorrow=Il y aura {0} minutes {1} secondes moins la lumière du jour demain 

   Build 3025 - new MySQL (6 options) and custom HTTP uploads (can invoke a PHP script) facilities
       Also introduces a second pass to read archive records in catch up for Davis stations only
       Debug logging, diagnostic data logging, and ftp logging can now be set in the UI
       Changes for reading from Fine Offset and Davis stations
       Improved console messages at start up to indicate whether station has been connected successfully
       Makes sure dayfile.txt entry is always logged to MXdiags to help in case has problems writing file
       'Stop second instance' option now implemented (there were problems with this, see later versions)
       Graph periods can now be configured

   build 3035 - archives the month.ini and year.ini file at the end of the month/year as monthYYYYMM.ini and yearYYYY.ini.
   Build 3041 - Support for FTP over SSL/TLS (FTPS) - enable in Internet Settings
   build 3046 - added weather diary database (Note the MX diary file is different to the Cumulus 1 diary file).
   build 3047 - Web token parser updated to cope with html tag characters "<>" in the format string (see web tag page).
   build 3049 - This build enables ability to upload data to Windy.com.

= Version 3.1.1

This release is mainly part of my attempts to add some of the Cumulus 1 features that are missing from CMX. 

   build 3054 - Adds a Current Conditions editor and an All Time Records editor to admin interface
   Build 3056 - Fix for the All Time Records editor

Version 3.2.5

adds editors for files that track extremes previously missing from MX 

   Build 3061 - that completes all the missing record editors from Cumulus 1

Version 3.4.0

The big change for this release is adding historic data "catch-up" for Davis WeatherLink Live devices. 

   Build 3064 - fixes bug in Monthly Records editor for dry/wet periods.

Version 3.4.5

This release continues attempts to add some of the Cumulus 1 features that are missing from CMX. 

       Build 3069 - Adds Editors for: Dayfile, Monthly Logs, Extra Logs

Version 3.5.0

adds the generation of a Moon phase image, and the ability to push data to MQTT brokers.

Version 3.5.4

Adds "Feel like temperature" as alternative to "apparent temperature"

The difference in how A Cumulus 1 user and a MX user see current values and change settings

Cumulus 1

Cumulus 1 was an all in one application, it both read the data from the weather station and provided the user interface for you to see the derived data and change the settings.

Accessing Admin Interface

MX is different, it consists of a stand-alone 'engine' which performs the reading and logging of data, uploading to a web site etc. This 'engine' is a command-line/terminal/console application which has no user interface. The separate admin interface is provided by virtue of the engine acting as a web server.

When you successfully start MX, the engine is running until it is terminated. You can view the admin interface by typing the URL of the built-in web server into your browser, either on the same machine, or on a separate machine sharing the same local network. The default URL if the browser is on the same machine as MX is http://localhost:8998/.

If you are using the browser on a different device on your local network to the device running MX, you cannot use local host. Instead you specify a IPv4 address, that is listed in your router (might be called a hub) for the device running MX, this IPv4 address will look like '192.168.y.z' (where y and z are numbers that vary between implementations).

Equally, if "localhost" is already in use for another web server (that you already run on your device), you will need to use the correct IPv4 address as above, even on the same device.

For security reasons, the admin interface should not be accessible via the public internet.


Operating Systems for Cumulus 1 and MX

MX runs on: 

   Microsoft Windows operating system (Cumulus 1 only runs on this)
       To run MX on Windows, you need .NET installed which is included on Windows 7 upwards.
   Unix derivatives Linux and Mac OS X.
       To run MX on the additional platforms, it requires the Mono runtime, and you will need to install this

As mentioned in the comparison between Cumulus 1 and MX section earlier, Mono is the open source version of the propriety .Net. Both are sponsored by Microsoft.


Configuration, Log, and Web files

Both Cumulus 1 and MX both use the same basic files: Cumulus.ini, dayfile.txt, today.ini, month.ini, year.ini, alltime.ini, monthlyalltime.ini, indexT.htm, todayT.htm, yesterdayT.htm, thismonthT.htm, thisyearT.htm. But there are differences in content in all these, and the web pages not mentioned there are totally different.

The configuration file Cumulus.ini has some differences between Cumulus 1 and MX, see that wiki page for more information, but essentially each flavour will ignore the parts they don't understand, and add the additional parts that they need but the other did not use.

While Cumulus MX can read the today.ini produced by Cumulus 1, you probably need to edit the date format in the date at the start of that file before Cumulus 1 can read a today.ini produced by MX. MX uses ISO format dates with year first, Cumulus 1 uses the date format defined on your PC system that might have year last. Look up in this wiki pages for the Log Files index page or the individual file pages just referenced to see the differences between file content, and what you may need to edit to use Cumulus 1 files with MX.

Both Cumulus 1 and Cumulus MX supply a number of templates that are processed into web pages, although the end produce web pages are (except for trends.htm) identical, the templates are not interchangeable. However, there are files that Cumulus 1 uses (for example it uses several image files for the trends web page and a weather diary in a XML file) that are not used by MX and also many files that MX creates (for example json files and a weather diary in a SQlite file) that were not part of Cumulus 1.


Changing from Cumulus 1 to MX

Remember, that depending on the operating system, MX may require you to install extra components, such as Mono, not included in distribution, see Cumulus MX article for details.

Files to copy from Cumulus 1 into where you are installing MX

MX requires all files from "data" and "Reports" folder created by Cumulus 1. You also need "strings.ini" (if you use that), and "Cumulus.ini", plus any other tailoring set-up files, batch processes etc. you might use.

You may be installing MX on a different device to the Windows PC that Cumulus 1 ran on, in that case follow first alternative below.

If you are installing MX on a PC that has been running Cumulus 1, then there are actually 2 alternatives to choose between: 

   Just create a new directory (recommended name CumulusMX) and unzip the contents into it. Then copy over your existing data files and your Cumulus.ini file, and any other configuration files that you may have created (e.g. strings.ini, twitter.txt etc). In this case you will need to edit that copied across "Cumulus.ini" so any lines that referenced the old folder are changed to reference the new folder, and you may need to edit a few other items either now, or via the settings functionality in MX user interface. The big advantage of this approach is that anytime you are not running MX you can go back to Cumulus 1 and let it run from where it left off (subject to availability of past data in your weather station).
   Alternatively, to run Cumulus MX with your existing Cumulus data, take a back up copy of your existing Cumulus directory, and then unzip Cumulus MX into the original Cumulus folder. This saves you from copying Cumulus 1 files, and removes the need to edit any reference to folders. However, using same folder stops you going back easily to Cumulus 1 if you have an issue, because MX changes the date formats in some files, so that Cumulus 1 can no longer understand them.

Editing files

Cumulus.ini

   If your "Cumulus.ini" was actually called "cumulus.ini" you should rename it to start with a capital letter.
   Please see Cumulus.ini#Parameters_changed to see the list of 2 parameters that must always be changed in this configuration file. They are also mentioned in next 2 sections here.
   There are some differences in how the contents of this file are used in the 2 flavours, one to remember now is to check the "StartDate=" line in the '[Station]' section is correct for your earliest data before you let MX read this configuration file for first time, as MX uses that to find the first log file to start reading from, it will ignore any log files for earlier dates; whilst Cumulus 1 always looks at all log files it finds in the data folder.
   There are several other differences in settings stored in "Cumulus.ini", but you can address those later through the MX settings screens.

Log files

   See individual log file pages already referenced (alternatively access from Log Files index page) where there is more information for how to edit your .ini files to work with MX.

Station connections

If your weather station used a port to connect to Cumulus 1, that port was set on the settings screen as a number and stored in Cumulus.ini in the station section as Port=n. In Cumulus MX, as it runs on various operating systems, the port is specified using text (instead of a number), again you select it within settings, on Station settings screen, but that is stored within Cumulus.ini in the station section as Comport=tttttttt. If your old number was 3, and you are still using Windows, the new setting would have value of COM3, for other devices it might be /dev/ttyUSB0.

web pages

There are differences in how to upload the standard files between Cumulus 1 and MX.

It is best to work through the Internet settings screen in the MX admin interface.

Consequently, there is a difference between the entries in Cumulus.in, remove the IncludeSTDImages=1 used in Cumulus 1 and replace it with IncludeGraphDataFiles=1 used in MX.

If you have been using the Cumulus 1 supplied web pages, you will find they do not work with MX, there is a new set provided with MX that work slightly differently. 

   Cumulus 1 uploads a number of images, these include a couple of images that combined show the moon phase, and a number of graphs are also uploaded as images (used on Cumulus 1 trends page).
   Cumulus MX does not upload any images (prior to Release 3.5.0, from then on it uploads a moon phase image), instead it uploads a series of .json files that hold time and value pairs that can be used to draw graphs. (Hence the difference in settings mentioned above).
   The gauges page provided with MX is based on Mark's implementation of steel series, so it is different to the old "Web Dashboard Components for FreeWX and FreeWX-Wi " that Cumulus 1 used.
   The other web pages look the same, and indeed are effectively functionally same.

What I did to Install MX

I have used Cumulus 1 for a decade or so, and been very happy with it, but I wanted to give MX a go without affecting my Cumulus 1 installation. Here is exactly what I did on my ex NHS Windows 10 Pro PC, step by step; I am hoping this list might help some readers. Can I just stress I downloaded version 3.4.5 (build 3069), there may be some changes that affect what I record below in more recent versions, I just noted what I had to do at that moment in time (March 2020). For example the latest version probably has some changes to folders and files included in the package.

Download and unzip

   I downloaded the CumulusMXDist3069 zip from the Current Release section on the downloads page. Although 3069 is no longer the latest distribution, still use the same link as it will always give you latest available.
   I unzipped the contents into a partition I use just for software downloads.
       You don't have to have somewhere separate to the installation, but many people will want to download on a separate device to the device where they will install MX
       Downloading to somewhere other than where you will install it, also makes it easier if you do any customisation of particular files and wish to keep copies of originals.
   I used a package that verifies the files it copies to duplicate the folder CumuluxMX within the zip onto the drive where I wanted to run Cumulus MX.
   As I was not intending to use the web pages that come with MX, so I selected to exclude in this verified copying everything from the "\CumulusMX\webfiles\*.*" folder, and its sub-folders in the download.
       If you intend to use the standard MX web pages, then you will need to:
           Ensure you do copy these files from download into where you are installing MX.
           You will also need to FTP all files and sub-folders within the webfiles folder (not the actual folder itself), to whatever folder on your web server you specify as the directory in the Web/FTP site part of the settings for MX.

Copying Cumulus 1 files into MX folder

The locale I used for Cumulus 1 is going to be the same I will use for MX (same PC!) so my copying across of my existing files was easy: 

   First, I copied my \Cumulus\strings.ini to \CumulusMX\strings.ini. This preserves any tailoring I have done of terminology.
   Remember, "\Cumulus\web" and "\CumulusMX\web" have different content, so don't do any copying between these.
   Next, I copied my existing Cumulus 1 alarm sounds in "\Cumulus" across to MX folder "\CumulusMX\interface\sounds" as these were referenced in my main Cumulus.ini file (I will edit that file next to reference the new location of these files).
   Now, I copy my \Cumulus\Cumulus1.ini (don't worry why my file had a "1" in its name, just remember that yours probably won't) to \CumulusMX folder as "Cumulus.ini". I then edit the MX "Cumulus.ini" file:
       In the [Alarms] section (your Cumulus.ini may have sections in a different order to mine) I edit all the parameter lines where the attribute ends in "File" to reflect their MX location in the sounds folder (there is no such folder in Cumulus 1).
       I checked that I had a "[FTP site]" section (yes, mine was named correctly with second word all lowercase, but if your has [FTP Site] you will need to edit that section title to put second word entirely in lower-case)
       I now had to decide whether
           I would use the Extra web files settings page in the MX admin interface to make the changes, in the local file column, necessary to define which web templates I wanted MX to process
           Or I would make the changes myself by editing the appropriate lines in this [FTP site] section. (I decided on this option).
       Although my web templates are stored in a folder "Templates for Cumulus to Process" outside the Cumulus 1 or Cumulus MX folder structure, I had to reference a new set of template files to cope with differences in the output parameters of MX web tags.
           Next where I want a transfer done only at end of day, I added lines like "ExtraEOD19=1" (when you run MX it will add a ExtraEODnn for each of the 100 possible values of nn that you have not created) and changed the previous "ExtraRealtime19=1" to "ExtraRealtime19=0" (I previously used realtime as that was only way in Cumulus 1 to ensure a template file was processed so it held correct values as close to end of day as possible, but it inefficiently also made huge numbers of unwanted transfers during the day). I had 9 such files being copied far too often, so those 9 changes will cause a huge reduction in processing load! I know I could also do this later using the MX admin interface, but it makes sense to me to do it as I am working through the file.
           I have a few files that are PHP scripts written as Cumulus templates; each PHP script has a number of PHP variables being set equal to a Cumulus web tag. You can find out about these scripts at Php webtags.
           I also have a few Cumulus templates that, like the standard ones I don't use, generate web pages and embed a lot of Cumulus web tags. I have had to edit these templates as in some cases the web tags have date/time output format parameters.
       I mentioned some of my web pages are generated from my own Cumulus templates. Despite now having Cumulus 1 and Cumulus MX templates with different names, these still both generate remote web pages given the same names, so I don't need to alter any navigation between pages on my web server.
       In the "[NOAA]" section I had MonthFileFormat="NOAAMO"MMMyyyy".txt" on a line, I checked at Webtags#The_format_used_for_naming that 'MMM' and 'yyyy' were valid in Cumulus MX as well as in Cumulus 1, they were, so no edit needed either to that line or to "YearFileFormat="NOAAYR"yyyy".txt" line.
   In the download, there is a file "\CumulusMX\Reports\.gitignore" provided. In this same MX folder, I copied in the 200 or so report files I currently have in "\Cumulus\Reports". Remember you don't need to rename these files, it is only how that naming is specified in the settings that might need changing.
   Next I see there is a "MXdiags" folder, so I don't copy across the Cumulus 1 "diags" folder. Equally I don't change anything in the new MX "webfiles" folder, but I do copy across to my web server the "cumuluscharts.js" and "logoSmall.png" files I see there that I have not seen before.
   So left to last is the "data" folder:
       I have copied all files from "\Cumulus\data" to "\CumulusMX\data" (except log.xml as MX uses a different file and MX does not provide a way to read the XML file into its diary.db).
           Some of my .ini contain date-time entries like "12/03/2019 14:50:45". I'm assured MX can read these, although it changes a date time like that to "2019-03-12T14:50:45", but I am waiting to see if that is true. I use the full stop character for all my decimal points, but if you use decimal commas, you might find you need to do some editing of your .ini files in this data folder before MX can read them.

Aside on .ini files in data folder

   I find after running the CumulusMX engine, that it edits those .ini files it needs to, and the new versions contain date-time entries in the "2019-03-16T12:45:00" style, but despite what I worried about from reading on the forum, I found you don't need to edit beforehand these entries in log files like month.ini as Cumulus MX can read the old formats like "12/03/2019 14:50:45" as well as the new year first formats.

Creating the shortcut to run MX

   I right clicked on the "CumulusMX.exe" entry in the top level folder and selected Send to ... desktop (create shortcut) .
   Ensure that your shortcut has a Start in: defined as the folder where your executable is stored. This is the most vital setting.
   You can optionally change other shortcut properties. I selected to use different settings in the colours tab, so when I have that terminal/console window open I don't confuse it with any other command window I might open. I selected to change options in the layout tab to position the terminal/console window on my second (smaller) screen. I also selected the Start minimised option, as I don't need the window for this MX engine taking up screen space all the time.
   I have renamed my shortcut to "MX_run" so I can recognise it as different to the folder name, as I have also created a shortcut for the folder.
   On the same right click menu I also selected Pin to taskbar.
   When I am happy with MX, I will copy the shortcut into C:\Users\Personal\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\ and then MX will start automatically after I (or Windows Update) reboot my PC.

One-off extra installation steps

   I clicked on one of my new shortcuts. Windows Defender popped up, so I told it allow all for Cumulus MX.
   I opened my Internet Security premium package, in the "unrecognised files" section I selected CumulusMX.exe then right clicked and selected Change File Rating to Trusted
   In the "Firewall" section of the Internet Security package I added port 8998 as one that was permitted.
   I viewed my hub (router) to see the IPv4 address allocated to my Cumulus MX computer (192.168.1.64), that told me that I would find the user interface by typing "http://192.168.1.64:8998/" into my browser while the MX engine command window remains open (so MX is actually running), so I typed that and I saw the user interface and navigated around it.
   I right clicked on my desktop (you may need to right click on the windows symbol at bottom left), selected command prompt as administrator that opened a command window where I typed

netsh http add urlacl url=http://http://192.168.1.64:8998/ user=\users

and I got the response "URL reservation successfully added", so I know it worked. This command is apparently to allow all users to bind to port 8998 (i.e. that used for the Cumulus interface). This also means you don't have to run the engine (CumulusMX.exe) in an administrator user, nor select "Run as administrator" from right click menu on the shortcut, nor set the properties for any shortcut to run in any special way.

IMPORTANT NOTE: I don't use "localhost:8998" for two reasons, first I already have a web server on my PC at IPv4 "127.0.0.1" using "localhost" as an alternative name (and port 81, selected because something called 'Skype' that I don't use had reserved port 80 that I had expected to use), and second using the IPv4 exact "http://192.168.1.64:8998/" address as a bookmark, I can view the Cumulus Admin Interface on my mobile phone which shares bookmarks with my PC and connects to my LAN via wifi.

End of my test of MX actions

When I am happy to stop using MX, I type Control + C into that MX command window on my PC and MX closes.

Testing MX while still using Cumulus 1

Obviously, you cannot have Cumulus 1 and Cumulus MX running at same time, accessing same weather station.

If you (like I was to begin with) are just experimenting with MX you may sometimes run one flavour (say MX) and sometimes the other (Cumulus 1).

Each time you swap, you must copy all the updated log files from the just used data folder to the data folder you are about to use when you close one favour before you start the other flavour, or you must have both executables in same top level folder to force them to share the data folder.

Please note, today.ini is a special case, the time-stamp line has a different format in Cumulus 1 (C1) and MX; while MX can read the format that C1 uses, C1 cannot understand the format that MX uses. Remember today.ini determines which stored entries in the weather station console need to be read to "catch-up" and it also holds the various figures that will inform what gets stored in dayfile.txt at the end of the day. So to have this file being read and updated correctly is vital.

If you don't do ensure both flavours use the same log files (.txt and .ini), various derivatives (e.g. Chill Hours) will become wrong and you may have conflicting rows in dayfile.txt (because its content is generated from what that flavour saw when it was running the previous day) and generally this will be particularly evident in any weather parameter that varies a lot like the wind vector (speed and direction). It also affects what is stored for any derivatives that rely on averaging (temperature, wind run, rain rate) as these are calculated biased towards the actual times when that flavour of Cumulus was actually running, so you can have issues if you run the 2 flavours in different folders/devices as if the other does not exist.

I have some batch scripts that Cumulus initiates, and a number of Cumulus templates, and in my case I had to be happy these were working before I stopped using Cumulus 1, and got MX as the flavour that auto-started on switching on my PC.

I also use output modifiers on a lot of the web tags I use in my custom web pages and it took me a long time to work out the necessary changes to get these templates correctly edited so that MX could process them and produce the web page content I wanted. I am not going to explain all the problems nor give the solutions, because you probably don't have web pages as complex as mine. 2.4.8 My migration from Cumulus 1 to MX

My installation of Cumulus MX has succeeded, and as the experiment did help me find a mistake in one of my Cumulus templates where I had not defined input parameters for "Recent History", it has been useful. Initially, I am continuing to use Cumulus 1 for the moment, until I am absolutely sure MX can do everything I want.

I believe MX will do some tasks better, but there is a lot more to learn about how to use MX. MX does lack some features that I used in Cumulus 1. I found the View Period screen in Cumulus 1, where you could look at any day, week, month, season, or year in the past extremely useful. MX does not have such functionality yet.

While I was using Cumulus 1, I found with my Chas Olsen Fine Offset I had to define EWpressureoffset=x.y otherwise Cumulus 1 frequently failed to read the correct pressure. In implementing MX, I decided to try without that line in the new Cumulus.ini file; I have not had any problems either when I first ran MX or indeed 2 months later when it is more than a month since I last ran Cumulus 1. Consequently, if you used to have pressure reading problems, you might find you don't with MX.

Some days after I first started trying MX, I have tried out more MX features, been happy with those, and as MX is now doing all I want I have stopped using Cumulus 1.

I still use my own PHP script to update my database tables, I tried the custom SQL and it does not do all I want.

I have done a little editing of the user interface, partly to discover how easy it is to edit, partly to understand better how it works. For each file I have edited (HTML or CSS) I have kept copy of original and made a second copy of my edited version, so I cam easily go back to original and if I download a new release I won't lose the copies of my edited versions of files.

You can judge my progress with trying features, because elsewhere in this Wiki I have expanded the text for those features I have tried and therefore understand.

Retrieved from "https://www.cumuluswiki.org/index.php?title=Migrating_from_Cumulus_1_to_MX&oldid=6334"