Updating MX to new version

Updating to a new MX release

The whole of this article is based on an assumption, that you already use MX (if you want to move from Cumulus 1 to MX, then read Moving_from_Cumulus_1_to_MX article).

The simplest update is from the immediate preceding version. Generally, skipping minor releases in an update (from 3.x.y to 3.x.z, i.e. only final digit of version number changing is a minor release) are still simple even if you miss out a few in-between minor releases, but you might need to do some extra action.

The ease of doing a major update (i.e. where middle part of version number changes) is more variable in difficulty depending on which features in MX you actually use. You can even update skipping some major updates, but as that can be more complex, there is a separate section later with advice on how to update in stages from an early build to a recent build.

CREDIT

Thanks to Billy on support forum for suggesting text for this article.

Knowing when a new release is available

Using forum notifications

As new releases are announced in Cumulus MX Announcements and Download - PLEASE READ FIRST topic, you can use the spanner tool to subscribe to this topic to receive notifications of a new post announcing a new release (or any other release-related announcement).

MX terminal message

If ...

1. ...you have a monitor to see the terminal output from the Cumulus MX engine (Windows calls this either a command window, powershell window, or a terminal window depending on how you invoke it, for Unix-based implementations this is the output window when using the terminal functionality), AND
2. ...your device running MX is connected to internet, AND
3. ...your MONO (if not Windows) is not obsolete (SSL certificate out of date), AND
4. ...you restart MX

... then you will see a prompt when a new version of MX is available.

It is worth stressing that if you leave MX running, then this feature will leave you blissfully unaware that an update is available; it only checks when MX is restarted.

MXDiags

In addition ...

1. ...if you can view the MXdiags file for the current session of MX, AND
2. ... MX is running with connection to the internet, AND
3. ...you restart MX

... if a new version of MX is available, the MXDiags file will say so (the message is not easy to spot as there is a lot of output before it, and variation in what output appears before it). Anyway, here is one example, just one example as in my experience the message has appeared at different places for each of the recent updates):

2020-05-27 04:18:48.326 Calculating sunrise and sunset times
2020-05-27 04:18:48.326 Sunrise: 04:58:11
2020-05-27 04:18:48.326 Sunset : 21:19:54
2020-05-27 04:18:48.326 Tomorrow sunrise: 04:57:08
2020-05-27 04:18:48.326 Tomorrow sunset : 21:21:11
2020-05-27 04:18:48.388 You are not running the latest version of CumulusMX, build 3080 is available.
2020-05-27 04:18:48.763 Station type:

There is no message in whatever place it can appear ...

• ... if you are using latest version, OR
• ... if you are not connected to the internet, OR
• ... if you keep MX running all the time!

Start/Stop script

When the Status option of this script is used, whenever a new release of MX is available, it will output a message.

What to read (and when) before updating

Updating from immediately preceding build

The release announcement

The release announcement is found in Cumulus MX Announcements and Download - PLEASE READ FIRST topic of the support forum.

• Generally, a release announcement WILL contain
  1. An overall purpose for that particular release (e.g. fixing bugs or adding functionality)
  2. Details of what functionality has been added
  3. Details of what bugs have been fixed, or where the processing has been improved
  4. A list of the main files that have been added or amended in the release (excluding "updates.txt" which is amended in each release)
• Sometimes, a release announcement MAY contain
  • one-off actions (like changing the schema if you use a database, or editing your web pages to take advantage of new web tags).
  • scripts to download and run to perform one-off actions.
• Be aware that it is worth while checking back on the release announcement for a few days after a release.
  • It may have been edited because the original announcement forgot to mention something.
  • It may have been edited to mention that some bugs have now been found
    • That may mean you are advised to regress to an earlier version and use that
    • It might mean that some supporting files in current version are wrong, and you only need to regress those named files
    • There might be an emergency release to fix the bugs, and you need to update to that emergency release
    • Finally you might be given advice to avoid using certain parts of the functionality or take some other action until the next release is available.

Other places where you can find information about release content

Although it is not always kept in step, a concise summary of all formal MX releases is available at Cumulus_MX_formal_release_versions.

You can also view the latest Updates.txt.

Deciding whether to update to new release

• Any new development or change in a new version of MX might cause problems for some users. You might want to stick with the version you are already using unless you really need any new functionality or the fixes gained by upgrading.
• Also remember that there are bugs in all versions of MX, this is a large and complicated package, and the current developer has not been able to test all the code with all possible settings and all possible weather stations.

Updating to a new minor build, skipping in-between minor builds

For a minor version build either the associated version number does not change or only the final section changes (3.x.y to 3.x.z).

Reading multiple release announcements

If you are skipping some intermediate builds, then you will need to read each of the formal release announcements for builds after the build you currently use. You need to apply the cumulative actions recommended. Otherwise, most of the advice above for updating from immediately preceding build still applies. The other sources of release information mentioned above may be consulted as an alternative.

Updating to a new major version

Generally, if the developer decides a new build warrants classification as a major version (i.e. 3.w.0) then the change being implemented is significant enough that updating might be more complex.

Examples of what might be classified as a major change

• Additions to fields in log files
• Additions or removals in configuration file
• New pages in Administrative Interface
• Catering for new weather station sensors, or new ways of communicating
• Any interface functionality changes
• Additions to files being generated for web server
• Schema changes for database tables

What you need to read

• Basically, check the corresponding release announcements for every version since the one you have been using before planning your upgrade.
  • Make a note of any one-off actions required at particular in-between versions, remember these actions are only in forum release announcements.
  • Although one-off actions will not be described in the Wiki (whether on the Software page or the Cumulus_MX_formal_release_versions page), the Wiki can give you an idea of what functionality has been improved to help you decide whether to update.
• It is still worth reading all the points made above for updating from immediately preceding build.

Updating from a very old version

Many people believe if it works, it don't need fixing. So they install whatever MX version is available when they start using MX and ignore all the bug fixing and new functionality being added, and stick with their existing installation.

Suddenly they realise that perhaps their version does lack functionality that would be useful to them. Maybe their version does not have an editor for correcting rogue extreme records and they realise correcting log files is something that is hard for them to do.

This section is therefore included to give advice on why you should not jump from an old version to the latest directly, but how it can be safely done in stages.

Recommendations for staged updating

If you use 3.0.0 (the MX original beta) now

Update to 3.5.1 by downloading it at https://github.com/cumulusmx/CumulusMX/releases/tag/b3072.

This gives you essential new functionality in the admin interface like editors for the log files and extreme records. But it also fixes multiple bugs in the beta you were using and adds some useful validation missing in the beta.

It does not involve any updates to the fields in the log files nor to the columns in any database tables you use.

It skips you past the problems in 3.5.0. It gives you benefits introduced in 3.1.x, 3.2.y, 3.3.z, and 3.4.w releases.

If you use 3.1.x, 3.2.y, 3.3.z, and 3.4.w releases

As above, update to 3.5.1 by downloading it at https://github.com/cumulusmx/CumulusMX/releases/tag/b3072.

If you use a 3.5.x release

My advice is to update directly to 3.7.0 or any subsequent release available at Software. This is because several 3.6.y releases have problems.

But there are actions you need to do depending on what features you use in MX:

• If you use database tables, be aware that the schema (Column names that must be in a table) varies between different versions in this range.
  • The updating feature in MX will only work if all the columns named in that update already exist in the database table it is trying to update.
  • Look at individual release announcements for both 3.6.0 and 3.7.0, to see the SQL provided for adding the columns added at particular versions.
  • At 3.6.0 the SQL was provided as a separate attachment to the release announcement (UpdateMYSQL-b3076.zip)
  • At 3.7.0 the SQL was provided as part of the main zip (AlterSQLTables3098.sql)
  • In both cases, the SQL provided assumes you are using the default names for database tables, it does not read Cumulus.ini to see whether you have selected different names for the tables (unlike ExportMySQL.exe and standard CumulusMX.exe uploads which check what tables names you have selected for updates).
• If you have your own customised web pages, then there are changes to web tags that might lead to you needing to edit your web pages.
• If you use commas to separate integer and decimal parts of real numbers, then various releases from 3.6.0 to 3.7.0 add "rc=y" to various web tags, that option will replace the decimal commas you use by decimal points that are required for some script languages (like the JavaScript used by HighCharts), and that makes it easier if you want to customise your web site.

Recommended Actions when Updating

Back-ups

It is always best to take a backup of your existing MX installation before you do an update, this allows you to regress back to the earlier version if either you mess up installing the new version, or the new version has a issue that prevents it working with the versions of other software (like MONO) that your installation uses.

The two approaches

Contributors to this section are named in text.

Some people upgrade by just copying in the files that the release announcement says have changed, others copy in all files from the downloaded zip. The first should only be used with caution, files like CumulusMX.exe.config can change between versions, but not be mentioned in a release announcement, and the developer will have been making edits to files since the previous release, and might forget exactly which files have been edited between releases. Also you may be upgrading from an earlier version and therefore be skipping several intermediate releases. You may be able to see the dates when files were changed within the zip and therefore be able to decide for yourself if you compare those dates with the previous release you were using if you have kept the download for the version you were using.

The rename old approach

1. The popular approach recommended by many forum contributors in many different posts (including at this post by Mark Crossley for example is to rename your current install directory, then unzip the new release, letting it create a new CumulusMX folder (or whatever name you prefer and specify in unzip options).
2. Copy across Cumulus.ini and string.ini into that new directory, and then copy the contents of the data and Reports directories from your current install to the new install.
3. Don't forget to copy any other set-up files across too.
   • The advantage (as Mark says) is that you ensure you do use all the files in the new release, and don't miss out any he may have forgotten to mention in his release announcement.
   • A key advantage is that you don't retain files that an old release migt have needed but the new release does not use.
   • Another advantage (as PaulMy says here for example) is that you retain your old set-up intact and can easily restore it should you have a problem with new release.
4. Various people advise that you don't delete your old installation for a week or so; as you may notice something from the older version that you haven't copied across!
5. Check again that you copied across strings.ini, twitter.txt, Cumulus.ini, and similar files in the same folder level as CumulusMX.exe, as well as all the files in the data and Reports sub-folders. Then, provided you are happy, you can delete the old release when you want to reduce clutter on your storage discs.

The overwrite approach

1. • if you have a lot of set-up files, or other custom files, (i.e. files not part of release), or
   • if you are downloading on a different device, or on a different disc to where you are running MX,
2. then David (see this post) recommends this different approach. After downloading a new release unzip it on the device/disc where you down load it. Next simply copy the files (optionally only those that have newer dates because they have changed) into the existing MX directory on the device where you run MX. Then you know all your existing files are there, and you can choose to only copy in files that have been updated.
3. The problem with this approach is that in some releases files are removed from the distribution, yet unless you check carefully, this approach may leave files that are no longer required in your installation, which you can continue to use instead of the intended alternative. This can cause you problems if those files are then still used by you, giving you a non-standard installation that it is difficult for others to support. So do check through file by file to ensure you only have files in the distribution and files that either configure your system or are log files.
   • The updated files can be tracked by their modification date, or by seeing which are specified in the release notes (find them on this forum or in Wiki here).
   • In general, especially if updating a major release, it is best to overwrite with all files in zip, whether they are listed as changed or not. This ensures you have a consistent set of files.

Example downloading on PC and installing on pi

1. On your pc: Download zip either from Software for latest version or from https://github.com/cumulusmx/CumulusMX/releases for earlier versions.
2. On your pc: Use Filezilla (or similar file transfer program, or it might even be a copy over network depending on way set up) to transfer zip to pi e.g. FileZilla CumulusMXDist3089.zip to /home/pi
3. Control C on pi to stop MX
4. On pi go to directory where zip now is cd /home/pi
5. Unzip the installation over your existing installation unzip -o CumulusMXDist3089.zip
6. Restart MX
7. Remove any files that were needed by earlier releases but release announcement says are not needed by the release you are installing

After an update if you use the standard web template files

The new web template files will be in your web folder and MX will process the new files and upload the web pages produced as before with no further action by you.

Files that are unzipped into the webfiles folder need to be uploaded to your website if you use the standard web templates.

The file \CumulusMX\webfiles\js\cumuluscharts.js has changed in a lot of releases. The standard web page trends.htm will only work if the correct version of that file has been uploaded to your web site and placed in a sub-folder "js" of where "trends.htm" is installed.

The other files in the webfiles folder have either never been updated or only rarely updated, but be prepared to upload them as well.

Updating when files within release might overwrite your own edits

web site files

If you have edited any files that sit on your web site, to give the look you desire, or the content you want (e.g. adding rain this month to this month page, or combining this month and this year page), see if the release notice says that file has been revised, if it has not then it is easy to keep your edited file by not copying in the replacement file from within the zip. If the release revises any file you previously edited, take a backup of your edited file, before you copy the new file into your folder. You can then use a file comparing tool to see what has changed in the release and what you changed and hopefully manage to merge to a new file that keeps any functionality change in a new release and keeps your customisation.

If you have done major customisation to the standard website then you probably have followed the guidance and stored your new web page templates in a different directory and you use Extra Files to specify where they are, so they cannot be overwritten and the new MX will still find them.

admin interface files

If you have done any customisation to the interface (perhaps you don't like seeing solar in the tables when you don't measure that) then again you can skip over these files when copying. Hopefully you will have copies of the files that you have customised of the interface folder so you have ability to copy them back into installation if you do overwrite with the release version.

You do have to be careful, as many releases change the interface in some way and all the various components of the interface have to work together as a coherent unit. For instance when feels like was added to MX, the api used for sending data from the MX engine to the admin interface was updated. In another release, adding a new web page, all the navigation menus were adjusted in all web pages. A HTML page may be edited to refer to a different styling page (so it and the .css file must be updated together) or to a different script (so the .HTML and the .js file must be updated together. Many web pages are dependent on the correct pair of json files to define the options or values that appear on that web page. So files are interdependent, and you cannot always update some but not all of the admin interface pages.

Be prepared to go back to the standard file for whatever you customised if something it depends upon has changed, after all you must not lose any vital functionality.

On my site, my own versions of interface files have a "_" (underline character) added to the standard MX file name (before the "." and the relevant extension). This applies to both HTML pages, and JavaScript files that I have edited. I edit the menu items within my edited pages so those all go to my versions where I have a HTML customised page, leaving unchanged the menu items that can still go to a standard MX web page where I don't have my own version of the .HTML page. This makes it easy for me to navigate between my pages, as all of them link to my other pages. If I am on a standard MX page and want to go to one of my customised pages, I select the equivalent standard page, then edit the URL to add the underline and get easily to my page. This naming means I can always use a standard page instead of my customised page when I need to and I never miss out on any new features.

After update - checking for bugs in MX or mistakes in your installation

Start the new installation of MX and watch out for any errors - If the device you run MX on has a monitor, then look in the terminal/command window. In all cases look at the latest file in the MXdiags folder to see if any errors are reported.

Also keep an eye on the support forum for first few days, some releases do have bugs, as developer cannot test all possible configurations.

Updating if you use a virus Checker

You may find that virus checkers such as Windows Defender reject your new version of MX. They need to be told it is safe.

Updating if you use the start/stop management script

This section contributed by Jank on support forum. Note the version on the forum might have been updated from the original included here.

1. look on Software download page, find the link to latest version, and fill out the '...' below appropriately as you run these 2 commands on your device where you do downloads:

cd /tmp
wget https://github.com/cumulusmx/CumulusMX/ ... .zip

2. Once that download is complete, start cumulusmx.sh with option -u

/home/pi/CumulusMX/cumulusmx.sh -u

3.When asked for the zip file, enter

/tmp/CumulusMXDist

and hit the TAB Button

4.Choose the zip file with the CumulusMX update and hit return.

5. Follow the on screen instructions

6. With each update component .....you can choose: [y]es, [n]o, [A]ll, [N]one, [r]ename

I would recommend select A as that will simply replace all files without further action.

CumulusMX will be restarted after update completes.

You can check if the update was successful by using option -s:

 /home/pi/CumulusMX/cumulusmx.sh -s