Thrifty - Cutils Command Qualifier

Introduction

While CumulusUtils was growing, it became clear that economising computer resources both for CPU as for the Internet usage could be economised. Especially for users on a paid line with low debit that would be useful. In that context it was decided to introduce the Thrifty qualifier to the All and Website commands. Later the qualifier was made available to all commands (single module runs).

Thrifty tries to effectively minimise what needs to be done to create the output i.e. it will not generate YADR output for years past as, once generated, the information will never change again. Every module has different methods and needs. This article describes how to invoke thrifty and what the user can expect.

Note that CumulusMX itself uploads the data for the charts to the website. It is not advised to switch that off completely but to verify that you only send the data for the charts you use require. The data will be the bulk of your transfers so it will count.

Thrifty was primarily developed with internet transfers in mind, in a second stage CPU usage and generation were added. This is a work in progress as user requests, new efficiency discoveries and efficiency code changes will change over time.

NOTE: With the coming of the ChartsCompiler the Thrifty algorithm for the Website command for the charts and for cumulsutuils.txt (containing the Home page charts) will change. Until then it is like below and if you use the ChartsCompiler you should run a non Thrifty Website command if you make changes.

Operation

Thrifty is activated when placed before the command. When using it for multiple modules it can be placed anywhere on the commandline. See the following examples:

 utils/bin/cumulusutils.exe Thrifty Website       => This activates the Thrifty command qualifier on the Website generation
 utils/bin/cumulusutils.exe Website Thrifty       => This only runs Website, Thrifty is ignored 
 utils/bin/cumulusutils.exe YADR Records Thrifty  => This activates the Thrifty command qualifier on the named modules
 utils/bin/cumulusutils.exe Thrifty YADR Records  => This activates the Thrifty command qualifier on the named modules
 utils/bin/cumulusutils.exe YADR Thrifty Records  => This activates the Thrifty command qualifier on the named modules

Output

The Thrifty qualifier itself has no output, the output of the individual modules is limited as far as possible and as described in the Inner Working.

Inifile parameters

Thrifty has the inifile parameters as listed below. These parameters give the cycle in days of when a module is to be run. So if the user runs CumulusUtils on a daily basis and the WindGraphPeriod is three - as shown in the listing - then the WindGraphs will only be generated once every three days (this calculation is based on the daynumber in the year for that date).

 [Thrifty]
 Top10RecordsPeriod=1
 RainGraphsPeriod=1
 TempGraphsPeriod=1
 WindGraphsPeriod=3
 MiscGraphsPeriod=1
 SolarGraphsPeriod=3

For specified parts or/of modules, periodicity can be specified for the Thrifty system. See also the table below. Numbers are in days. Default is 1 day for each parameter. So if you give a periodicity of 2 for e.g. the temperature graphs (not individual graphs), then every other day the temperature graphs will be refreshed.

If the following is true then the module is calculated:

 ${current\ day\ of\ year}\,{\bmod {\,}}{periodicity}\equiv 0$

So, for that temperature graph, on a 123th day of the year, with a periodicity of 3 those graphs will be created. With a periodicity of 4 it will not.

Advise: Start with the defaults and if it takes too long, start augmenting the periodicity, balancing it to your need for up to date graphs. First graphs to look at would be the Wind graphs (which includes the WindRose).

Inner working

In this section for every module the effect of Thrifty is described and a summary is given. The explanation of thrifty starts by what you can do manually.

NOTE: When updating always run without Thrifty first time after the update!

Uploading manually

When you upload manually you just need to know, it is not required to upload everything. Just upload the files which have changed or which have changed so much that you think it useful to upload them.

Always upload index.html after generating the website.

So here is the list from the distribution which you can safely NOT upload if there are no changes to the system (e.g. an Update), if they have been uploaded once:

1. HighchartsDefaults.js
2. gauges.js
3. suncalc.js
4. tween.min.js
5. language.js
6. RGraph.common.core.js
7. RGraph.rose.js
8. steelseries.min.js

These files will be generated by the Website command. If you did not change anything, they do NOT need to be uploaded:

1. cumulusutils.js
2. cumuluscharts.txt
3. HighchartsLanguage.js

For each of the modules you can safely rarely upload the following because the changes are none or minimal. The files are listed with a minimal advised upload frequency:

1. forecast.txt [advise: only initial (if you are using the Norwegian system or SpotWX) or daily (if you are using yourweather). No advise for WXSIM]
2. graphsmisc.txt, graphsrain.txt, graphstemp.txt, graphswind.txt, graphssolar.txt [advise: every other day]
3. noaa.txt [advise: every 2d day of the month]
4. Yadr.txt [advise: Every year, 2 January]
5. Yadr[func][year].txt [advise: Every day] where func is : Rain, Hum, Temp, Press and Wind and where year is THE CURRENT YEAR. All other data files from previous years are already uploaded and won't change and therefore don't need to be uploaded nor generated]
6. The records files (top10, records and dayrecords) need only upload if a new records is set.
7. Stationmap.txt [advise: once per year]

The above is the advise if you are uploading manually and make the decisions on your own. It is also important for understanding what is happening in the automatic Thrifty system. If you upload automatically everything this is being taken care of : a module is generated and uploaded only according to the above specification.

Uploading Automatically

Automatic Thrifty handling assumes a daily run of CumulusUtils.

For proper understanding we split the usage in modules and website.

Thrifty Website

Under Thrifty, the following holds for the website files and the website infrastructure:

1. Always upload index.html after generating the website.
2. The runtime javascript environment files will NOT be generated and NOT be uploaded - to prevent old existing files to upload - under the Thrifty Website command.
3. The library files from the distribution will NOT upload under the Thrifty Website command:

Runtime javascript environment files:

cumulusutils.js
cumuluscharts.txt
HighchartsLanguage.js

Library files from the distribution:

HighchartsDefaults.js
gauges.js
suncalc.js
tween.min.js
language.js
RGraph.common.core.js
RGraph.rose.js
steelseries.min.js

NOTE: When updating, copy those files from the distribution and DO NOT USE THRIFTY

Thrifty Modules

The decision to automatically upload a file or not - assuming DoUploadFTP=true = is done on the basis of two parameters:

1. Have the cycle conditions become true
2. Has a file been modified

Some modules have a period in days associated with them which determines if the module is run and uploaded at all. An example is e.g. every third day for the graphs because either the user does not look at this so often of the chart does not change that much on the basis of one or two days. Default the cycles are set to one day which means the cycle condition will always be true.

The modification of the file is being done either during the data selection (in which case the generation of the module output will be finished) or because of the algorithm in which case some of the output will not be generated. This is done by setting the Dirty bit. Which is simply a flag that - if true - says : yes the output has changed since yesterday so it needs to be generated and uploaded.

Examples:

1. When the records output needs to be generated, during that generation it is determined whether any records have been set at all. If no record has been set then the Dirty bit remains false. If possible the generation of the output is cancelled and the upload will not be done.
2. For the Yadr module under the Thrifty condition we definitely know the past years will not have been modified so those years need not to be generated nor uploaded again. So only the current year is generated and uploaded. The menu actually only changes when a new year is added which is done on the 2^nd of January so that is the only day Yadr.txt will be uploaded under Thrifty.

Then at the end during uploading for each module under Thrifty condition it is checked if any of the Thrifty conditions has become true (periodic or Dirty bit) and if so, the module output is uploaded.

Thrifty Summary Table

Module	Files	Thrifty Generation	Thrifty Upload
Website	index.html cumulusutils.js cumuluscharts.txt highchartsLanguage.js	Always Never Never Never	Yes Never Never Never
Charts	graphsmisc.txt graphstemp.txt graphsrain.txt graphswind.txt graphssolar.txt	periodic periodic yesterday rain OR periodic periodic periodic	If generated if generated if generated if generated if generated
Top10	Top10Table.txt	when dirty / periodic	when dirty / periodic1
Records	records.txt	when dirty	when dirty
Day Records	dayrecords.txt	when dirty	when dirty
NOAA	noaa.txt	2d day of month	2d day of month
Forecast	forecast.txt	always	always
Yadr	yadr.txt yadrtemp{year}.txt yadrrain{year}.txt yadrwind{year}.txt yadrwindrun{year}.txt yadrpress{year}.txt yadrhum{year}	2d day of year current year current year current year current year current year current year	2d day of year current year current year current year current year current year current year
AirLink	airlink.txt	Never	Never
StationMap	Stationmap.txt	Never	Never