Thrifty - Cutils Command Qualifier

From Cumulus Wiki

Introduction

While CumulusUtils was growing, it became clear that economising computer resources both for CPU as for the Internet usage could be economised. Especially for users on a paid line with low debit that would be useful. In that context it was decided to introduce the Thrifty qualifier to the All and Website commands. Later the qualifier was made available to all commands (single module runs).

Thrifty tries to effectively minimise what needs to be done to create the output i.e. it will not generate YADR output for years past as, once generated, the information will never change again. Every module has different methods and needs. This article describes how to invoke thrifty and what the user can expect.

Note that CumulusMX itself uploads the data for the charts to the website. It is not advised to switch that off completely but to verify that you only send the data for the charts you use require. The data will be the bulk of your transfers so it will count.

Thrifty was primarily developed with internet transfers in mind, in a second stage CPU usage and generation were added. This is a work in progress as user requests, new efficiency discoveries and efficiency code changes will change over time.

NOTE: With the coming of the ChartsCompiler the Thrifty algorithm for the Website command for the charts and for cumulsutuils.txt (containing the Home page charts) will change. Until then it is like below and if you use the ChartsCompiler you should run a non Thrifty Website command if you make changes.

Operation

Thrifty is activated when placed before the command. When using it for multiple modules it can be placed anywhere on the commandline. See the following examples:

 utils/bin/cumulusutils.exe Thrifty Website       => This activates the Thrifty command qualifier on the Website generation
 utils/bin/cumulusutils.exe Thrifty All           => This activates the Thrifty command qualifier on the All modules generation
 utils/bin/cumulusutils.exe Website Thrifty       => This only runs Website, Thrifty is ignored 
 utils/bin/cumulusutils.exe YADR Records Thrifty  => This activates the Thrifty command qualifier on the named modules
 utils/bin/cumulusutils.exe Thrifty YADR Records  => This activates the Thrifty command qualifier on the named modules
 utils/bin/cumulusutils.exe YADR Thrifty Records  => This activates the Thrifty command qualifier on the named modules

Output

The Thrifty qualifier itself has no output, the output of the individual modules is limited as far as possible and as described in the Inner Working.

Inifile parameters

Thrifty has the inifile parameters as listed below. These parameters give the cycle in days of when a module is to be run. So if the user runs CumulusUtils on a daily basis and the WindGraphPeriod is three - as shown in the listing - then the WindGraphs will only be generated once every three days (this calculation is based on the daynumber in the year for that date).

 [Thrifty]
 Top10RecordsPeriod=1
 RainGraphsPeriod=1
 TempGraphsPeriod=1
 WindGraphsPeriod=3
 MiscGraphsPeriod=1
 SolarGraphsPeriod=3

For specified parts or/of modules, periodicity can be specified for the Thrifty system. See also the table below. Numbers are in days. Default is 1 day for each parameter. So if you give a periodicity of 2 for e.g. the temperature graphs (not individual graphs), then every other day the temperature graphs will be refreshed.

If the following is true then the module is calculated:

 

So, for that temperature graph, on a 123th day of the year, with a periodicity of 3 those graphs will be created. With a periodicity of 4 it will not.

Advise: Start with the defaults and if it takes too long, start augmenting the periodicity, balancing it to your need for up to date graphs. First graphs to look at would be the Wind graphs (which includes the WindRose).

Inner working

In this section for every module the effect of Thrifty is described and a summary is given. The explanation of thrifty starts by what you can do manually.

NOTE: When updating always run without Thrifty first time after the update!

Uploading manually

When you upload manually you just need to know, it is not required to upload everything. Just upload the files which have changed or which have changed so much that you think it useful to upload them.

Always upload index.html after generating the website.

So here is the list from the distribution which you can safely NOT upload if there are no changes to the system (e.g. an Update), if they have been uploaded once:

  1. HighchartsDefaults.js
  2. gauges.js
  3. suncalc.js
  4. tween.min.js
  5. language.js
  6. RGraph.common.core.js
  7. RGraph.rose.js
  8. steelseries.min.js

These files will be generated by the Website command. If you did not change anything, they do NOT need to be uploaded:

  1. cumulusutils.js
  2. cumuluscharts.txt
  3. HighchartsLanguage.js

For each of the modules you can safely rarely upload the following because the changes are none or minimal. The files are listed with a minimal advised upload frequency:

  1. forecast.txt [advise: only initial (if you are using the Norwegian system or SpotWX) or daily (if you are using yourweather). No advise for WXSIM]
  2. graphsmisc.txt, graphsrain.txt, graphstemp.txt, graphswind.txt, graphssolar.txt [advise: every other day]
  3. noaa.txt [advise: every 2d day of the month]
  4. Yadr.txt [advise: Every year, 2 January]
  5. Yadr[func][year].txt [advise: Every day] where func is : Rain, Hum, Temp, Press and Wind and where year is THE CURRENT YEAR. All other data files from previous years are already uploaded and won't change and therefore don't need to be uploaded nor generated]
  6. The records files (top10, records and dayrecords) need only upload if a new records is set.
  7. Stationmap.txt [advise: once per year]

The above is the advise if you are uploading manually and make the decisions on your own. It is also important for understanding what is happening in the automatic Thrifty system. If you upload automatically everything this is being taken care of : a module is generated and uploaded only according to the above specification.

Uploading Automatically

Automatic Thrifty handling assumes a daily run of CumulusUtils.

For proper understanding we split the usage in modules and website.

Thrifty Website

Under Thrifty, the following holds for the website files and the website infrastructure:

  1. Always upload index.html after generating the website.
  2. The runtime javascript environment files will NOT be generated and NOT be uploaded - to prevent old existing files to upload - under the Thrifty Website command.
  3. The library files from the distribution will NOT upload under the Thrifty Website command:

Runtime javascript environment files:

cumulusutils.js
cumuluscharts.txt
HighchartsLanguage.js

Library files from the distribution:

HighchartsDefaults.js
gauges.js
suncalc.js
tween.min.js
language.js
RGraph.common.core.js
RGraph.rose.js
steelseries.min.js

NOTE: When updating, copy those files from the distribution and DO NOT USE THRIFTY

Thrifty Modules

The decision to automatically upload a file or not - assuming DoUploadFTP=true = is done on the basis of two parameters:

  1. Have the cycle conditions become true
  2. Has a file been modified

Some modules have a period in days associated with them which determines if the module is run and uploaded at all. An example is e.g. every third day for the graphs because either the user does not look at this so often of the chart does not change that much on the basis of one or two days. Default the cycles are set to one day which means the cycle condition will always be true.

The modification of the file is being done either during the data selection (in which case the generation of the module output will be finished) or because of the algorithm in which case some of the output will not be generated. This is done by setting the Dirty bit. Which is simply a flag that - if true - says : yes the output has changed since yesterday so it needs to be generated and uploaded.

Examples:

  1. When the records output needs to be generated, during that generation it is determined whether any records have been set at all. If no record has been set then the Dirty bit remains false. If possible the generation of the output is cancelled and the upload will not be done.
  2. For the Yadr module under the Thrifty condition we definitely know the past years will not have been modified so those years need not to be generated nor uploaded again. So only the current year is generated and uploaded. The menu actually only changes when a new year is added which is done on the 2nd of January so that is the only day Yadr.txt will be uploaded under Thrifty.

Then at the end during uploading for each module under Thrifty condition it is checked if any of the Thrifty conditions has become true (periodic or Dirty bit) and if so, the module output is uploaded.

Thrifty Summary Table

Module Files Thrifty Generation Thrifty Upload[1]
Website



index.html
cumulusutils.js
cumuluscharts.txt
highchartsLanguage.js
Always
Never
Never
Never
Yes
Never
Never
Never
Charts[2]





graphsmisc.txt
graphstemp.txt
graphsrain.txt

graphswind.txt
graphssolar.txt
periodic
periodic
yesterday rain OR
periodic
periodic
periodic
If generated
if generated
if generated

if generated
if generated
Top10 Top10Table.txt when dirty / periodic[3] when dirty / periodic1[4]
Records records.txt when dirty when dirty
Day Records dayrecords.txt when dirty when dirty
NOAA noaa.txt 2d day of month 2d day of month
Forecast forecast.txt always always
Yadr yadr.txt
yadrtemp{year}.txt
yadrrain{year}.txt
yadrwind{year}.txt
yadrwindrun{year}.txt
yadrpress{year}.txt
yadrhum{year}
2d day of year
current year
current year
current year
current year
current year
current year
2d day of year
current year
current year
current year
current year
current year
current year
AirQuality airquality.txt Never Never
StationMap Stationmap.txt Never Never
  1. When periodic see inifile parameters section Thrifty. Period in days.
  2. If all charts of a group are switched off then no menu is generated and the corresponding charts are neither generated nor uploaded.
  3. Dirty means that yesterday had a record. The consequence of this is that thrifty works best with a daily run of CumulusUtils with the Thrifty qualifier.
  4. Periodic to remove accented records if 30 days have past