README.md

   1 InfoEx AutoWx (IEAW)
   2 =============
   3
   4 This program fetches data from an NRCS SNOTEL or MesoWest station, or
   5 your own custom data source, and pushes it into the InfoEx system using
   6 the new automated weather system implementation.
   7
   8 Licensed under the ISC license (see file: LICENSE).
   9
  10 Disclaimer
  11 ----------
  12
  13 Your usage of the NRCS, MesoWest, and/or InfoEx systems is bound by
  14 their respective terms and this software makes no attempt or claim to
  15 abide by any such terms.
  16
  17 Installation
  18 ------------
  19
  20 It's recommended to use venv and pip with this program. Here's a typical
  21 sequence of commands for a new setup:
  22
  23 `$ cd /path/to/src`
  24 `$ python3 -m venv env`
  25 `$ . env/bin/activate`
  26 `$ pip install -r requirements.txt`
  27
  28 How to use it
  29 -------------
  30
  31 This program is designed to be run from the command line (or via
  32 cron(8)) and administered via a simple, concise configuration file.
  33
  34 This design allows you to run a separate program instance for each NRCS
  35 or MesoWest weather station from which you'd like to automate the
  36 importation of data into your InfoEx subscriber account.
  37
  38 To get started, copy the included example config file
  39 (`config.ini.example` in the root source directoy) and modify the values
  40 for your own use.
  41
  42 To run ad-hoc (be sure to activate the virtual environment, as detailed in the
  43 Installation section):
  44
  45 `./infoex-autowx.py --config path/to/config-file.ini [--dry-run] [--log-level debug|info|warning]`
  46
  47 **NOTE: Specifying --dry-run will not clean up the generated CSV file.**
  48 This is so that you can more easily debug any issues that arise in the
  49 setup process.
  50
  51 You can also specify `--log-level` as debug, info, warning. The
  52 log messages produced by the program will try to be logged to journald,
  53 but if that's not available, they will be printed to stdout. This output
  54 can be helpful early on in the setup process.
  55
  56 Automation
  57 ----------
  58
  59 Here's an example of a crontab(5) with two SNOTEL sites, each of which
  60 will run once per hour (note that this will activate the virtual environment
  61 created earlier):
  62
  63 `2 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config laurance-lake.ini'`
  64 `4 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config mud-ridge.ini'`
  65
  66 Configuration File
  67 ------------------
  68
  69 The configuration file is separated into two parts, the [station]
  70 portion, and the [infoex] portion.
  71
  72 The [station] values describe which weather station's data you're after.
  73 See the next section in this README for instructions on obtaining these
  74 values.
  75
  76 The [infoex] values describe your credentials for the InfoEx automated
  77 weather station FTP server and other InfoEx-related configuration
  78 options.
  79
  80 `[station]`
  81 `type = # mesowest, nrcs, or python #`
  82 `token = # MesoWest API token -- only applies when type is mesowest #`
  83 `station_id = # the NRCS/MesoWest identifier for a particular station #`
  84 `desired_data = # a comma-delimited list of fields you're interested in #`
  85 `units = # either english or metric -- only applies when type is mesowest #`
  86 `path = # the filesystem path to the Python program -- only applies when type is python #`
  87
  88 `[infoex]`
  89 `host = # InfoEx FTP host address #`
  90 `uuid = # InfoEx-supplied UUID #`
  91 `api_key = # InfoEx-supplied API Key #`
  92 `csv_filename = # arbitrary name of the file that will be uploaded to InfoEx #`
  93 `location_uuid = # the UUID used by InfoEx to identify your automated Wx site #`
  94
  95 Finding your NRCS `station` values
  96 ----------------------------------
  97
  98 To complete the [station] configuration section for an NRCS station, you
  99 must fill in the attributes of the NRCS SNOTEL site from which you want
 100 to import data.
 101
 102 Here are the steps to do that:
 103
 104 1. Find your station by clicking through this website:
 105
 106    https://www.wcc.nrcs.usda.gov/snow/sntllist.html
 107
 108    Or by using the interactive map:
 109
 110    https://www.nrcs.usda.gov/wps/portal/wcc/home/quicklinks/imap
 111
 112 2. Once you've found your station, look for the "Station ID" (a 3- or
 113    4-digit number).
 114
 115 3. Combine your Station ID, state abbreviation, and the network type
 116    "SNTL" to get your NRCS station triplet (`station_id`, in the
 117    configuration file). For example:
 118
 119    655:OR:SNTL
 120
 121    would represent the Mud Ridge station (Station ID 655) in the state
 122    of Oregon (OR). SNTL just represents that the station is in the
 123    SNOTEL network and is used internally by NRCS.
 124
 125 Once you have your station ID, fill in the field in your configuration
 126 file. Now you must select which data you'd like to pull from NRCS to
 127 push into InfoEx.
 128
 129 For that, visit the NRCS web service:
 130
 131 https://wcc.sc.egov.usda.gov/awdbWebService/webservice/testwebservice.jsf?webserviceName=/awdbWebService
 132
 133 Click "getElements" on the left, and then click, "Test Operation." This
 134 will return a long list of elements to your web browser from which you
 135 can choose. Each returned element has its identifier and a description.
 136
 137 Once you've chosen your elements, combine all of their respective
 138 "elementCd" values into a comma-delimited string and put that into your
 139 configuration file as the `desired_data` value.
 140
 141 A complete [station] section example:
 142
 143 `[station]`
 144 `type = nrcs`
 145 `station_id = 655:OR:SNTL`
 146 `desired_data = TOBS,PREC`
 147
 148 indicates that I'd like to import "AIR TEMPERATURE OBSERVED" and
 149 "PRECIPITATION ACCUMULATION" from the NRCS SNOTEL site at Mud Ridge, OR,
 150 into InfoEx.
 151
 152 Finding your MesoWest `station` values
 153 --------------------------------------
 154
 155 MesoWest has great documentation which can be found here:
 156
 157 https://developers.synopticdata.com/mesonet/v2/getting-started/
 158
 159 To complete the [station] configuration section for a MesoWest station,
 160 you must fill in the attributes of the MesoWest station ID from which
 161 you want to import data. Here are the steps to do that:
 162
 163 1. Firstly, get set up with MesoWest's API by going to the above
 164    'Getting Started' link. Once you're set up, you can copy a token from
 165    the MesoWest web portal into your configuration file's `token` value.
 166
 167 2. Next, you will want to find the Station ID for the MesoWest weather
 168    station of interest and copy it to the `station_id` value.
 169
 170 3. Finally, you must choose what data types you want to push into
 171    InfoEx and compile them into a comma-separated list. MesoWest refers
 172    to these as 'field names' or 'station variables' and a list is
 173    available here:
 174
 175 https://developers.synopticdata.com/about/station-variables/
 176
 177 The MesoWest API supports on-the-fly unit conversion. If desired, that
 178 can be specified to infoex-autowx via the configuration option `units`.
 179 This can be either 'english' or 'metric', with 'english' meaning
 180 imperial units as used in the United States.
 181
 182 A complete [station] section example:
 183
 184 `[station]`
 185 `type = mesowest`
 186 `token = # token id copied from MesoWest web account #`
 187 `station_id = OD110`
 188 `desired_data = air_temp,snow_depth`
 189 `units = english`
 190
 191 indicates that I'd like to import "Temperature" and "Precipitation
 192 accumulated" from the MesoWest station at Santiam Pass, OR, into InfoEx,
 193 and that I want that data in imperial units.
 194
 195 Custom weather station support
 196 ------------------------------
 197
 198 This program supports custom weather station data by allowing the user
 199 to specify the path to an external Python program. The external Python
 200 program should emit its data in the form expected by infoex-autowx.
 201
 202 This is a powerful feature which enables the user to upload data from
 203 any source imaginable into InfoEx. Common examples are a local database
 204 or a remote web page which requires some custom parsing.
 205
 206 Please see the program located at examples/custom-wx.example.py for a
 207 complete description of what's required.
 208
 209 A note on supported measurements
 210 --------------------------------
 211
 212 While this program supports several measurements, and will faithfully
 213 request all of the ones you specify (provided they're supported), the
 214 weather station may not record them. In this case, the data will simply
 215 be ignored (i.e. it will NOT log "0" when there's no measurement
 216 available).
 217
 218 InfoEx provides a mechanism for inspecting your automated weather
 219 station data, so use that after setting this program up and compare it
 220 with the data you see in your web browser.
 221
 222 Here's the list of measurements currently supported:
 223
 224 **NRCS:**
 225 PREC
 226 TOBS
 227 SNWD
 228 PRES
 229 RHUM
 230 WSPD
 231 WDIR
 232
 233 **MesoWest:**
 234 precip\_accum
 235 air\_temp
 236 snow\_depth
 237 pressure
 238 relative\_humidity
 239 wind\_speed
 240 wind\_direction
 241 wind\_gust
 242
 243 **Custom Wx program**
 244 *infoex-autowx expects a custom Wx data provider to provide at least one
 245 of the following:*
 246 precipitationGauge
 247 tempPres
 248 tempMaxHour
 249 tempMinHour
 250 hS
 251 baro
 252 rH
 253 windSpeedNum
 254 windDirectionNum
 255 windGustSpeedNum
 256
 257 Future plans
 258 ------------
 259
 260 - Implement unit conversion for NRCS stations
 261
 262 Version history
 263 ---------------
 264
 265 - 3.0.2 (Jan 2021)
 266
 267   Use UTCC time when asking MesoWest for data.
 268
 269 - 3.0.1 (Jan 2021)
 270
 271   General fixes.
 272
 273   - MesoWest wind data (speed and gust speed) units are now transformed
 274     from their origin unit (meters per second) to the unit expected by
 275     InfoEx (miles per hour).
 276
 277   - Relative humidity is now rounded to one decimal place, preventing
 278     InfoEx from reddening the auto-filled value.
 279
 280 - 3.0.0 (Nov 2020)
 281
 282   Implement Custom Wx data providers.
 283
 284   This release enables the user to write their own Python programs and
 285   specify them to infoex-autowx as a data provider.
 286
 287   This in turn enables the user to pull data from e.g. a local database
 288   or an HTML page and push it into their InfoEx auto station data,
 289   limited only by the imagination.
 290
 291 - 2.2.0 (Nov 2020)
 292
 293   Add support for Tmin/Tmax values (directly from MesoWest/NRCS).
 294
 295 - 2.1.0 (Nov 2020)
 296
 297   Adjust precision of certain values before sending them to InfoEx.
 298
 299 - 2.0.2 (Jul 2020)
 300
 301   Fix issues shown by pylint(1).
 302
 303 - 2.0.1 (Jul 2020)
 304
 305   Major restructuring, but nothing which should impact the end user.
 306
 307   - Took the monolithic main () routine and broke it out into logical
 308     components.
 309   - Improved the names of variables.
 310
 311 - 2.0.0 (Jul 2020)
 312
 313   Implement MesoWest integration.
 314
 315   This release also makes significant changes to the configuration file,
 316   hence the major version bump. Such changes are not taken lightly but
 317   given the desire to support multiple data sources, were necessary.
 318
 319   Other minor changes include:
 320
 321   - New switches: --log-level and --version.
 322   - Better documentation.
 323   - Expanded supported measurement types (from three to eight, in number).
 324
 325 - 1.0.0 (Jun 2020)
 326
 327   First released version. Cleaned up the program and design.
 328   Implemented configuration file, added LICENSE, README, docs, etc.
 329
 330 - 0.8.0 (Apr 2020)
 331
 332   First finished (unreleased) version.
 333
 334 - pre-0.8.0 (Apr 2020)
 335
 336   First (private) finished implementation with successful importation of
 337   NRCS data into InfoEx.