README.md

   1 InfoEx AutoWx (IEAW)
   2 =============
   3
   4 This program fetches data from an NRCS SNOTEL or MesoWest station, or
   5 your own custom data source, and pushes it into the InfoEx system using
   6 the new automated weather system implementation.
   7
   8 Licensed under the ISC license (see file: LICENSE).
   9
  10 Disclaimer
  11 ----------
  12
  13 Your usage of the NRCS, MesoWest, and/or InfoEx systems is bound by
  14 their respective terms and this software makes no attempt or claim to
  15 abide by any such terms.
  16
  17 Installation
  18 ------------
  19
  20 It's recommended to use venv and pip with this program. Here's a typical
  21 sequence of commands for a new setup:
  22
  23 `$ cd /path/to/src`
  24 `$ python3 -m venv env`
  25 `$ . env/bin/activate`
  26 `$ pip install -r requirements.txt`
  27
  28 How to use it
  29 -------------
  30
  31 This program is designed to be run from the command line (or via
  32 cron(8)) and administered via a simple, concise configuration file.
  33
  34 This design allows you to run a separate program instance for each NRCS
  35 or MesoWest weather station from which you'd like to automate the
  36 importation of data into your InfoEx subscriber account.
  37
  38 To get started, copy the included example config file
  39 (`config.ini.example` in the root source directoy) and modify the values
  40 for your own use.
  41
  42 To run ad-hoc (be sure to activate the virtual environment, as detailed in the
  43 Installation section):
  44
  45 `./infoex-autowx.py --config path/to/config-file.ini [--dry-run] [--log-level debug|info|warning]`
  46
  47 **NOTE: Specifying --dry-run will not clean up the generated CSV file.**
  48 This is so that you can more easily debug any issues that arise in the
  49 setup process.
  50
  51 You can also specify `--log-level` as debug, info, warning. The
  52 log messages produced by the program will try to be logged to journald,
  53 but if that's not available, they will be printed to stdout. This output
  54 can be helpful early on in the setup process.
  55
  56 Automation
  57 ----------
  58
  59 Here's an example of a crontab(5) with two SNOTEL sites, each of which
  60 will run once per hour (note that this will activate the virtual environment
  61 created earlier):
  62
  63 `2 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config laurance-lake.ini'`
  64 `4 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config mud-ridge.ini'`
  65
  66 Configuration File
  67 ------------------
  68
  69 The configuration file is separated into two parts, the [station]
  70 portion, and the [infoex] portion.
  71
  72 The [station] values describe which weather station's data you're after.
  73 See the next section in this README for instructions on obtaining these
  74 values.
  75
  76 The [infoex] values describe your credentials for the InfoEx automated
  77 weather station FTP server and other InfoEx-related configuration
  78 options.
  79
  80 `[station]`
  81 `type = # mesowest, nrcs, or python #`
  82 `token = # MesoWest API token -- only applies when type is mesowest #`
  83 `station_id = # the NRCS/MesoWest identifier for a particular station #`
  84 `desired_data = # a comma-delimited list of fields you're interested in #`
  85 `units = # either english or metric -- only applies when type is mesowest #`
  86 `tz = # any entry from the Olson tz database e.g. America/Denver #`
  87 `path = # the filesystem path to the Python program -- only applies when type is python #`
  88
  89 `[infoex]`
  90 `host = # InfoEx FTP host address #`
  91 `uuid = # InfoEx-supplied UUID #`
  92 `api_key = # InfoEx-supplied API Key #`
  93 `csv_filename = # arbitrary name of the file that will be uploaded to InfoEx #`
  94 `location_uuid = # the UUID used by InfoEx to identify your automated Wx site #`
  95
  96 Finding your NRCS `station` values
  97 ----------------------------------
  98
  99 To complete the [station] configuration section for an NRCS station, you
 100 must fill in the attributes of the NRCS SNOTEL site from which you want
 101 to import data.
 102
 103 Here are the steps to do that:
 104
 105 1. Find your station by clicking through this website:
 106
 107    https://www.wcc.nrcs.usda.gov/snow/sntllist.html
 108
 109    Or by using the interactive map:
 110
 111    https://www.nrcs.usda.gov/wps/portal/wcc/home/quicklinks/imap
 112
 113 2. Once you've found your station, look for the "Station ID" (a 3- or
 114    4-digit number).
 115
 116 3. Combine your Station ID, state abbreviation, and the network type
 117    "SNTL" to get your NRCS station triplet (`station_id`, in the
 118    configuration file). For example:
 119
 120    655:OR:SNTL
 121
 122    would represent the Mud Ridge station (Station ID 655) in the state
 123    of Oregon (OR). SNTL just represents that the station is in the
 124    SNOTEL network and is used internally by NRCS.
 125
 126 Once you have your station ID, fill in the field in your configuration
 127 file. Now you must select which data you'd like to pull from NRCS to
 128 push into InfoEx.
 129
 130 For that, visit the NRCS web service:
 131
 132 https://wcc.sc.egov.usda.gov/awdbWebService/webservice/testwebservice.jsf?webserviceName=/awdbWebService
 133
 134 Click "getElements" on the left, and then click, "Test Operation." This
 135 will return a long list of elements to your web browser from which you
 136 can choose. Each returned element has its identifier and a description.
 137
 138 Once you've chosen your elements, combine all of their respective
 139 "elementCd" values into a comma-delimited string and put that into your
 140 configuration file as the `desired_data` value.
 141
 142 A complete [station] section example:
 143
 144 `[station]`
 145 `type = nrcs`
 146 `station_id = 655:OR:SNTL`
 147 `desired_data = TOBS,PREC`
 148
 149 indicates that I'd like to import "AIR TEMPERATURE OBSERVED" and
 150 "PRECIPITATION ACCUMULATION" from the NRCS SNOTEL site at Mud Ridge, OR,
 151 into InfoEx.
 152
 153 Finding your MesoWest `station` values
 154 --------------------------------------
 155
 156 MesoWest has great documentation which can be found here:
 157
 158 https://developers.synopticdata.com/mesonet/v2/getting-started/
 159
 160 To complete the [station] configuration section for a MesoWest station,
 161 you must fill in the attributes of the MesoWest station ID from which
 162 you want to import data. Here are the steps to do that:
 163
 164 1. Firstly, get set up with MesoWest's API by going to the above
 165    'Getting Started' link. Once you're set up, you can copy a token from
 166    the MesoWest web portal into your configuration file's `token` value.
 167
 168 2. Next, you will want to find the Station ID for the MesoWest weather
 169    station of interest and copy it to the `station_id` value.
 170
 171 3. Finally, you must choose what data types you want to push into
 172    InfoEx and compile them into a comma-separated list. MesoWest refers
 173    to these as 'field names' or 'station variables' and a list is
 174    available here:
 175
 176 https://developers.synopticdata.com/about/station-variables/
 177
 178 The MesoWest API supports on-the-fly unit conversion. If desired, that
 179 can be specified to infoex-autowx via the configuration option `units`.
 180 This can be either 'english' or 'metric', with 'english' meaning
 181 imperial units as used in the United States.
 182
 183 A complete [station] section example:
 184
 185 `[station]`
 186 `type = mesowest`
 187 `token = # token id copied from MesoWest web account #`
 188 `station_id = OD110`
 189 `desired_data = air_temp,snow_depth`
 190 `units = english`
 191
 192 indicates that I'd like to import "Temperature" and "Precipitation
 193 accumulated" from the MesoWest station at Santiam Pass, OR, into InfoEx,
 194 and that I want that data in imperial units.
 195
 196 Custom weather station support
 197 ------------------------------
 198
 199 This program supports custom weather station data by allowing the user
 200 to specify the path to an external Python program. The external Python
 201 program should emit its data in the form expected by infoex-autowx.
 202
 203 This is a powerful feature which enables the user to upload data from
 204 any source imaginable into InfoEx. Common examples are a local database
 205 or a remote web page which requires some custom parsing.
 206
 207 Please see the program located at examples/custom-wx.example.py for a
 208 complete description of what's required.
 209
 210 A note on time zones
 211 --------------------
 212
 213 This program is aware of time zones via the pytz library. The way in
 214 which NRCS and MesoWest deal with time zones differs as follows:
 215
 216 NRCS expects the request to come in the appropriate time zone, and the
 217 data retrieved will be in the same time zone (no transformation
 218 required before sending to InfoEx).
 219
 220 MesoWest expects the request to come in UTC, and the data retrieved will
 221 be in the same time zone (transformation from UTC to the desired time
 222 zone is required before sending to InfoEx).
 223
 224 As long as you specify the correct timezone in your configuration file,
 225 all will be handled correctly. The list of time zones comes from the
 226 Olson tz database. See that for more information.
 227
 228 If you specify an invalid time zone, the program will exit and inform
 229 you of such.
 230
 231 Lastly, InfoEx itself is timezone aware. If you notice that the data
 232 which makes it into your operation is inaccurate, start your
 233 investigation with time zone-related issues and move on only once you've
 234 ruled this out as a cause of the inaccuracy.
 235
 236 Unit conversions
 237 ----------------
 238
 239 Desired units may be specified in the configuration file.
 240
 241 For MesoWest, the desired unit will be passed along in the API request
 242 and the conversion will take place through the MesoWest/Synoptic API.
 243
 244 For NRCS, this program will do the conversion manually, as NRCS does not
 245 permit specifying the desired unit.
 246
 247 A note on supported measurements
 248 --------------------------------
 249
 250 While this program supports several measurements, and will faithfully
 251 request all of the ones you specify (provided they're supported), the
 252 weather station may not record them. In this case, the data will simply
 253 be ignored (i.e. it will NOT log "0" when there's no measurement
 254 available).
 255
 256 InfoEx provides a mechanism for inspecting your automated weather
 257 station data, so use that after setting this program up and compare it
 258 with the data you see in your web browser.
 259
 260 Here's the list of measurements currently supported:
 261
 262 **NRCS:**
 263 PREC
 264 TOBS
 265 SNWD
 266 PRES
 267 RHUM
 268 WSPD
 269 WDIR
 270
 271 **MesoWest:**
 272 precip\_accum
 273 air\_temp
 274 snow\_depth
 275 pressure
 276 relative\_humidity
 277 wind\_speed
 278 wind\_direction
 279 wind\_gust
 280
 281 **Custom Wx program**
 282 *infoex-autowx expects a custom Wx data provider to provide at least one
 283 of the following:*
 284 precipitationGauge
 285 tempPres
 286 tempMaxHour
 287 tempMinHour
 288 hS
 289 baro
 290 rH
 291 windSpeedNum
 292 windDirectionNum
 293 windGustSpeedNum
 294
 295 Version history
 296 ---------------
 297
 298 - 3.2.2 (Feb 2021)
 299
 300   Various small fixes.
 301
 302   - Round precipitation accumulation values to 2 decimal places.
 303   - Catch requests' ConnectionException.
 304   - Improve logging output when using stdout.
 305
 306 - 3.2.1 (Feb 2021)
 307
 308   Fix config validation bug with units and custom Python program.
 309
 310 - 3.2.0 (Feb 2021)
 311
 312   Implement NRCS unit conversion.
 313
 314 - 3.1.1 (Feb 2021)
 315
 316   Fix relative humidity rounding.
 317
 318 - 3.1.0 (Jan 2021)
 319
 320   Implement time zone support.
 321
 322 - 3.0.2 (Jan 2021)
 323
 324   Use UTC time when asking MesoWest for data.
 325
 326 - 3.0.1 (Jan 2021)
 327
 328   General fixes.
 329
 330   - MesoWest wind data (speed and gust speed) units are now transformed
 331     from their origin unit (meters per second) to the unit expected by
 332     InfoEx (miles per hour).
 333
 334   - Relative humidity is now rounded to one decimal place, preventing
 335     InfoEx from reddening the auto-filled value.
 336
 337 - 3.0.0 (Nov 2020)
 338
 339   Implement Custom Wx data providers.
 340
 341   This release enables the user to write their own Python programs and
 342   specify them to infoex-autowx as a data provider.
 343
 344   This in turn enables the user to pull data from e.g. a local database
 345   or an HTML page and push it into their InfoEx auto station data,
 346   limited only by the imagination.
 347
 348 - 2.2.0 (Nov 2020)
 349
 350   Add support for Tmin/Tmax values (directly from MesoWest/NRCS).
 351
 352 - 2.1.0 (Nov 2020)
 353
 354   Adjust precision of certain values before sending them to InfoEx.
 355
 356 - 2.0.2 (Jul 2020)
 357
 358   Fix issues shown by pylint(1).
 359
 360 - 2.0.1 (Jul 2020)
 361
 362   Major restructuring, but nothing which should impact the end user.
 363
 364   - Took the monolithic main () routine and broke it out into logical
 365     components.
 366   - Improved the names of variables.
 367
 368 - 2.0.0 (Jul 2020)
 369
 370   Implement MesoWest integration.
 371
 372   This release also makes significant changes to the configuration file,
 373   hence the major version bump. Such changes are not taken lightly but
 374   given the desire to support multiple data sources, were necessary.
 375
 376   Other minor changes include:
 377
 378   - New switches: --log-level and --version.
 379   - Better documentation.
 380   - Expanded supported measurement types (from three to eight, in number).
 381
 382 - 1.0.0 (Jun 2020)
 383
 384   First released version. Cleaned up the program and design.
 385   Implemented configuration file, added LICENSE, README, docs, etc.
 386
 387 - 0.8.0 (Apr 2020)
 388
 389   First finished (unreleased) version.
 390
 391 - pre-0.8.0 (Apr 2020)
 392
 393   First (private) finished implementation with successful importation of
 394   NRCS data into InfoEx.