README.md

   1 InfoEx AutoWx (IEAW)
   2 =============
   3
   4 This program fetches data from an NRCS SNOTEL or MesoWest station and
   5 pushes it into the InfoEx system using the new automated weather system
   6 implementation.
   7
   8 Licensed under the ISC license (see file: LICENSE).
   9
  10 Disclaimer
  11 ----------
  12
  13 Your usage of the NRCS, MesoWest, and/or InfoEx systems is bound by
  14 their respective terms and this software makes no attempt or claim to
  15 abide by any such terms.
  16
  17 Installation
  18 ------------
  19
  20 It's recommended to use venv and pip with this program. Here's a typical
  21 sequence of commands for a new setup:
  22
  23 `$ cd /path/to/src`
  24 `$ python3 -m venv env`
  25 `$ . env/bin/activate`
  26 `$ pip install -r requirements.txt`
  27
  28 How to use it
  29 -------------
  30
  31 This program is designed to be run from the command line (or via
  32 cron(8)) and administered via a simple, concise configuration file.
  33
  34 This design allows you to run a separate program instance for each NRCS
  35 or MesoWest weather station from which you'd like to automate the
  36 importation of data into your InfoEx subscriber account.
  37
  38 To get started, copy the included example config file
  39 (`config.ini.example` in the root source directoy) and modify the values
  40 for your own use.
  41
  42 To run ad-hoc (be sure to activate the virtual environment, as detailed in the
  43 Installation section):
  44
  45 `./infoex-autowx.py --config path/to/config-file.ini [--dry-run] [--log-level debug|info|warning]`
  46
  47 **NOTE: Specifying --dry-run will not clean up the generated CSV file.**
  48 This is so that you can more easily debug any issues that arise in the
  49 setup process.
  50
  51 You can also specify `--log-level` as debug, info, warning. The
  52 log messages produced by the program will try to be logged to journald,
  53 but if that's not available, they will be printed to stdout. This output
  54 can be helpful early on in the setup process.
  55
  56 Automation
  57 ----------
  58
  59 Here's an example of a crontab(5) with two SNOTEL sites, each of which
  60 will run once per hour (note that this will activate the virtual environment
  61 created earlier):
  62
  63 `2 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config laurance-lake.ini'`
  64 `4 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config mud-ridge.ini'`
  65
  66 Configuration File
  67 ------------------
  68
  69 The configuration file is separated into two parts, the [station]
  70 portion, and the [infoex] portion.
  71
  72 The [station] values describe which weather station's data you're after.
  73 See the next section in this README for instructions on obtaining these
  74 values.
  75
  76 The [infoex] values describe your credentials for the InfoEx automated
  77 weather station FTP server and other InfoEx-related configuration
  78 options.
  79
  80 `[station]`
  81 `type = # either mesowest or nrcs #`
  82 `token = # MesoWest API token -- ignored when type is nrcs #`
  83 `station_id = # the NRCS/MesoWest identifier for a particular station #`
  84 `desired_data = # a comma-delimited list of fields you're interested in #`
  85 `units = # either english or metric -- ignored when type is nrcs #`
  86
  87 `[infoex]`
  88 `host = # InfoEx FTP host address #`
  89 `uuid = # InfoEx-supplied UUID #`
  90 `api_key = # InfoEx-supplied API Key #`
  91 `csv_filename = # arbitrary name of the file that will be uploaded to InfoEx #`
  92 `location_uuid = # the UUID used by InfoEx to identify your automated Wx site #`
  93
  94 Finding your NRCS `station` values
  95 ----------------------------------
  96
  97 To complete the [station] configuration section for an NRCS station, you
  98 must fill in the attributes of the NRCS SNOTEL site from which you want
  99 to import data.
 100
 101 Here are the steps to do that:
 102
 103 1. Find your station by clicking through this website:
 104
 105    https://www.wcc.nrcs.usda.gov/snow/sntllist.html
 106
 107    Or by using the interactive map:
 108
 109    https://www.nrcs.usda.gov/wps/portal/wcc/home/quicklinks/imap
 110
 111 2. Once you've found your station, look for the "Station ID" (a 3- or
 112    4-digit number).
 113
 114 3. Combine your Station ID, state abbreviation, and the network type
 115    "SNTL" to get your NRCS station triplet (`station_id`, in the
 116    configuration file). For example:
 117
 118    655:OR:SNTL
 119
 120    would represent the Mud Ridge station (Station ID 655) in the state
 121    of Oregon (OR). SNTL just represents that the station is in the
 122    SNOTEL network and is used internally by NRCS.
 123
 124 Once you have your station ID, fill in the field in your configuration
 125 file. Now you must select which data you'd like to pull from NRCS to
 126 push into InfoEx.
 127
 128 For that, visit the NRCS web service:
 129
 130 https://wcc.sc.egov.usda.gov/awdbWebService/webservice/testwebservice.jsf?webserviceName=/awdbWebService
 131
 132 Click "getElements" on the left, and then click, "Test Operation." This
 133 will return a long list of elements to your web browser from which you
 134 can choose. Each returned element has its identifier and a description.
 135
 136 Once you've chosen your elements, combine all of their respective
 137 "elementCd" values into a comma-delimited string and put that into your
 138 configuration file as the `desired_data` value.
 139
 140 A complete [station] section example:
 141
 142 `[station]`
 143 `type = nrcs`
 144 `station_id = 655:OR:SNTL`
 145 `desired_data = TOBS,PREC`
 146
 147 indicates that I'd like to import "AIR TEMPERATURE OBSERVED" and
 148 "PRECIPITATION ACCUMULATION" from the NRCS SNOTEL site at Mud Ridge, OR,
 149 into InfoEx.
 150
 151 Finding your MesoWest `station` values
 152 --------------------------------------
 153
 154 MesoWest has great documentation which can be found here:
 155
 156 https://developers.synopticdata.com/mesonet/v2/getting-started/
 157
 158 To complete the [station] configuration section for a MesoWest station,
 159 you must fill in the attributes of the MesoWest station ID from which
 160 you want to import data. Here are the steps to do that:
 161
 162 1. Firstly, get set up with MesoWest's API by going to the above
 163    'Getting Started' link. Once you're set up, you can copy a token from
 164    the MesoWest web portal into your configuration file's `token` value.
 165
 166 2. Next, you will want to find the Station ID for the MesoWest weather
 167    station of interest and copy it to the `station_id` value.
 168
 169 3. Finally, you must choose what data types you want to push into
 170    InfoEx and compile them into a comma-separated list. MesoWest refers
 171    to these as 'field names' or 'station variables' and a list is
 172    available here:
 173
 174 https://developers.synopticdata.com/about/station-variables/
 175
 176 The MesoWest API supports on-the-fly unit conversion. If desired, that
 177 can be specified to infoex-autowx via the configuration option `units`.
 178 This can be either 'english' or 'metric', with 'english' meaning
 179 imperial units as used in the United States.
 180
 181 A complete [station] section example:
 182
 183 `[station]`
 184 `type = mesowest`
 185 `token = # token id copied from MesoWest web account #`
 186 `station_id = OD110`
 187 `desired_data = air_temp,snow_depth`
 188 `units = english`
 189
 190 indicates that I'd like to import "Temperature" and "Precipitation
 191 accumulated" from the MesoWest station at Santiam Pass, OR, into InfoEx,
 192 and that I want that data in imperial units.
 193
 194 A note on supported measurements
 195 --------------------------------
 196
 197 While this program supports several measurements, and will faithfully
 198 request all of the ones you specify (provided they're supported), the
 199 weather station may not record them. In this case, the data will simply
 200 be ignored (i.e. it will NOT log "0" when there's no measurement
 201 available).
 202
 203
 204 InfoEx provides a mechanism for inspecting your automated weather
 205 station data, so use that after setting this program up and compare it
 206 with the data you see in your web browser.
 207
 208 Here's the list of measurements currently supported:
 209
 210 **NRCS:**
 211 PREC
 212 TOBS
 213 SNWD
 214 PRES
 215 RHUM
 216 WSPD
 217 WDIR
 218
 219 **MesoWest:**
 220 precip\_accum
 221 air\_temp
 222 snow\_depth
 223 pressure
 224 relative\_humidity
 225 wind\_speed
 226 wind\_direction
 227 wind\_gust
 228
 229 Future plans
 230 ------------
 231
 232 - Improve the documentation
 233 - Implement unit conversion for NRCS stations
 234
 235 Version history
 236 ---------------
 237
 238 - 2.2.0 (Nov 2020)
 239
 240   Add support for Tmin/Tmax values (directly from MesoWest/NRCS).
 241
 242 - 2.1.0 (Nov 2020)
 243
 244   Adjust precision of certain values before sending them to InfoEx.
 245
 246 - 2.0.2 (Jul 2020)
 247
 248   Fix issues shown by pylint(1).
 249
 250 - 2.0.1 (Jul 2020)
 251
 252   Major restructuring, but nothing which should impact the end user.
 253
 254   - Took the monolithic main () routine and broke it out into logical
 255     components.
 256   - Improved the names of variables.
 257
 258 - 2.0.0 (Jul 2020)
 259
 260   Implement MesoWest integration.
 261
 262   This release also makes significant changes to the configuration file,
 263   hence the major version bump. Such changes are not taken lightly but
 264   given the desire to support multiple data sources, were necessary.
 265
 266   Other minor changes include:
 267
 268   - New switches: --log-level and --version.
 269   - Better documentation.
 270   - Expanded supported measurement types (from three to eight, in number).
 271
 272 - 1.0.0 (Jun 2020)
 273
 274   First released version. Cleaned up the program and design.
 275   Implemented configuration file, added LICENSE, README, docs, etc.
 276
 277 - 0.8.0 (Apr 2020)
 278
 279   First finished (unreleased) version.
 280
 281 - pre-0.8.0 (Apr 2020)
 282
 283   First (private) finished implementation with successful importation of
 284   NRCS data into InfoEx.