README.md

   1 InfoEx AutoWx (IEAW)
   2 =============
   3
   4 This program fetches data from an NRCS SNOTEL or MesoWest station and
   5 pushes it into the InfoEx system using the new automated weather system
   6 implementation.
   7
   8 License under the ISC license (see file: LICENSE).
   9
  10 Disclaimer
  11 ----------
  12
  13 Your usage of the NRCS, MesoWest, and/or InfoEx systems is bound by
  14 their respective terms and this software makes no attempt or claim to
  15 abide by any such terms.
  16
  17 Installation
  18 ------------
  19
  20 It's recommended to use venv and pip with this program. Here's a typical
  21 sequence of commands for a new setup:
  22
  23 `$ cd /path/to/src`
  24 `$ python3 -m venv env`
  25 `$ . env/bin/activate`
  26 `$ pip install -r requirements.txt`
  27
  28 How to use it
  29 -------------
  30
  31 This program is designed to be run from the command line (or via
  32 cron(8)) and administered via a simple, concise configuration file.
  33
  34 This design allows you to run a separate program instance for each NRCS
  35 or MesoWest weather station from which you'd like to automate the
  36 importation of data into your InfoEx subscriber account.
  37
  38 To run ad-hoc (be sure to activate the virtual environment, as detailed in the
  39 Installation section):
  40
  41 `./infoex-autowx.py --config path/to/config-file.ini [--dry-run] [--log-level debug|info|warning]`
  42
  43 **NOTE: Specifying --dry-run will also not clean up the generated CSV
  44 file.** This is so that you can debug any issues more easily.
  45
  46 You can also specify `--log-level` as debug, info, warning. The
  47 resultant log messages will try to be logged to journald, but if that's
  48 not available, they will be printed to stdout. This output can be
  49 helpful early on in the setup process.
  50
  51 Here's an example of a crontab(5) with two SNOTEL sites, each of which
  52 will run once per hour (note that this will activate the virtual environment
  53 created earlier):
  54
  55 `2 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config laurance-lake.ini'`
  56 `4 * * * * /usr/bin/env bash -c 'cd /home/user/infoex-autowx && source env/bin/activate && ./infoex-autowx.py --config mud-ridge.ini'`
  57
  58 Configuration File
  59 ------------------
  60
  61 The configuration file is separated into two parts, the [station]
  62 portion, and the [infoex] portion.
  63
  64 The [station] values describe which weather station's data you're after.
  65 See the next section in this README for instructions on obtaining these
  66 values.
  67
  68 The [infoex] values describe your credentials for the InfoEx automated
  69 weather station FTP server and other InfoEx-related configuration
  70 options.
  71
  72 `[station]`
  73 `type = # either mesowest or nrcs #`
  74 `token = # MesoWest API token -- ignored when type is nrcs #`
  75 `station_id = # the NRCS/MesoWest identifier for a particular station #`
  76 `desired_data = # a comma-delimited list of fields you're interested in #`
  77 `units = # either english or metric -- ignored when type is nrcs #`
  78
  79 `[infoex]`
  80 `host = # InfoEx FTP host address #`
  81 `uuid = # InfoEx-supplied UUID #`
  82 `api_key = # InfoEx-supplied API Key #`
  83 `csv_filename = # arbitrary name of the file that will be uploaded to InfoEx #`
  84 `location_uuid = # the UUID used by InfoEx to identify your automated Wx site #`
  85
  86 Finding your NRCS `station` values
  87 ----------------------------------
  88
  89 To complete the [station] configuration section for an NRCS station, you
  90 must fill in the attributes of the NRCS SNOTEL site from which you want
  91 to import data.
  92
  93 Here are the steps to do that:
  94
  95 1. Find your station by clicking through this website:
  96
  97    https://www.wcc.nrcs.usda.gov/snow/sntllist.html
  98
  99    Or by using the interactive map:
 100
 101    https://www.nrcs.usda.gov/wps/portal/wcc/home/quicklinks/imap
 102
 103 2. Once you've found your station, look for the "Station ID" (a 3- or
 104    4-digit number).
 105
 106 3. Combine your Station ID, state abbreviation, and the network type
 107    "SNTL" to get your NRCS station triplet (`station_id`, in the
 108    configuration file). For example:
 109
 110    655:OR:SNTL
 111
 112    would represent the Mud Ridge station (Station ID 655) in the state
 113    of Oregon (OR). SNTL just represents that the station is in the
 114    SNOTEL network and is used internally by NRCS.
 115
 116 Once you have your station ID, fill in the field in your configuration
 117 file. Now you must select which data you'd like to pull from NRCS to
 118 push into InfoEx.
 119
 120 For that, visit the NRCS web service:
 121
 122 https://wcc.sc.egov.usda.gov/awdbWebService/webservice/testwebservice.jsf?webserviceName=/awdbWebService
 123
 124 Click "getElements" on the left, and then click, "Test Operation." This
 125 will return a long list of elements to your web browser from which you
 126 can choose. Each returned element has its identifier and a description.
 127
 128 Once you've chosen your elements, combine all of their respective
 129 "elementCd" values into a comma-delimited string and put that into your
 130 configuration file as the `desired_data` value.
 131
 132 A complete example:
 133
 134 `[station]`
 135 `type = nrcs`
 136 `station_id = 655:OR:SNTL`
 137 `desired_data = TOBS,PREC`
 138
 139 indicates that I'd like to import "AIR TEMPERATURE OBSERVED" and
 140 "PRECIPITATION ACCUMULATION" from the NRCS SNOTEL site at Mud Ridge, OR,
 141 into InfoEx.
 142
 143 Finding your MesoWest `station` values
 144 --------------------------------------
 145
 146 MesoWest has great documentation which can be found here:
 147
 148 https://developers.synopticdata.com/mesonet/v2/getting-started/
 149
 150 To complete the [station] configuration section for a MesoWest station,
 151 you must fill in the attributes of the MesoWest station ID from which
 152 you want to import data. Here are the steps to do that:
 153
 154 1. Firstly, get set up with MesoWest's API by going to the above
 155    'Getting Started' link. Once you're set up, you can copy a token from
 156    the MesoWest web portal into your configuration file's `token` value.
 157
 158 2. Next, you will want to find the Station ID for the MesoWest weather
 159    station of interest and copy it to the `station_id` value.
 160
 161 3. Finally, you must choose what data types you want to push into
 162    InfoEx and compile them into a comma-separated list. MesoWest refers
 163    to these as 'field names' or 'station variables' and a list is
 164    available here:
 165
 166 https://developers.synopticdata.com/about/station-variables/
 167
 168 The MesoWest API supports on-the-fly unit conversion, so that can be
 169 specified to infoex-autowx via the configuration option `units`. This
 170 can be either 'english' or 'metric', with 'english' meaning imperial
 171 units as used in the United States.
 172
 173 A complete example:
 174
 175 `[station]`
 176 `type = mesowest`
 177 `token = # token id copied from MesoWest web account #`
 178 `station_id = OD110`
 179 `desired_data = air_temp,snow_depth`
 180 `units = english`
 181
 182 indicates that I'd like to import "Temperature" and "Precipitation
 183 accumulated" from the MesoWest station at Santiam Pass, OR, into InfoEx,
 184 and that I want that data in imperial units.
 185
 186 A note on supported measurements
 187 --------------------------------
 188
 189 While this program supports several measurements, and will faithfully
 190 request all of the ones you specify (provided they're supported), the
 191 weather station may not record them. In this case, the data will simply
 192 be ignored (i.e. it will NOT log "0" when there's no measurement
 193 available).
 194
 195
 196 InfoEx provides a mechanism for inspecting your automated weather
 197 station data, so use that after setting this program up and compare it
 198 with the data you see in your web browser.
 199
 200 Here's the list of measurements currently supported:
 201
 202 **NRCS:**
 203 PREC
 204 TOBS
 205 SNWD
 206 PRES
 207 RHUM
 208 WSPD
 209 WDIR
 210
 211 **MesoWest:**
 212 precip\_accum
 213 air\_temp
 214 snow\_depth
 215 pressure
 216 relative\_humidity
 217 wind\_speed
 218 wind\_direction
 219 wind\_gust
 220
 221 Future plans
 222 ------------
 223
 224 - Improve the documentation
 225 - Implement unit conversion for NRCS stations
 226
 227 Version history
 228 ---------------
 229
 230 - 2.0.0 (Jul 2020)
 231
 232   Implement MesoWest integration.
 233
 234   This release also makes significant changes to the configuration file,
 235   hence the major version bump. Such changes are not taken lightly but
 236   given the desire to support multiple data sources, were necessary.
 237
 238 - 1.0.0 (Jun 2020)
 239
 240   First released version. Cleaned up the program and design.
 241   Implemented configuration file, added LICENSE, README, docs, etc.
 242
 243 - 0.8.0 (Apr 2020)
 244
 245   First finished (unreleased) version.
 246
 247 - pre-0.8.0 (Apr 2020)
 248
 249   First (private) finished implementation with successful importation of
 250   NRCS data into InfoEx.