InfoEx AutoWx (IEAW)
=============
-This program fetches data from an NRCS SNOTEL site and pushes it into
-the InfoEx system using the new automated weather system implementation.
+This program fetches data from an NRCS SNOTEL or MesoWest station and
+pushes it into the InfoEx system using the new automated weather system
+implementation.
-License under the MIT license (see file: LICENSE).
+Licensed under the ISC license (see file: LICENSE).
Disclaimer
----------
-Your usage of the NRCS and InfoEx systems is bound by their respective
-terms and this software makes no attempt or claim to abide by any such
-terms.
+Your usage of the NRCS, MesoWest, and/or InfoEx systems is bound by
+their respective terms and this software makes no attempt or claim to
+abide by any such terms.
Installation
------------
cron(8)) and administered via a simple, concise configuration file.
This design allows you to run a separate program instance for each NRCS
-weather station from which you'd like to automate the importation of
-data into your InfoEx subscriber account.
+or MesoWest weather station from which you'd like to automate the
+importation of data into your InfoEx subscriber account.
+
+To get started, copy the included example config file
+(`config.ini.example` in the root source directoy) and modify the values
+for your own use.
To run ad-hoc (be sure to activate the virtual environment, as detailed in the
Installation section):
-`./infoex-autowx.py --config [path/to/config-file.ini] [--dry-run]`
+`./infoex-autowx.py --config path/to/config-file.ini [--dry-run] [--log-level debug|info|warning]`
+
+**NOTE: Specifying --dry-run will not clean up the generated CSV file.**
+This is so that you can more easily debug any issues that arise in the
+setup process.
-**NOTE: Specifying --dry-run will also not clean up the generated CSV
-file.** This is so that you can debug any issues more easily.
+You can also specify `--log-level` as debug, info, warning. The
+log messages produced by the program will try to be logged to journald,
+but if that's not available, they will be printed to stdout. This output
+can be helpful early on in the setup process.
+
+Automation
+----------
Here's an example of a crontab(5) with two SNOTEL sites, each of which
will run once per hour (note that this will activate the virtual environment
Configuration File
------------------
-The configuration file is separated into two parts, the [wxsite]
-portion, and the [ftp] portion.
+The configuration file is separated into two parts, the [station]
+portion, and the [infoex] portion.
-The [wxsite] values describe which NRCS SNOTEL site's data you're after.
+The [station] values describe which weather station's data you're after.
See the next section in this README for instructions on obtaining these
values.
-The [ftp] values describe your credentials for the InfoEx automated
-weather station FTP server.
+The [infoex] values describe your credentials for the InfoEx automated
+weather station FTP server and other InfoEx-related configuration
+options.
-`[wxsite]`
-`station_triplet = [The NRCS identifier for a particular SNOTEL site]`
-`desired_data = [A comma-delimited list of NRCS elements you're interested in]`
-`location_uuid = [The UUID used by InfoEx to identify your automated Wx site]`
-`csv_filename = [Arbitrary name of the file that will be uploaded to InfoEx]`
+`[station]`
+`type = # either mesowest or nrcs #`
+`token = # MesoWest API token -- ignored when type is nrcs #`
+`station_id = # the NRCS/MesoWest identifier for a particular station #`
+`desired_data = # a comma-delimited list of fields you're interested in #`
+`units = # either english or metric -- ignored when type is nrcs #`
-`[ftp]`
-`host = [InfoEx FTP host address]`
-`uuid = [InfoEx-supplied UUID]`
-`api_key = [InfoEx-supplied API Key]`
+`[infoex]`
+`host = # InfoEx FTP host address #`
+`uuid = # InfoEx-supplied UUID #`
+`api_key = # InfoEx-supplied API Key #`
+`csv_filename = # arbitrary name of the file that will be uploaded to InfoEx #`
+`location_uuid = # the UUID used by InfoEx to identify your automated Wx site #`
-Finding Your WXSITE values
---------------------------
+Finding your NRCS `station` values
+----------------------------------
+
+To complete the [station] configuration section for an NRCS station, you
+must fill in the attributes of the NRCS SNOTEL site from which you want
+to import data.
-To complete the [wxsite] configuration section, you must fill in the
-attributes of the NRCS SNOTEL site from which you want to import data.
Here are the steps to do that:
1. Find your station by clicking through this website:
4-digit number).
3. Combine your Station ID, state abbreviation, and the network type
- "SNTL" to get your station triplet (`station_triplet`, in the
+ "SNTL" to get your NRCS station triplet (`station_id`, in the
configuration file). For example:
655:OR:SNTL
of Oregon (OR). SNTL just represents that the station is in the
SNOTEL network and is used internally by NRCS.
-Once you have your station triplet, fill in the field in your
-configuration file. Now you must select which data you'd like to pull
-from NRCS to push into InfoEx.
+Once you have your station ID, fill in the field in your configuration
+file. Now you must select which data you'd like to pull from NRCS to
+push into InfoEx.
For that, visit the NRCS web service:
"elementCd" values into a comma-delimited string and put that into your
configuration file as the `desired_data` value.
-For example:
+A complete [station] section example:
-`station_triplet = 655:OR:SNTL`
+`[station]`
+`type = nrcs`
+`station_id = 655:OR:SNTL`
`desired_data = TOBS,PREC`
indicates that I'd like to import "AIR TEMPERATURE OBSERVED" and
"PRECIPITATION ACCUMULATION" from the NRCS SNOTEL site at Mud Ridge, OR,
into InfoEx.
-Version History
+Finding your MesoWest `station` values
+--------------------------------------
+
+MesoWest has great documentation which can be found here:
+
+https://developers.synopticdata.com/mesonet/v2/getting-started/
+
+To complete the [station] configuration section for a MesoWest station,
+you must fill in the attributes of the MesoWest station ID from which
+you want to import data. Here are the steps to do that:
+
+1. Firstly, get set up with MesoWest's API by going to the above
+ 'Getting Started' link. Once you're set up, you can copy a token from
+ the MesoWest web portal into your configuration file's `token` value.
+
+2. Next, you will want to find the Station ID for the MesoWest weather
+ station of interest and copy it to the `station_id` value.
+
+3. Finally, you must choose what data types you want to push into
+ InfoEx and compile them into a comma-separated list. MesoWest refers
+ to these as 'field names' or 'station variables' and a list is
+ available here:
+
+https://developers.synopticdata.com/about/station-variables/
+
+The MesoWest API supports on-the-fly unit conversion. If desired, that
+can be specified to infoex-autowx via the configuration option `units`.
+This can be either 'english' or 'metric', with 'english' meaning
+imperial units as used in the United States.
+
+A complete [station] section example:
+
+`[station]`
+`type = mesowest`
+`token = # token id copied from MesoWest web account #`
+`station_id = OD110`
+`desired_data = air_temp,snow_depth`
+`units = english`
+
+indicates that I'd like to import "Temperature" and "Precipitation
+accumulated" from the MesoWest station at Santiam Pass, OR, into InfoEx,
+and that I want that data in imperial units.
+
+A note on supported measurements
+--------------------------------
+
+While this program supports several measurements, and will faithfully
+request all of the ones you specify (provided they're supported), the
+weather station may not record them. In this case, the data will simply
+be ignored (i.e. it will NOT log "0" when there's no measurement
+available).
+
+
+InfoEx provides a mechanism for inspecting your automated weather
+station data, so use that after setting this program up and compare it
+with the data you see in your web browser.
+
+Here's the list of measurements currently supported:
+
+**NRCS:**
+PREC
+TOBS
+SNWD
+PRES
+RHUM
+WSPD
+WDIR
+
+**MesoWest:**
+precip\_accum
+air\_temp
+snow\_depth
+pressure
+relative\_humidity
+wind\_speed
+wind\_direction
+wind\_gust
+
+Future plans
+------------
+
+- Improve the documentation
+- Implement unit conversion for NRCS stations
+
+Version history
---------------
+- 2.0.0 (Jul 2020)
+
+ Implement MesoWest integration.
+
+ This release also makes significant changes to the configuration file,
+ hence the major version bump. Such changes are not taken lightly but
+ given the desire to support multiple data sources, were necessary.
+
+ Other minor changes include:
+
+ - New switches: --log-level and --version.
+ - Better documentation.
+ - Expanded supported measurement types (from three to eight, in number).
+
- 1.0.0 (Jun 2020)
First released version. Cleaned up the program and design.
# -*- coding: utf-8 -*-
"""
-InfoEx <-> NRCS Auto Wx implementation
+InfoEx <-> NRCS/MesoWest Auto Wx implementation
Alexander Vasarab
Wylark Mountaineering LLC
-Version 1.0.0
+Version 2.0.0
-This program fetches data from an NRCS SNOTEL site and pushes it to
-InfoEx using the new automated weather system implementation.
+This program fetches data from either an NRCS SNOTEL site or MesoWest
+weather station and pushes it to InfoEx using the new automated weather
+system implementation.
It is designed to be run hourly, and it asks for the last three hours
of data of each desired type, and selects the most recent one. This
from ftplib import FTP
from optparse import OptionParser
+import requests
+
import zeep
import zeep.cache
import zeep.transports
+__version__ = '2.0.0'
+
log = logging.getLogger(__name__)
-log.setLevel(logging.DEBUG)
+log.setLevel(logging.NOTSET)
try:
from systemd.journal import JournalHandler
log.addHandler(JournalHandler())
except:
- # fallback to syslog
- import logging.handlers
- log.addHandler(logging.handlers.SysLogHandler())
+ ## fallback to syslog
+ #import logging.handlers
+ #log.addHandler(logging.handlers.SysLogHandler())
+ # fallback to stdout
+ handler = logging.StreamHandler(sys.stdout)
+ log.addHandler(handler)
+
+parser = OptionParser(version=__version__)
-parser = OptionParser()
parser.add_option("--config",
dest="config",
metavar="FILE",
help="location of config file")
+
+parser.add_option("--log-level",
+ dest="log_level",
+ default=None,
+ help="set the log level (debug, info, warning)")
+
parser.add_option("--dry-run",
action="store_true",
dest="dry_run",
config = configparser.ConfigParser(allow_no_value=False)
if not options.config:
- print("Please specify a configuration file via --config")
+ parser.print_help()
+ print("\nPlease specify a configuration file via --config.")
sys.exit(1)
config.read(options.config)
-log.debug('STARTING UP')
+# ugly, but passable
+if options.log_level in [None, 'debug', 'info', 'warning']:
+ if options.log_level == 'debug':
+ log.setLevel(logging.DEBUG)
+ elif options.log_level == 'info':
+ log.setLevel(logging.INFO)
+ elif options.log_level == 'warning':
+ log.setLevel(logging.WARNING)
+ else:
+ log.setLevel(logging.NOTSET)
+else:
+ parser.print_help()
+ print("\nPlease select an appropriate log level or remove the switch (--log-level).")
+ sys.exit(1)
-wsdl = 'https://www.wcc.nrcs.usda.gov/awdbWebService/services?WSDL'
+log.debug('STARTING UP')
try:
infoex = {
- 'host': config['ftp']['host'],
- 'uuid': config['ftp']['uuid'],
- 'api_key': config['ftp']['api_key'],
- 'location_uuid': config['wxsite']['location_uuid'],
+ 'host': config['infoex']['host'],
+ 'uuid': config['infoex']['uuid'],
+ 'api_key': config['infoex']['api_key'],
+ 'csv_filename': config['infoex']['csv_filename'],
+ 'location_uuid': config['infoex']['location_uuid'],
'wx_data': {}, # placeholder key, values to come later
- 'csv_filename': config['wxsite']['csv_filename']
}
- station_triplet = config['wxsite']['station_triplet']
+ data = dict()
+ data['provider'] = config['station']['type']
+
+ if data['provider'] not in ['nrcs', 'mesowest']:
+ print("Please specify either nrcs or mesowest as the station type.")
+ sys.exit(1)
+
+ if data['provider'] == 'nrcs':
+ data['source'] = 'https://www.wcc.nrcs.usda.gov/awdbWebService/services?WSDL'
+ data['station_id'] = config['station']['station_id']
+
+ try:
+ desired_data = config['station']['desired_data'].split(',')
+ except:
+ # desired_data malformed or missing, setting default
+ desired_data = [
+ 'TOBS', # AIR TEMPERATURE OBSERVED (degF)
+ 'SNWD', # SNOW DEPTH (in)
+ 'PREC' # PRECIPITATION ACCUMULATION (in)
+ ]
+
+ # XXX: For NRCS, we're manually overriding units for now! Once
+ # unit conversion is supported for NRCS, REMOVE THIS!
+ if 'units' not in data:
+ data['units'] = 'imperial'
+
+ if data['provider'] == 'mesowest':
+ data['source'] = 'https://api.synopticdata.com/v2/stations/timeseries'
+ data['station_id'] = config['station']['station_id']
+ data['units'] = config['station']['units']
+
+ try:
+ desired_data = config['station']['desired_data']
+ except:
+ # desired_data malformed or missing, setting default
+ desired_data = 'air_temp,snow_depth'
+
+ # construct full API URL (sans start/end time, added later)
+ data['source'] = data['source'] + '?token=' + config['station']['token'] + '&within=60&units=' + data['units'] + '&stid=' + data['station_id'] + '&vars=' + desired_data
- try:
- desired_data = config['wxsite']['desired_data'].split(',')
- except:
- # desired_data malformed or missing, setting default
- desired_data = [
- 'TOBS', # AIR TEMPERATURE OBSERVED (degF)
- 'SNWD', # SNOW DEPTH (in)
- 'PREC' # PRECIPITATION ACCUMULATION (in)
- ]
except KeyError as e:
log.critical("%s not defined in %s" % (e, options.config))
exit(1)
fmap['hstAuto'] = 27 ; final_data[27] = None
fmap['hstAutoUnit'] = 28 ; final_data[28] = 'in'
-# one final mapping, the NRCS fields that this program supports to
+# one final mapping, the NRCS/MesoWest fields that this program supports to
# their InfoEx counterpart
iemap = {}
-iemap['PREC'] = 'precipitationGauge'
-iemap['TOBS'] = 'tempPres'
-iemap['SNWD'] = 'hS'
+
+if data['provider'] == 'nrcs':
+ iemap['PREC'] = 'precipitationGauge'
+ iemap['TOBS'] = 'tempPres'
+ iemap['SNWD'] = 'hS'
+ iemap['PRES'] = 'baro'
+ iemap['RHUM'] = 'rH'
+ iemap['WSPD'] = 'windSpeedNum'
+ iemap['WDIR'] = 'windDirectionNum'
+ # unsupported by NRCS:
+ # windGustSpeedNum
+elif data['provider'] == 'mesowest':
+ iemap['precip_accum'] = 'precipitationGauge'
+ iemap['air_temp'] = 'tempPres'
+ iemap['snow_depth'] = 'hS'
+ iemap['pressure'] = 'baro'
+ iemap['relative_humidity'] = 'rH'
+ iemap['wind_speed'] = 'windSpeedNum'
+ iemap['wind_direction'] = 'windDirectionNum'
+ iemap['wind_gust'] = 'windGustSpeedNum'
+
+# override units if user selected metric
+#
+# NOTE: to update this, use the fmap<->final_data mapping laid out above
+#
+# NOTE: this only 'works' with MesoWest for now, as the MesoWest API
+# itself handles the unit conversion; in the future, we will also
+# support NRCS unit conversion, but this must be done by this
+# program.
+if data['units'] == 'metric':
+ final_data[fmap['tempPresUnit']] = 'C'
+ final_data[fmap['hsUnit']] = 'm'
+ final_data[fmap['windSpeedUnit']] = 'm/s'
+ final_data[fmap['windGustSpeedNumUnit']] = 'm/s'
# floor time to nearest hour
dt = datetime.datetime.now()
microseconds=dt.microsecond)
begin_date = end_date - datetime.timedelta(hours=3)
-transport = zeep.transports.Transport(cache=zeep.cache.SqliteCache())
-client = zeep.Client(wsdl=wsdl, transport=transport)
-time_all_elements = time.time()
-
+# get the data
log.debug("Getting %s data from %s to %s" % (str(desired_data),
str(begin_date), str(end_date)))
-for elementCd in desired_data:
- time_element = time.time()
-
- # get the last three hours of data for this elementCd
- tmp = client.service.getHourlyData(
- stationTriplets=[station_triplet],
- elementCd=elementCd,
- ordinal=1,
- beginDate=begin_date,
- endDate=end_date)
-
- log.info("Time to get elementCd '%s': %.3f sec" % (elementCd,
- time.time() - time_element))
+time_all_elements = time.time()
- values = tmp[0]['values']
+# NRCS-specific code
+if data['provider'] == 'nrcs':
+ transport = zeep.transports.Transport(cache=zeep.cache.SqliteCache())
+ client = zeep.Client(wsdl=data['source'], transport=transport)
+
+ for elementCd in desired_data:
+ time_element = time.time()
+
+ # get the last three hours of data for this elementCd
+ tmp = client.service.getHourlyData(
+ stationTriplets=[data['station_id']],
+ elementCd=elementCd,
+ ordinal=1,
+ beginDate=begin_date,
+ endDate=end_date)
+
+ log.info("Time to get elementCd '%s': %.3f sec" % (elementCd,
+ time.time() - time_element))
+
+ values = tmp[0]['values']
+
+ # sort and isolate the most recent
+ #
+ # NOTE: we do this because sometimes there are gaps in hourly data
+ # in NRCS; yes, we may end up with slightly inaccurate data,
+ # so perhaps this decision will be re-evaluated in the future
+ if values:
+ ordered = sorted(values, key=lambda t: t['dateTime'], reverse=True)
+ infoex['wx_data'][elementCd] = ordered[0]['value']
+ else:
+ infoex['wx_data'][elementCd] = None
+
+# MesoWest-specific code
+elif data['provider'] == 'mesowest':
+ # massage begin/end date format
+ begin_date_str = begin_date.strftime('%Y%m%d%H%M')
+ end_date_str = end_date.strftime('%Y%m%d%H%M')
+
+ # construct final, completed API URL
+ api_req_url = data['source'] + '&start=' + begin_date_str + '&end=' + end_date_str
+ req = requests.get(api_req_url)
- # sort and isolate the most recent
- #
- # NOTE: we do this because sometimes there are gaps in hourly data
- # in NRCS; yes, we may end up with slightly inaccurate data,
- # so perhaps this decision will be re-evaluated in the future
- if values:
- ordered = sorted(values, key=lambda t: t['dateTime'], reverse=True)
- infoex['wx_data'][elementCd] = ordered[0]['value']
- else:
- infoex['wx_data'][elementCd] = None
+ try:
+ json = req.json()
+ except ValueError:
+ log.error("Bad JSON in MesoWest response")
+ sys.exit(1)
-log.info("Time to get all elementCds : %.3f sec" % (time.time() -
+ try:
+ observations = json['STATION'][0]['OBSERVATIONS']
+ except ValueError:
+ log.error("Bad JSON in MesoWest response")
+ sys.exit(1)
+
+ pos = len(observations['date_time']) - 1
+
+ for elementCd in desired_data.split(','):
+ # sort and isolate the most recent, see note above in NRCS for how and
+ # why this is done
+ #
+ # NOTE: Unlike in the NRCS case, the MesoWest API respones contains all
+ # data (whereas with NRCS, we have to make a separate request for
+ # each element we want. This is nice for network efficiency but
+ # it means we have to handle this part differently for each.
+ #
+ # NOTE: Also unlike NRCS, MesoWest provides more granular data; NRCS
+ # provides hourly data, but MesoWest can often provide data every
+ # 10 minutes -- though this provides more opportunity for
+ # irregularities
+
+ # we may not have the data at all
+ key_name = elementCd + '_set_1'
+ if key_name in observations:
+ if observations[key_name][pos]:
+ infoex['wx_data'][elementCd] = observations[key_name][pos]
+ else:
+ infoex['wx_data'][elementCd] = None
+ else:
+ infoex['wx_data'][elementCd] = None
+
+log.info("Time to get all data : %.3f sec" % (time.time() -
time_all_elements))
log.debug("infoex[wx_data]: %s", str(infoex['wx_data']))
wylark / infoex-autowx.git / commitdiff