From: Alexander Vasarab Date: Wed, 1 Jul 2020 01:43:09 +0000 (-0700) Subject: Merge branch 'mesowest' X-Git-Tag: v2.0.0^0 X-Git-Url: https://wylark.com/src/infoex-autowx.git/commitdiff_plain/e7abb7e4b661ecbf466c1934ce3e34e1a9ea1a68?hp=a585efb4e491a6a2dab7df5e858b34fe5235f662 Merge branch 'mesowest' --- diff --git a/.gitignore b/.gitignore index 0a764a4..520dcbc 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,4 @@ env +.CSV +scratch/ +configs/ diff --git a/README.md b/README.md index 4433828..edd134b 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,18 @@ InfoEx AutoWx (IEAW) ============= -This program fetches data from an NRCS SNOTEL site and pushes it into -the InfoEx system using the new automated weather system implementation. +This program fetches data from an NRCS SNOTEL or MesoWest station and +pushes it into the InfoEx system using the new automated weather system +implementation. -License under the MIT license (see file: LICENSE). +Licensed under the ISC license (see file: LICENSE). Disclaimer ---------- -Your usage of the NRCS and InfoEx systems is bound by their respective -terms and this software makes no attempt or claim to abide by any such -terms. +Your usage of the NRCS, MesoWest, and/or InfoEx systems is bound by +their respective terms and this software makes no attempt or claim to +abide by any such terms. Installation ------------ @@ -31,16 +32,29 @@ This program is designed to be run from the command line (or via cron(8)) and administered via a simple, concise configuration file. This design allows you to run a separate program instance for each NRCS -weather station from which you'd like to automate the importation of -data into your InfoEx subscriber account. +or MesoWest weather station from which you'd like to automate the +importation of data into your InfoEx subscriber account. + +To get started, copy the included example config file +(`config.ini.example` in the root source directoy) and modify the values +for your own use. To run ad-hoc (be sure to activate the virtual environment, as detailed in the Installation section): -`./infoex-autowx.py --config [path/to/config-file.ini] [--dry-run]` +`./infoex-autowx.py --config path/to/config-file.ini [--dry-run] [--log-level debug|info|warning]` + +**NOTE: Specifying --dry-run will not clean up the generated CSV file.** +This is so that you can more easily debug any issues that arise in the +setup process. -**NOTE: Specifying --dry-run will also not clean up the generated CSV -file.** This is so that you can debug any issues more easily. +You can also specify `--log-level` as debug, info, warning. The +log messages produced by the program will try to be logged to journald, +but if that's not available, they will be printed to stdout. This output +can be helpful early on in the setup process. + +Automation +---------- Here's an example of a crontab(5) with two SNOTEL sites, each of which will run once per hour (note that this will activate the virtual environment @@ -52,32 +66,38 @@ created earlier): Configuration File ------------------ -The configuration file is separated into two parts, the [wxsite] -portion, and the [ftp] portion. +The configuration file is separated into two parts, the [station] +portion, and the [infoex] portion. -The [wxsite] values describe which NRCS SNOTEL site's data you're after. +The [station] values describe which weather station's data you're after. See the next section in this README for instructions on obtaining these values. -The [ftp] values describe your credentials for the InfoEx automated -weather station FTP server. +The [infoex] values describe your credentials for the InfoEx automated +weather station FTP server and other InfoEx-related configuration +options. -`[wxsite]` -`station_triplet = [The NRCS identifier for a particular SNOTEL site]` -`desired_data = [A comma-delimited list of NRCS elements you're interested in]` -`location_uuid = [The UUID used by InfoEx to identify your automated Wx site]` -`csv_filename = [Arbitrary name of the file that will be uploaded to InfoEx]` +`[station]` +`type = # either mesowest or nrcs #` +`token = # MesoWest API token -- ignored when type is nrcs #` +`station_id = # the NRCS/MesoWest identifier for a particular station #` +`desired_data = # a comma-delimited list of fields you're interested in #` +`units = # either english or metric -- ignored when type is nrcs #` -`[ftp]` -`host = [InfoEx FTP host address]` -`uuid = [InfoEx-supplied UUID]` -`api_key = [InfoEx-supplied API Key]` +`[infoex]` +`host = # InfoEx FTP host address #` +`uuid = # InfoEx-supplied UUID #` +`api_key = # InfoEx-supplied API Key #` +`csv_filename = # arbitrary name of the file that will be uploaded to InfoEx #` +`location_uuid = # the UUID used by InfoEx to identify your automated Wx site #` -Finding Your WXSITE values --------------------------- +Finding your NRCS `station` values +---------------------------------- + +To complete the [station] configuration section for an NRCS station, you +must fill in the attributes of the NRCS SNOTEL site from which you want +to import data. -To complete the [wxsite] configuration section, you must fill in the -attributes of the NRCS SNOTEL site from which you want to import data. Here are the steps to do that: 1. Find your station by clicking through this website: @@ -92,7 +112,7 @@ Here are the steps to do that: 4-digit number). 3. Combine your Station ID, state abbreviation, and the network type - "SNTL" to get your station triplet (`station_triplet`, in the + "SNTL" to get your NRCS station triplet (`station_id`, in the configuration file). For example: 655:OR:SNTL @@ -101,9 +121,9 @@ Here are the steps to do that: of Oregon (OR). SNTL just represents that the station is in the SNOTEL network and is used internally by NRCS. -Once you have your station triplet, fill in the field in your -configuration file. Now you must select which data you'd like to pull -from NRCS to push into InfoEx. +Once you have your station ID, fill in the field in your configuration +file. Now you must select which data you'd like to pull from NRCS to +push into InfoEx. For that, visit the NRCS web service: @@ -117,18 +137,118 @@ Once you've chosen your elements, combine all of their respective "elementCd" values into a comma-delimited string and put that into your configuration file as the `desired_data` value. -For example: +A complete [station] section example: -`station_triplet = 655:OR:SNTL` +`[station]` +`type = nrcs` +`station_id = 655:OR:SNTL` `desired_data = TOBS,PREC` indicates that I'd like to import "AIR TEMPERATURE OBSERVED" and "PRECIPITATION ACCUMULATION" from the NRCS SNOTEL site at Mud Ridge, OR, into InfoEx. -Version History +Finding your MesoWest `station` values +-------------------------------------- + +MesoWest has great documentation which can be found here: + +https://developers.synopticdata.com/mesonet/v2/getting-started/ + +To complete the [station] configuration section for a MesoWest station, +you must fill in the attributes of the MesoWest station ID from which +you want to import data. Here are the steps to do that: + +1. Firstly, get set up with MesoWest's API by going to the above + 'Getting Started' link. Once you're set up, you can copy a token from + the MesoWest web portal into your configuration file's `token` value. + +2. Next, you will want to find the Station ID for the MesoWest weather + station of interest and copy it to the `station_id` value. + +3. Finally, you must choose what data types you want to push into + InfoEx and compile them into a comma-separated list. MesoWest refers + to these as 'field names' or 'station variables' and a list is + available here: + +https://developers.synopticdata.com/about/station-variables/ + +The MesoWest API supports on-the-fly unit conversion. If desired, that +can be specified to infoex-autowx via the configuration option `units`. +This can be either 'english' or 'metric', with 'english' meaning +imperial units as used in the United States. + +A complete [station] section example: + +`[station]` +`type = mesowest` +`token = # token id copied from MesoWest web account #` +`station_id = OD110` +`desired_data = air_temp,snow_depth` +`units = english` + +indicates that I'd like to import "Temperature" and "Precipitation +accumulated" from the MesoWest station at Santiam Pass, OR, into InfoEx, +and that I want that data in imperial units. + +A note on supported measurements +-------------------------------- + +While this program supports several measurements, and will faithfully +request all of the ones you specify (provided they're supported), the +weather station may not record them. In this case, the data will simply +be ignored (i.e. it will NOT log "0" when there's no measurement +available). + + +InfoEx provides a mechanism for inspecting your automated weather +station data, so use that after setting this program up and compare it +with the data you see in your web browser. + +Here's the list of measurements currently supported: + +**NRCS:** +PREC +TOBS +SNWD +PRES +RHUM +WSPD +WDIR + +**MesoWest:** +precip\_accum +air\_temp +snow\_depth +pressure +relative\_humidity +wind\_speed +wind\_direction +wind\_gust + +Future plans +------------ + +- Improve the documentation +- Implement unit conversion for NRCS stations + +Version history --------------- +- 2.0.0 (Jul 2020) + + Implement MesoWest integration. + + This release also makes significant changes to the configuration file, + hence the major version bump. Such changes are not taken lightly but + given the desire to support multiple data sources, were necessary. + + Other minor changes include: + + - New switches: --log-level and --version. + - Better documentation. + - Expanded supported measurement types (from three to eight, in number). + - 1.0.0 (Jun 2020) First released version. Cleaned up the program and design. diff --git a/config.ini.example b/config.ini.example index 018a6fc..17544ae 100644 --- a/config.ini.example +++ b/config.ini.example @@ -1,10 +1,14 @@ -[wxsite] -station_triplet = -desired_data = -location_uuid = -csv_filename = +[station] +type = # (mesowest|nrcs) # +token = # MesoWest API token (ignored for NRCS) # +station_id = # NRCS/MesoWest Station ID # +desired_data = # Comma-separated list of values # +units = # (english|metric) (ignored for NRCS) # + +[infoex] +host = # InfoEx FTP host address # +uuid = # InfoEx-supplied UUID # +api_key = # InfoEx-supplied API Key # +csv_filename = # Name of file to upload to InfoEx FTP # +location_uuid = # InfoEx-supplied Location UUID # -[ftp] -host = -uuid = -api_key = diff --git a/infoex-autowx.py b/infoex-autowx.py index ea91bef..e4befad 100755 --- a/infoex-autowx.py +++ b/infoex-autowx.py @@ -2,14 +2,15 @@ # -*- coding: utf-8 -*- """ -InfoEx <-> NRCS Auto Wx implementation +InfoEx <-> NRCS/MesoWest Auto Wx implementation Alexander Vasarab Wylark Mountaineering LLC -Version 1.0.0 +Version 2.0.0 -This program fetches data from an NRCS SNOTEL site and pushes it to -InfoEx using the new automated weather system implementation. +This program fetches data from either an NRCS SNOTEL site or MesoWest +weather station and pushes it to InfoEx using the new automated weather +system implementation. It is designed to be run hourly, and it asks for the last three hours of data of each desired type, and selects the most recent one. This @@ -35,26 +36,40 @@ from collections import OrderedDict from ftplib import FTP from optparse import OptionParser +import requests + import zeep import zeep.cache import zeep.transports +__version__ = '2.0.0' + log = logging.getLogger(__name__) -log.setLevel(logging.DEBUG) +log.setLevel(logging.NOTSET) try: from systemd.journal import JournalHandler log.addHandler(JournalHandler()) except: - # fallback to syslog - import logging.handlers - log.addHandler(logging.handlers.SysLogHandler()) + ## fallback to syslog + #import logging.handlers + #log.addHandler(logging.handlers.SysLogHandler()) + # fallback to stdout + handler = logging.StreamHandler(sys.stdout) + log.addHandler(handler) + +parser = OptionParser(version=__version__) -parser = OptionParser() parser.add_option("--config", dest="config", metavar="FILE", help="location of config file") + +parser.add_option("--log-level", + dest="log_level", + default=None, + help="set the log level (debug, info, warning)") + parser.add_option("--dry-run", action="store_true", dest="dry_run", @@ -66,36 +81,79 @@ parser.add_option("--dry-run", config = configparser.ConfigParser(allow_no_value=False) if not options.config: - print("Please specify a configuration file via --config") + parser.print_help() + print("\nPlease specify a configuration file via --config.") sys.exit(1) config.read(options.config) -log.debug('STARTING UP') +# ugly, but passable +if options.log_level in [None, 'debug', 'info', 'warning']: + if options.log_level == 'debug': + log.setLevel(logging.DEBUG) + elif options.log_level == 'info': + log.setLevel(logging.INFO) + elif options.log_level == 'warning': + log.setLevel(logging.WARNING) + else: + log.setLevel(logging.NOTSET) +else: + parser.print_help() + print("\nPlease select an appropriate log level or remove the switch (--log-level).") + sys.exit(1) -wsdl = 'https://www.wcc.nrcs.usda.gov/awdbWebService/services?WSDL' +log.debug('STARTING UP') try: infoex = { - 'host': config['ftp']['host'], - 'uuid': config['ftp']['uuid'], - 'api_key': config['ftp']['api_key'], - 'location_uuid': config['wxsite']['location_uuid'], + 'host': config['infoex']['host'], + 'uuid': config['infoex']['uuid'], + 'api_key': config['infoex']['api_key'], + 'csv_filename': config['infoex']['csv_filename'], + 'location_uuid': config['infoex']['location_uuid'], 'wx_data': {}, # placeholder key, values to come later - 'csv_filename': config['wxsite']['csv_filename'] } - station_triplet = config['wxsite']['station_triplet'] + data = dict() + data['provider'] = config['station']['type'] + + if data['provider'] not in ['nrcs', 'mesowest']: + print("Please specify either nrcs or mesowest as the station type.") + sys.exit(1) + + if data['provider'] == 'nrcs': + data['source'] = 'https://www.wcc.nrcs.usda.gov/awdbWebService/services?WSDL' + data['station_id'] = config['station']['station_id'] + + try: + desired_data = config['station']['desired_data'].split(',') + except: + # desired_data malformed or missing, setting default + desired_data = [ + 'TOBS', # AIR TEMPERATURE OBSERVED (degF) + 'SNWD', # SNOW DEPTH (in) + 'PREC' # PRECIPITATION ACCUMULATION (in) + ] + + # XXX: For NRCS, we're manually overriding units for now! Once + # unit conversion is supported for NRCS, REMOVE THIS! + if 'units' not in data: + data['units'] = 'imperial' + + if data['provider'] == 'mesowest': + data['source'] = 'https://api.synopticdata.com/v2/stations/timeseries' + data['station_id'] = config['station']['station_id'] + data['units'] = config['station']['units'] + + try: + desired_data = config['station']['desired_data'] + except: + # desired_data malformed or missing, setting default + desired_data = 'air_temp,snow_depth' + + # construct full API URL (sans start/end time, added later) + data['source'] = data['source'] + '?token=' + config['station']['token'] + '&within=60&units=' + data['units'] + '&stid=' + data['station_id'] + '&vars=' + desired_data - try: - desired_data = config['wxsite']['desired_data'].split(',') - except: - # desired_data malformed or missing, setting default - desired_data = [ - 'TOBS', # AIR TEMPERATURE OBSERVED (degF) - 'SNWD', # SNOW DEPTH (in) - 'PREC' # PRECIPITATION ACCUMULATION (in) - ] except KeyError as e: log.critical("%s not defined in %s" % (e, options.config)) exit(1) @@ -154,12 +212,43 @@ fmap['hn24AutoUnit'] = 26 ; final_data[26] = 'in' fmap['hstAuto'] = 27 ; final_data[27] = None fmap['hstAutoUnit'] = 28 ; final_data[28] = 'in' -# one final mapping, the NRCS fields that this program supports to +# one final mapping, the NRCS/MesoWest fields that this program supports to # their InfoEx counterpart iemap = {} -iemap['PREC'] = 'precipitationGauge' -iemap['TOBS'] = 'tempPres' -iemap['SNWD'] = 'hS' + +if data['provider'] == 'nrcs': + iemap['PREC'] = 'precipitationGauge' + iemap['TOBS'] = 'tempPres' + iemap['SNWD'] = 'hS' + iemap['PRES'] = 'baro' + iemap['RHUM'] = 'rH' + iemap['WSPD'] = 'windSpeedNum' + iemap['WDIR'] = 'windDirectionNum' + # unsupported by NRCS: + # windGustSpeedNum +elif data['provider'] == 'mesowest': + iemap['precip_accum'] = 'precipitationGauge' + iemap['air_temp'] = 'tempPres' + iemap['snow_depth'] = 'hS' + iemap['pressure'] = 'baro' + iemap['relative_humidity'] = 'rH' + iemap['wind_speed'] = 'windSpeedNum' + iemap['wind_direction'] = 'windDirectionNum' + iemap['wind_gust'] = 'windGustSpeedNum' + +# override units if user selected metric +# +# NOTE: to update this, use the fmap<->final_data mapping laid out above +# +# NOTE: this only 'works' with MesoWest for now, as the MesoWest API +# itself handles the unit conversion; in the future, we will also +# support NRCS unit conversion, but this must be done by this +# program. +if data['units'] == 'metric': + final_data[fmap['tempPresUnit']] = 'C' + final_data[fmap['hsUnit']] = 'm' + final_data[fmap['windSpeedUnit']] = 'm/s' + final_data[fmap['windGustSpeedNumUnit']] = 'm/s' # floor time to nearest hour dt = datetime.datetime.now() @@ -168,41 +257,93 @@ end_date = dt - datetime.timedelta(minutes=dt.minute % 60, microseconds=dt.microsecond) begin_date = end_date - datetime.timedelta(hours=3) -transport = zeep.transports.Transport(cache=zeep.cache.SqliteCache()) -client = zeep.Client(wsdl=wsdl, transport=transport) -time_all_elements = time.time() - +# get the data log.debug("Getting %s data from %s to %s" % (str(desired_data), str(begin_date), str(end_date))) -for elementCd in desired_data: - time_element = time.time() - - # get the last three hours of data for this elementCd - tmp = client.service.getHourlyData( - stationTriplets=[station_triplet], - elementCd=elementCd, - ordinal=1, - beginDate=begin_date, - endDate=end_date) - - log.info("Time to get elementCd '%s': %.3f sec" % (elementCd, - time.time() - time_element)) +time_all_elements = time.time() - values = tmp[0]['values'] +# NRCS-specific code +if data['provider'] == 'nrcs': + transport = zeep.transports.Transport(cache=zeep.cache.SqliteCache()) + client = zeep.Client(wsdl=data['source'], transport=transport) + + for elementCd in desired_data: + time_element = time.time() + + # get the last three hours of data for this elementCd + tmp = client.service.getHourlyData( + stationTriplets=[data['station_id']], + elementCd=elementCd, + ordinal=1, + beginDate=begin_date, + endDate=end_date) + + log.info("Time to get elementCd '%s': %.3f sec" % (elementCd, + time.time() - time_element)) + + values = tmp[0]['values'] + + # sort and isolate the most recent + # + # NOTE: we do this because sometimes there are gaps in hourly data + # in NRCS; yes, we may end up with slightly inaccurate data, + # so perhaps this decision will be re-evaluated in the future + if values: + ordered = sorted(values, key=lambda t: t['dateTime'], reverse=True) + infoex['wx_data'][elementCd] = ordered[0]['value'] + else: + infoex['wx_data'][elementCd] = None + +# MesoWest-specific code +elif data['provider'] == 'mesowest': + # massage begin/end date format + begin_date_str = begin_date.strftime('%Y%m%d%H%M') + end_date_str = end_date.strftime('%Y%m%d%H%M') + + # construct final, completed API URL + api_req_url = data['source'] + '&start=' + begin_date_str + '&end=' + end_date_str + req = requests.get(api_req_url) - # sort and isolate the most recent - # - # NOTE: we do this because sometimes there are gaps in hourly data - # in NRCS; yes, we may end up with slightly inaccurate data, - # so perhaps this decision will be re-evaluated in the future - if values: - ordered = sorted(values, key=lambda t: t['dateTime'], reverse=True) - infoex['wx_data'][elementCd] = ordered[0]['value'] - else: - infoex['wx_data'][elementCd] = None + try: + json = req.json() + except ValueError: + log.error("Bad JSON in MesoWest response") + sys.exit(1) -log.info("Time to get all elementCds : %.3f sec" % (time.time() - + try: + observations = json['STATION'][0]['OBSERVATIONS'] + except ValueError: + log.error("Bad JSON in MesoWest response") + sys.exit(1) + + pos = len(observations['date_time']) - 1 + + for elementCd in desired_data.split(','): + # sort and isolate the most recent, see note above in NRCS for how and + # why this is done + # + # NOTE: Unlike in the NRCS case, the MesoWest API respones contains all + # data (whereas with NRCS, we have to make a separate request for + # each element we want. This is nice for network efficiency but + # it means we have to handle this part differently for each. + # + # NOTE: Also unlike NRCS, MesoWest provides more granular data; NRCS + # provides hourly data, but MesoWest can often provide data every + # 10 minutes -- though this provides more opportunity for + # irregularities + + # we may not have the data at all + key_name = elementCd + '_set_1' + if key_name in observations: + if observations[key_name][pos]: + infoex['wx_data'][elementCd] = observations[key_name][pos] + else: + infoex['wx_data'][elementCd] = None + else: + infoex['wx_data'][elementCd] = None + +log.info("Time to get all data : %.3f sec" % (time.time() - time_all_elements)) log.debug("infoex[wx_data]: %s", str(infoex['wx_data'])) diff --git a/requirements.txt b/requirements.txt index 739dc36..2aec4ac 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,2 @@ +requests>=2.0.0 zeep>=3.4.0