UnicodeDecodeError: 'ascii' codec can't decode byte 0x85 in position 297464: ordinal not in range(128) - METAR Plotting

[edrewitz]

Hey everyone!

I am trying to plot metar data from https://thredds-test.unidata.ucar.edu/thredds/catalog/noaaport/text/metar/catalog.xml and sometimes it works while other times it doesn't. Lately my code has been crashing more than normally and I am trying to figure out a way to fix this. From what I gather, if I am correct it seems like there are sometimes non-ascii characters making their way into those metar files and messing everything up. Please see the screenshot of the code and the error message associated with block 11. Thank you for all of your help!!

Regards,

Eric Drewitz

[edrewitz]

One thing I will add in here is through some trial and error, when I changed 'ascii' to 'latin-1', I am not getting these errors anymore. I'm still new to dealing with bytes so I just want to double check with you that this is still a good format for METAR data. I think it should be as I read that English has complete coverage under latin-1 and when I compare my plot it looks similar to the observations I see on the NWS obs page. Thanks!

[dopplershift]

A few notes:

1. Yes, the latin-1 encoding should be fine, the only difference from ascii will be for characters >127, which shouldn't really be present. For at least one case I just found in the data, though, I think utf-8 seems to produce slightly better results. I'm not really sure what's going on, other than some random junk is creeping into the data flow.
2. These data files are now available on https://thredds.ucar.edu (our production server), so I'd recommend using that rather than our test server
3. There's actually a direct way with remote_open() and Siphon to get what you want without manually using StringIO.

from datetime import datetime, time

from siphon.catalog import TDSCatalog

dt1 = datetime.utcnow()
metar_cat = TDSCatalog('https://thredds.ucar.edu/thredds/catalog/noaaport/text/metar/catalog.xml')
metar_file = metar_cat.datasets.filter_time_nearest(dt1).remote_open(mode='t', encoding='utf-8')

Also, I'll note for other readers that nothing here actually pertains specifically to MetPy's METAR reader, this is all interacting with Siphon to access data from THREDDS.