By Kathleen Wong, UC Natural Reserve System
Climate is the hottest topic in California these days. As global warming heats up the West, it’s set off a cascade of effects ranging from toxic algal blooms, to tree die-offs, to wildfires of unprecedented size and ferocity. The situation has everyone asking the same question: what is it like outside?
That’s especially true for scientists studying climate change, ecosystems, and environmental health. Understanding how drought affects plants and animals, or the prevalence of disease-carrying ticks, or the water budget of ecosystems, requires data on temperature, precipitation, and more exotic factors such as soil moisture and solar radiation.
The UC Natural Reserve System has amassed a treasure trove of climate data to augment the research conducted on its protected landscapes and beyond. The NRS Climate Monitoring Network consists of 35 stations located at 30 NRS reserves from Mendocino to San Diego. Each station measures conditions such as temperature, precipitation, soil moisture, and solar radiation. Regularly checked for accuracy and monitored to ensure continuous operation, the network delivers high quality data from thousands of sensors suitable for supporting scientific research.
“We’re providing data from a diverse range of ecosystems, which is helpful for climate change research,” says Wendy Baxter, the lead technician responsible for maintaining the stations.
For those weather records to be useful, people need a way to access the data. Fortunately, viewing NRS climate data is a snap. That’s due to Dendra, a bespoke system designed to extract the records each user wants from a vast database, graph it out to expose any irregularities, and account for sensor data hiccups, all via an intuitive web interface.
“Many people assume that if you place a sensor out in the field, magic happens and data shows up on the web. Dendra is the service that does the hard work to make that actually happen,” says Dendra coinventor and Berkeley data scientist Collin Bode.
Together, the Climate Monitoring Network and Dendra serve up an unparalleled view of environmental conditions at locations up and down California.
Climate tools for research
The NRS’s integrated system of weather stations has humble roots. The first reserve weather stations were one-offs assembled by researchers who needed environmental data for their projects, or reserve directors hoping to aid visiting scientists.
A number of reserve directors soon realized that making their weather station instruments more uniform, and installing stations at more reserves, would make the data immensely more valuable. Mark Stromberg, then director of Hastings Natural History Reservation, and Eileen Lacey, Hastings’ faculty director, spearheaded a grant proposal to the National Science Foundation to establish just such a network. Funded in 2013, the grant enabled the NRS to dramatically expand its seven solid existing stations from seven to 28, standardize the sensor equipment, and make the data available online. The Western Regional Climate Center, a program of the National Oceanic and Atmospheric Information, and Nevada’s Desert Research Institute initially installed and maintained the NRS’s networked weather stations with guidance from NRS Information Manager Kevin Browne,.
The network has continued to thrive with maintenance funding from UC’s Office of the President. Contributions from individual reserves help establish additional stations. This year, Baxter is assuming the maintenance of a station at Sagehen Creek Field Station near Truckee. Plans are also in the works to adopt an abandoned station at Yosemite Field Station in Wawona, and establish two new stations at Oasis de los Osos (a Coachella Valley satellite of James San Jacinto Mountains Reserve), and Año Nuevo Island Reserve on the San Mateo Coast.
Keeping every weather station in the network operating over the long term is crucial, Baxter says. “The longer we’re able to collect weather data, the more valuable these datasets become,” Baxter says. The more extensive the record of measurements is, the easier it is to spot climate trends underway in a particular area.
This is why rescuing or adopting defunct stations is a priority for Baxter—they have added value. The fact that many older stations come with infrastructure such as towers is a bonus.
Fuel for climate science
Information from the NRS Climate Monitoring Network meets a multitude of needs. For example, reserve weather stations sometimes provide the only verified climate data within a large area. Their data can bridge major geographic gaps in weather information. Two of the more remote weather stations are located at White Mountain Research Center’s Barcroft Station, a dizzying 12,000 feet above sea level in eastern California, and at Sweeney Granite Mountains Desert Research Center in the eastern Mojave.
Data from some NRS weather stations also feed National Oceanic and Atmospheric Administration climate models, such as those used to generate federal drought index maps.
On a more prosaic level, climate stations help visitors to reserves understand what conditions to expect during their stay. There are even members of the public who use the data to inform recreational outings near reserves. “At Younger Lagoon, there’s somebody who’s really interested in the wind data for his water activities,” Baxter says.
Accuracy is job one
Keeping the stations humming requires a lot of time and travel. Baxter checks most days to see whether stations are still submitting data. If she can’t fix a problem remotely, she’ll often tap reserve managers to provide on-site assistance. “A lot of it is pinging people and saying hey, can you power cycle this? Or can you see why this is offline? Their troubleshooting can often save me a trip,” Baxter says.
Baxter and her colleague Jim Norris, who tends to work on southern California stations, also audit the performance of each station once every two years. This involves traveling to a reserve, setting up an audit weather station, and letting it run for 24 hours to ensure the local equipment produces similar readings. Any replaced equipment is operated for another full day to ensure it’s functioning properly.
The fact that Baxter is a rock climber and research tree climber (she is among the few people in the world who visits the top of old-growth coast redwoods and giant sequoias) comes in handy for her day job.
“Feeling comfortable at heights is definitely part of the job requirement,” Baxter says with characteristic understatement. “We remove the anemometer (the wind speed and direction sensor) for testing in the audits. You have to climb to the top of what is usually a 30-foot tower to do that.”
The audit cycle, plus any equipment failures, mean Baxter and Norris spend about a week or so every month staying at reserves. They typically replace up to $40,000 of sensors and equipment every year.
That figure climbed even higher in 2020, when wildfires swept through nine NRS reserves. The intense heat melted equipment at McLaughlin Natural and Quail Ridge reserves, which Baxter maintains, as well as a third station at Landels-Hill Big Creek Reserve.
Adventures in station maintenance
Natural disasters aren’t the only factors that can take down weather stations. Meddling animals post far more frequent problems.
Baxter recalls repeatedly having to reconnect station sensors to their transmitter boxes at Chickering American River Reserve after a bear took a liking to the taste of data cables. She eventually had run the wires through PVC pipe.
Hawks, too, have upended wind data by perching on station anemometers at multiple reserves. It took bird psychology insights from the instrument manufacturer to develop a solution. “That’s the highest point on a lot of the weather stations,” Baxter says. “Providing an alternative highest point for them to perch has seemed to work.”
More alarming are the paper wasps that have taken up residence in the weather station wiring enclosures at McLaughlin Reserve. Not even plugging gaps in the box exterior has been able to keep them out. Baxter, however, takes the presence of even these insects in stride. “Luckily they’re not super aggressive, so it’s not like they all come out and start stinging you like yellowjackets,” she says.
Making environmental data available
Gathering good environmental data is only half the battle. For the information to be useful, it must be reliable, visible, and easy to access. That feat is accomplished by a cyberinfrastructure service known as Dendra.
Dendra is a collection of databases and several dozen programs that work together to manage sensor feeds. It can ingest data beamed from equipment in the field (telemetry); offers methods to correct wonky data (curation); and can deliver the information to those who want to use it via an easy-to-use website (delivery). Dendra is the only service that integrates sensor data telemetry, curation, and distribution.
Dendra has proven so superior at that organizations outside the University have begun adopting the system to manage their own sensor networks. These include the California Department of Fish and Wildlife, Pepperwood Preserve, and even the University of New Hampshire, which uses the system to store and serve up data from a buoy in the Piscataqua Estuaries 50 miles north of Boston.
An embarrassment of sensor riches
Dendra was born out of data logger desperation. Bode was working for a UC Berkeley project called Keck Hydro Watch at the NRS’s Angelo Coast Range Reserve in Mendocino. To understand the fate of raindrops after they fall from the sky, the researchers installed a dizzying array of instruments along one hillslope.
“We ended up with about 1,000 instruments: 14 wells, sap flow sensors, tiny weather stations at the tops of trees—you name it, all within the same little watershed. And they’re sending in data every five to fifteen minutes,” Bode says. “We rapidly realized that manually downloading all of this data, and trying to manage the files was going to drive everybody crazy.”
The team tried an existing program developed for hydrology researchers. But it lacked the ability to correct data, a feature the project sorely needed.
Instead, Bode designed his own system with the help of UC Berkeley computer programmer Ginger Ogle. Housed on a single UC Berkeley server, this workhorse chugged along for eight years and racked up a billion or so records. By 2016, however, its performance had slowed to a crawl, and needed a technology upgrade to boot.
Crafting an all-in-one system
That’s when Bode ran into just the man for the job at Taekwondo practice in San Francisco. A full-stack developer, Smith suggested converting the sensor data program into a cloud-based system. Over the next few years, they created Dendra.
“At that point I had ten years of experience dealing with sensors on the ground, so I knew pretty clearly what researchers wanted, as opposed to what programmers wanted to build. Scott was willing to listen to my experience, and brought expertise in the architecture. It’s been an amazing collaboration,” Bode says.
The team received initial support from both the UC Natural Reserve System and the Angelo research effort, which had morphed into the Eel River Critical Zone Observatory. They also programmed in a means to cope with the problem of bad data.
Keeping data clean
Questionable data is inevitable among equipment placed in the wild. It’s only a matter of time before hailstorms, torrential rains, wildfires, or meddling animals take their toll. “When it’s machines vs. nature, nature wins,” Bode says.
When sensors go on the blink, Dendra provides ways to correct the data, note what took place, and let the user know what happened and why there are gaps in the data.
Their solution was to build in an annotation system. For example, once Baxter realized a bear had made a meal out of climate sensor wiring, she used the annotation feature to mark the data streams affected, labeled the event ‘bear attack on Chickering weather station,’ and indicated the dates the data would be invalid.
“I call annotations ‘field notes with teeth,’” Bode says.
Nondestructive adjustments
If someone requests annotated data from Dendra, the program will pull up the original records, filter out the bad data on the fly, and provide the adjusted data plus the annotation to the user. Dendra can even add calculations to fix sensor data that has drifted.
Importantly, Dendra never destroys original records. Any annotations and filtering occur on top of, but never overwrite, data. “If you make an annotation mistake, you can just repair it. You don’t have to drop an entire database of clean data, which is how we normally do things in data management,” Bode says.
Data delivery
Once data has been stored and cleaned, Dendra makes it easy for users to obtain and manipulate the information. The Dendra website allows lets users search for sensors via maps and lists, and specify the dates of the records they need. Dendra can also feed information directly to other weather systems, models, or groups.
By making it easy for users to obtain only the data they need, Dendra allows people to avoid time-consuming “data munching.” “If you process data manually, you run the risk of copying it into the wrong columns and then get the wrong analysis, or you have to spend hours writing code so you don’t make mistakes. One of the big focuses for us was trying to eliminate that part,” Bode says.
A peek under the hood
At present, Dendra takes in about 3,000 data streams and has grown to over 2 billion time series measurements. The fact that it is modular means it can be expanded indefinitely without much impact to performance.
A cloud service, Dendra is hosted on the Extreme Science and Engineering Discovery Environment (XSEDE), the supercomputing group of the National Science Foundation. The costs of computational services for Dendra can run to $20,000 per year. Fortunately, XSEDE provides annual resource grants that cover all of those expenses.
Getting better all the time
Bode and Smith launched Dendra in 2017. Since then, the site has already become a popular resource for sensor information. More than 2,500 unique users visited Dendra between September 2020 and September 2021, with nearly a quarter returning on a regular basis.
Yet Dendra’s developers are not ready to rest on their laurels. This September, they received a three-year, $827,000 NSF grant to enable the system to interface with the tools in EarthCube, the nation’s earth science cyberinformatics initiative. The work will allow scientists to connect to Dendra and pull out data in real time, skipping the need to download information; make Dendra searchable within other data repositories within the EarthCube family as well as within Google; and enable Dendra to host data from other EarthCub-compliant sensor systems.
Joining the EarthCube environment
“Almost all of the reserves have some US Geologic Survey sensors near them. Being able to pull that data in would be extremely valuable to any researchers interested in a particular reserve,” Bode says.
Likewise, other EarthCube information systems could add data streams currently seen only within Dendra. “If we can show up as one of their providers, other people who don’t even know we exist can use our data,” Bode adds.
Additional goals include creating a smartphone app for Dendra to enable Baxter and other scientists to manage sensor data while in the field, adding explainers and documentation for users, and even enabling people to connect their sensors via the Internet of Things.
“There’s many aspects of what we do on Dendra that can’t be found anywhere else. So I’m happy that people appreciate it and want to adopt it,” Bode says.