During this coronavirus lock down, IGIS has set out to revamp its data infrastructure to address our growing needs for big data storage and management moving forward. In particular, over the past few years we have accumulated over a dozen terabytes of drone data and associated mapping products, constituting tens of thousands of project files, and the quantity of this data is only expected to keep growing. Typically this drone data has been processed on a number of local desktop computers, and then backed up onto RAID hard drives or the cloud for cold storage; however, this is far from ideal in terms of consistent organization, versioning and ease of distribution.
As a solution to the problem, IGIS purchased a web server, equipped with multiple virtual machines (for processing, analysis and web services) along with a 30TB RAID data store/repository. The repository was networked to our various IGIS computers and RAID storage devices, so that all of our drone data could be transferred over to it. After much consideration, we settled on a standardized file structure, which could accommodate both datasets from past and future drone projects, with room for growth as needed. A python script was written to automatically generate this file structure, with some metadata inputs for each project, and our previous projects' data were then moved into their appropriate slots in the new structure, while jettisoning unwanted intermediary processing files; freeing up a ton of storage space. It would be correct to assume that this process of moving data was quite time consuming. However, moving forward, it will be easy to automatically set up our projects' file structures right from the inception of every new project, beginning with running the python script in ArcGIS Pro's Jupyter Notebook utility in the field, to eventually be delivered to the server repository down the pipeline, in a nicely organized package (similar to what we would provide to our non-IGIS project collaborators).
That alone is a big step in the right direction, but it gets better. Because all of this data is now in a standardized file structure, with standardized folder naming conventions, scripting our ArcGIS portal to automatically connect with the data via the imager server was only a small step away. With this complete, now any IGIS team member can access our entire post-processed, GIS-ready, drone data inventory of layers via ArcGIS Online or ArcGIS Pro.
Ultimately this has been a big leap forward, in terms of IGIS's informatics infrastructure; to compliment our significantly evolved pipeline for drone data collection and processing, depicted below.
Recently I was fortunate to work with the IGIS team in Santa Rosa and Sonoma to explore why so many homes and buildings were lost in the October Tubbs and Nuns Fires. With the IGIS's Shane Feirer we collected drone-based video to record how the fires burned through the vegetation near and around the lost structures.
We observed several sites where there was little fire activity in the forests or woodlands, yet the homes burned. This type of video helps us document how devastating a wind-driven ember fire can be and of the important lessons we can learn to be better prepared for wildfire.
From this experience I came away with a painful reminder that we all need to do a better job at focusing on fuels near our homes (e.g. combustible wood mulches used in landscaping, lawn furniture, leaf accumulations, dry landscape plants, etc.), especially in the 5 feet immediately adjacent to our homes. While the Tubbs Fire originated in grassy area in Calistoga it easily picked up embers from the burning vegetation which were moved by the 40-70 mph winds and created spot fires ahead of the flaming front. In short time these embers were blasted into homes via attic or soffit vents (critical to let moisture out of a building) or they ignited combustible materials close to buildings; these types of exposures are the primary way the Tubbs Fire started to consume homes. Eventually the Tubbs Fire moved to the more densely populated areas of the Fountain Grove subdivision in Santa Rosa and with each new home that was ignited a new source of embers were created. The embers that came from the burning buildings included 2 x 4s, chunks of wood the size of a frisbee, and other materials. These materials were blasted over Highway 101 on to homes and businesses in the urban center of Santa Rosa- a place most thought could not be impacted by wildfire. The winds persisted till mid-morning on October 9th providing considerable time for an ember to find a weakness in the home. All of us hope we never have a fire like this again, but as history shows us, California's most damaging fires typically occur in the September and October and are often wind-driven.
For many years UC has worked in educating homeowners about fire preparedness in the Wildland Urban Interface (WUI). These fires have resulted in the largest number of structure losses to date in California and we all need tools to better understand how to learn from these experiences. I greatly appreciate IGIS's willingness to help me collect some critical data in a time sensitive manner.
- Author: Sean Hogan
IGIS is thrilled to have a fantastic relationships with Parrot S.A. (one of the largest consumer drone corporations in the world), and Parrot's subsidiary company, Pix4D (the leading developer of drone data processing applications in the world), who together have been have been extremely generous and supportive of the IGIS drone program, and the UC System at large.
Please check out Parrot's Educational Blog, which recently featured an outstanding short article on IGIS's research into rangeland applications of drone technology, including the use of Parrot's Sequoia multispectral camera, by clicking here:
Also - Please stay tuned for the upcoming February 2017 issue of California Agriculture, which features an article on unmanned aerial systems (UAS) for agriculture and natural resources (co-authored by myself, Dr. Maggi Kelly, Brandon Stark, and Dr. YangQuan Chen), which will also be accompanied by an editorial written by Dr. Kelly on the need for a coordinated approach to scale up drone research and teaching across the UC system.
- Author: Robert Johnson
I recently attended the Pacific Research Platform Science Engagement Workshop hosted by CITRIS and Calit2 at UC Merced. The PRP is a "data freeway" currently being developed by researchers at UC San Diego and UC Berkeley to connect the major research universities and other research institutions on the west coast with the goal of sharing large datasets and computational resources at speeds of 10-100 Gbps.
The focus of the workshop were a number of digital archaeology projects that were either already using the PRP architecture or had the potential to do so. It was quite interesting to see the efforts being undertaken to preserve at-risk heritage sites using drone imagery, 3D scans of artifacts and 360° virtual reality imagery. All of these techniques produce massive datasets (often several terabytes) which require extensive post-processing and therefore are exactly the type of projects that the PRP was designed for.
After the presentations, we headed over to UC Merced's new WAVE (Wide-Area Visualization Environment) facility to check out some of the virtual reality imagery that had been discussed. The WAVE consists of twenty 4k 3D monitors arranged in a parabolic curve to create an immersive VR environment. We saw several incredibly detailed image sets of archaeological sites in Greece, Egypt, Saudi Arabia and Belize that had been processed and were being served through the PRP architecture. The effect was quite impressive, it really was the next best thing to being at the actual site.
All in all, this was a real eye-opening workshop that gave a compelling picture of the future of sharing and visualization of Big Data.
- Author: Sean Hogan
Today marks the end of this year's California Naturalist Conference (http://calnat.ucanr.edu/2016conference/), where IGIS was once again honored to be a contributor at the fantastic gathering. Highlights from the event include:
Working with Greg Ira of the California Naturalist Program, I led a Smartphone GPS and Mapping Skills workshop, which provided an introduction to free and inexpensive apps for collecting map data in the field and creating custom online maps out of the data that you collect. Meanwhile, our friend Ken-ichi Ueda led a similarly themed workshop that covered the outstanding mobile and online app that he developed, iNaturalist (http://www.inaturalist.org/); as a tool for furthering citizen science efforts. Citizen Science was a strong theme in this year's conference and we were pleased to see how much interest participants expressed in using spatial technologies to facilitate greater public engagement in science. Further reinforcing this theme, our UC Cooperative Extension Specialist colleague, Roger Baldwin, had a table at the conference displaying the Wild Pig Damage App that IGIS just built and published to the Google Play Store (https://play.google.com/store/apps/details?id=com.ucanr.wildpigdamage). This app was created through a collaborative effort between UC Cooperative Extension and IGIS to facilitate public collection of spatial records and pictures pertaining to wild pig damage on range, forest and agricultural lands throughout the state of California. On the final night of the conference our IGIS Academic Coordinator, Andy Lyons, presented a poster, which showcased the variety of activities and services that IGIS is currently engaged in, including mobile data collection, web mapping, geospatial analytics, training, and drone operations. Complementing Andy's poster session, I set up an adjacent table display with an assorted of IGIS's new drone equipment. This attracted a lot of attention to the scientific interests of IGIS and prompted many discussions of future applications for drone technology in relation to ecological monitoring and restoration.
I want to thank the leadership of UC ANR's California Naturalist Statewide Program (http://calnat.ucanr.edu/About_the_program/) for inviting IGIS to this year's conference, as well as UC ANR's Program Support Unit for going above and beyond to make it such a great event. The community of people who are involved with the California Naturalist Program are a truly inspiring group of people, and I look forward to our continued cooperation in the future.