- Author: Andy Lyons
The Magic Behind Cal-Adapt: The API
If you're reading this you've probably heard of Cal-Adapt, California's official source for downscaled climate data from the 4th Climate Change Assessment. Most people have engaged with climate data on Cal-Adapt through the website, which provides over a dozen well-designed interactive tools that allow you to create charts and maps of wide range of climate variables, from temperature to snow pack to wildfire risk, for your location of interest. For the intrepid user, the website also has a download tool where you can download the data as CSV or raster files, or the source files in NetCDF format.
Cal-Adapt provides access to a wealth of downscaled climate data for
California, Nevada, and parts of neighboring states
Sample chart from Cal-Adapt.org
A lesser known fact is that all this magic is made possible under the hood by a well-designed open API. An API, or Application Programming Interface, is essentially a common language that allows computer programs to talk to each other. Using the API, the Cal-Adapt website can retrieve data from the Cal-Adapt server, on demand, to make all those beautiful charts and maps.
But guess what? This same API, which is fully open and documented on the Cal-Adapt website, can be used by things other than websites. Any programming language that supports standard web protocols can send specially formatted commands through the API to get climate data directly from Cal-Adapt, without ever having to go through the website. This includes R and Python, the two most popular programming languages for data science, with users in the hundreds of thousands.
Introducing caladaptR
Why is this a big deal, you might ask? We can start with the huge number of R users, particularly in research and academia. The science of climate change impacts and adaptation planning are still young and rapidly evolving, and R is by far the favorite computing environment for researchers. Before caladaptR, a data scientist would need to go through the Cal-Adapt website and download the raw data as rasters or CSV files, one variable at a time, and one zip file at a time. She would then have to import those files into R using the file names as a reminder of which data it contains. This process works, and many people have done, but it's time consuming and klunky. caladaptR replaces all of those manual steps with a few lines of code that can be easily tweaked for new locations, climate variables, and models.
For civilians out there, this is still a big deal because it means adaptation planners and researchers can not only work more efficiently, they can also package up their analyses in user-friendly output formats that R is really good at. This includes markdown reports, as well as tailored decision support tools.
One of R's most popular features is R-Shiny, a package which allows R users to bundle their analyses up into user friendly interactive web sites, or apps. These web apps aren't limited to show pre-canned results, they can actually fetch data "on the fly" and run analyses based on user input. caladaptR is a "Shiny ready" package that includes a widget to show your data download status. A couple of sample Shiny apps are available on the website, including one that computes the projected amount of winter chill in the coming decades, which can help tree crop growers decide which variety of tree nuts to plant, or whether to get out of tree nuts all together.
Not surprisingly, we've recently seen growing interest in climate change impacts, and a growing recognition that data from climate models need to be transformed into measures that make sense to regular folks. Data science platforms like R, that can tap into cloud data and services via APIs, are the kinds of tools that will build the next generation of climate science translation services. With R and caladaptR, analyzing climate impacts and developing decision support tools is well within reach of anyone who has intermediate R skills.
Learning More
For more details, check out the following overview videos. When you're ready to start writing some code, head over to the caladaptR website which has installation instructions, getting started code, sample notebooks, workshop recordings, and long-form articles on some of the more advanced features. You can also attend an upcoming workshop (the next caladaptR workshop will be February 2022) and sign-up for IGIS Office Hours if you have any questions.
/h2>/h2>/h2>
- Author: Sean Hogan
- Author: Brandon Stark
- Author: Maggi Kelly
non-recreational use of small unmanned aerial systems (sUAS - aka. drones) was added to the FAA's Code of Federal Regulations (CFR) in mid-2016, there have been a number of updates to the rules over the years, including an Academic Exemption for education and research purposes (which is only apply to institutions of higher education and not other state agencies or conservation groups). The most recent of these updates occurred in January 2021, when the regulations were further amended to allow for the operation of drones during the night and over people in limited circumstances. For some operators, these have been very welcome changes. However, maybe more widely relevant to our community of drone enthusiasts, in June 2021 the FAA announced the requirement of "The Recreational UAS Safety Test" (TRUST) for all recreational pilots; with the objective of ensuring that these pilots are likewise knowledgeable in the FAA's rules and regulations for drone operations.
Amidst these updates and changes to the regulations, a common question that IGIS receives is "what do I need to do to legally fly a drone?" To answer this question, the head of our UC System's Center of Excellence on Unmanned Aircraft System Safety (UASSafety), Dr. Brandon Stark, has provided the following simplified guidelines to help steer people in the right directions, for whether they need the TRUST and/or Part 107 certifications:
- Coursework -> TRUST
- Research in access-controlled field -> TRUST
- Research in public area with people -> Part 107
- Collection of data for another agency -> Part 107
- Promotional media -> Part 107
- Demonstration for students -> TRUST
- Demonstration for non-students -> Part 107
- Inspecting any structure for repairs -> Part 107
- Flying above the FAA's Facility Map Altitude -> Part 107
- Flying above 400 ft (AGL) -> Part 107
- Any operation that could profit the pilot in any way -> Part 107
- Strictly for fun (even if you have a Part 107 certification) -> TRUST
Note - The Academic Exception is not a loophole. Everyone now needs to have either completed a TRUST or Part 107 certification, and preferably both.
For the sake of drone operators everywhere, it is extremely important that we all abide by these rules and regulations; as it could take only one grievous mistake (even if the mistake is unwitting) to legally set back all of the progress that has been made for the allowance of sUAS operations, both within and outside of the UC System.
For more information on the TRUST and Part 107 certifications, please refer to the FAA's official site: https://faadronezone.faa.gov/
For all UC drone operations, please do not forget to file your flights in advance through the UC's UAS Safety App, https://ehs.ucop.edu/drones/. This is not only required by the UC System, but also provides the valuable benefit of insurance coverage for you and your drone in the event of an accident. Additionally, the recently created UC Drones knowledge portal is an incredible resource for a wide range of drone safety and regulation information: https://ucdrones.github.io/
- Author: Sean Hogan
- Author:
Ben Satzman
- Author: Maggi Kelly
July 2021 marks the fifth anniversary of Drone Camp, and thanks to an all-star lineup of presenters and instructors and a fantastic and diverse group of over 255 attendees from all over the world, it was a massive success this year. Initially launched in 2016 by the UC ANR, IGIS Statewide program, DroneCamp has now evolved into a multi-campus and industry collaboration, with a network of drone experts hailing from UC ANR, UC Santa Cruz, UC Berkeley, UC Merced, UC Davis, CSU Monterey Bay, and the Monterey Bay Drone Automation and Robotics Technology corporation. From 2016 to 2019, this event was held in-person at Davis, San Diego, and then in Monterey, CA.; however, in 2020 it was moved online due to COVID-19. The move indoors, to discuss a very outdoor-oriented topic, came with some challenges. We had to shelve our important (and fun) hands-on equipment and flight training, for example. Yet it had rewards. We were able to reach a more diverse and a greater number of participants, and widen the scope of content. Ultimately, 2020 was a great success, but in the process we recognized that it could be even better with additional help from our network of drone expert friends from around the state.
Like last year, we came together online for DroneCamp 2021 in July. Over 255 people joined from around the world to learn about theory, application, regulation, and data processing. We learned about the practical aspects of maintaining safety while flying, we took deep dives into various software workflows, and explored agricultural, forestry, and vegetation mapping examples. This year we hosted 5 Plenary sessions, and had some electrifying plenary talk sessions from cutting-edge scientists from around California: Crashing drones! Precision Agriculture! Citizen Science! Mapping aquatic environments!
Because the overall objective of DroneCamp remains to provide the most practical and comprehensive learning experience possible for attendees, we are working on an in-person, hands-on training day that is being scheduled for October, to be held in Monterey/Marina California. And, as the threat of COVID-19 subsides, additional in-person training sessions will be added around the state of California in the coming year.
DroneCamp Overview:
DroneCamp is designed for a wide range of skill levels and interests, for those who are interested in using drones for anything other than non-recreation use (calling for a part 107 remote pilot's license), from complete beginners with little to no experience in drone technology, to intermediate users who want to learn more advanced data processing and analysis. Between presentations on contemporary applications of drones in environmental and agricultural research, and hands-on data processing and analysis exercises, attendees have the opportunity to fully immerse themselves in the following topics:
-
Flight Skills: Safe Launch and Landings, Basic Operations, Traversing and Avoiding Obstacles, Night Flying
-
Safety and Regulations: Safety and Regulations Overview; How to be a Good Visual Observer; Operating in Controlled Airspace
-
Hardware: Sensors, Platforms and Field Accessories
-
Data Collection: Mission Planning; High Precision Mapping
-
Data Processing: Stitching Drone Images with Pix4D, Agisoft Metashape, ArcGIS Pro, and OpenDroneMap; Analyzing Processed Drone Data in QGIS and ArcGIS Pro
-
Data Analysis and Management: Vegetation Analysis, Vegetation Analysis and Classification in ArcGIS Pro;Analysis of the Intertidal Zone; A case study of data management, from collection to storage and sharing of data outputs.
It was super fun and rewarding, and a great success for ANR and all the other collaborators. We built networks, increased collaboration, learned some very cool technical stuff, and got updated on current regulations, including the fact that you can now renew your 107 license easily here.
Some inspiring quotes from anonymous attendee reviews:
I was so appreciative of the extremely high caliber faculty/instructors that were recruited to give presentations, demos, and use of software. You all worked so well together to impart different pieces of expert knowledge. You all are brilliant and I'm inspired!!!
Loved it. We started off heavy which blew my mind but all the talks were so informative and fascinating. Really appreciate the diverse group of people you gathered together. Just wish it was in person! I would love to meet everyone.
Excellent presentation that allowed those of us with ArcMap experience to see the similarities and differences offered by ArcGis Pro. Again, a wonderful presentation (by a professional) that accounts for all the practical steps involved with data manipulation rolled into a final product.
Once I read through my notes, look at my screen captures and watch some of the presentations again, I will be able to structure my drone classes for my students. I'm developing a drone program from scratch for middle and high school students at a local charter school.
Consider signing up for the in-person training in October, and keep DroneCamp 2022 in your sights! Further information will be coming soon to the DroneCamp website, https://dronecampca.org/
- Author: Andy Lyons
- Contributor: Sean Hogan
- Contributor: Nina Maggi Kelly
Like many things that start small and then go viral, the growth of DroneCamp has been welcome but challenging to sustain. The situation just got a whole lot better thanks to a grant from the UC Vice President for Research and Innovation.
DroneCamp was developed and launched in 2017 by the Informatics and GIS program in UC ANR. From the very beginning, the multi-day bootcamp style short course was designed to be highly collaborative and dynamic, with instructors coming from UC ANR, the UC Unmanned Aircraft System Safety center, and drone photogrammetry companies including Pix4D and ESRI. This was not only strategic for building long-term working relationships among campuses and the private sector, but also necessary. The expertise needed to use drones for mapping and data collection is scattered across departments and campuses, and no one unit has the resources to cover all the bases.
Over the next few years, the program grew by leaps and bounds as we went "on the road" to UC San Diego in 2018 and CSU Monterey Bay in 2019. Each year the class size, curriculum, and program complexity grew. In San Diego, the program was hosted by the UCSD Environmental Health and Safety office, with additional sessions by the UCSD Drone Lab; one of the most advanced drone labs in the UC system. DroneCamp 2019 represented a step increase in collaboration, thanks in large part to the efforts of UC ANR Vice President Glenda Humiston who connected us with Monterey Bay DART (Drones, Automation and Robotics Technology), a technology-focused economic development program based out of the nascent UC Monterey Bay Education Science & Technology Center (UCMBEST). The Institute for Innovation & Economic Development at CSU Monterey Bay, under Dean Andrew Lawson, hosted the event, opening their doors to a new academic building and an extremely modern computer lab. Flight training was held at the nearby UC NRS Fort Ord Natural Reserve, managed by UC Santa Cruz. The whole program was planned to coincide with an industry symposium implemented by DART, so that the day after the instructional sessions concluded we shifted gears to an invigorating all-day symposium on technology, policy, and regional economic development.
The scope of DroneCamp took another great leap forward in 2020, when the virtual format blew the lid off the scale and breadth of instructors, workshop topics, and participants. The talented pool of instructors in 2020 (most of whom are returning for DroneCamp 2021), hailed from across California and as far away as Ohio, including:
- UC ANR
- CSU Monterey Bay
- UC Merced
- UC Santa Cruz
- UC Davis
- UC Berkeley
- UC NRS
- Palomar College
- Open Drone Map consortium
The virtual format not only allowed the Planning Committee to expand the core instructional content covering the deep workflow of drone mapping, but we were also able to add panels in specific research applications including coastal environments and agriculture, as well as industry perspectives and technology updates. To manage the load of what was now a medium sized instructional conference, IGIS developed a custom content management system which included an automated registration dashboard to handle the many concurrent Zoom sessions. The beneficiaries of this growth included the over 330 people from 5 countries, but also the expanding network of instructors, researchers, and industry partners.
Growing however comes with costs, and although the success has been rewarding, the complexity of the program has become increasingly unwieldy for the ad hoc Planning Committee to strategically plan and coordinate. Which is why we're thrilled to share that a recent research innovation grant to UC Santa Cruz from UC Vice President for Research and Innovation Theresa Maldonado will support UCSC's growing drone program, including coordination support for DroneCamp. The new funding will allow Dr. Justin Cummings, a UAS operator and data analyst with the California Heartbeat Initiative, to spend more time coordinating the many moving parts of DroneCamp, and support the Planning Committee in continuing to deliver what has evolved over the last 5 years into one of the premiere drone mapping training events in the country.
The support and stability will also allow the Planning Committee to make inroads on some long-standing programming goals. These include developing a stronger needs assessment and evaluation framework, connecting with workforce development initiatives and continuing education programs, expanding fundraising and sponsorships, reviving the scholarship program for under-represented groups, organizing panels that bring together scientists, industry leaders, and funders, and packaging instructional materials into extension products that can be used in other settings as well as asynchronous learning.
The future of DroneCamp is bright, and 2021 should be another information packed week of drone technology and data analysis. So get your propellers on, clear some space on your hard drive, and plan to come to DroneCamp July 26-30, 2021.
DroneCamp Planning Committee:
- Sean Hogan, UC ANR IGIS Program
- Justin Cummings, UC NRS, UCSC
- Brad Barbeau, CSU Monterey Bay
- Chris Bley, InsightUp Solutions
- Becca Fenwick, UC NRS, UCSC
- Corey Garza, CSU Monterey Bay
- Pat Iampietro, CSU Monterey Bay
- Maggi Kelly, UC ANR IGIS Program
- Andy Lyons, UC ANR IGIS Program
- Michael Matkin, UC Santa Cruz
- Josh Metz, DART
- Mallika Nocco, UC ANR
- Brandon Stark, UC Merced
- Author: Andy Lyons
For the past several years, IGIS has been offering online Office Hours for the ANR community on a range of topics related to geospatial science and technology. In 2020, we increased our Office Hours appointments to 4 hours a week. Today the IGIS team is pleased to announce the addition of three new topic areas to our lineup of support for 2021:
Google Earth Engine (January-May 2021 only). Earth Engine is an online platform from Google that provides access to a vast array of imagery and a powerful scripting language for analysis and visualization. Annie Taylor, who taught two extremely popular Earth Engine workshops in 2020, is an Earth Engine whiz and will help you get over any Earth Engine bumps you may be facing.
Cal-Adapt. Cal-Adapt provides access to climate data and visualization tools from California's 4th Climate Change assessment. We work closely with the Cal-Adapt team on a couple of projects and love to talk about climate data. Need to work with Cal-Adapt data in a programming environment? IGIS's Shane Feirer and Andy Lyons have developed add-ons to work with Cal-Adapt data in Python, ArcGIS Pro, and R.
Advanced Google Sheets and Google Apps Script. Underlying Google Sheets, Forms, Docs, and other GSuite apps is a powerful scripting language called Apps Script. We use these tools a lot (in fact the sign-up system for Office Hours is driven by Apps Script), so if you want to do something out-of-the-box with Google Sheets, automate your work, or develop integrated workflows across applications, we might be able to help.
These new topics are in addition to our usual portfolio of support subjects, including:
GIS - desktop software, web GIS, mobile data collection, automation
drones - safety and regulations, equipment, mission planning, data processing and analysis, Pix4D, data management
spatial data and analysis - going from questions to workflows, finding data, spatial analysis
R - programming, working with spatial data in R, statistical analysis, Shiny apps
remote sensing - finding data, image processing and analysis, ENVI
databases - designing data structures, MS Access, SQL
We love Office Hours because it gives us a chance to meet new people, share what we know, and keep us sharp. So please don't be shy when you have a question or don't know where to start - sign up for Office Hours!