Here is Erez Cohen's excellent talk from the BIDS feed: http://bids.berkeley.edu/resources/videos/big-data-mapping-modern-tools-geographic-analysis-and-visualization
Title: Big Data Mapping: Modern Tools for Geographic Analysis and Visualization
Speaker: Erez Cohen, Co-Founder and CEO of Mapsense
We'll discuss how smart spatial indexes can be used for performant search and filtering for generating interactive and dynamic maps in the browser over massive datasets. We'll go over vector maps, quadtree indices, geographic simplification, density sampling, and real-time ingestion. We'll use example datasets featuring real-time maps of tweets, California condors, and crimes in San Francisco.
The BIDS Data Science Lecture Series is co-hosted by BIDS and the Data, Science, and Inference Seminar.
About the Speaker
Erez is co-founder and CEO at Mapsense, which is builds software for the analysis and visualization of massive spatial datasets. Previously Erez was an engineer at Palantir Technologies, where he worked with credit derivatives and mortgage portfolio datasets. Erez holds a BS/MS from UC Berkeley's Industrial Engineer and Operations Research Department. He was a PhD candidate in the same department at Columbia University./h3>
FOSS4G NA 2015 is going on this week in the Bay Area, and so far, it has been a great conference.
Monday had a great line-up of tutorials (including mine on PySAL and Rasterio), and yesterday was full of inspiring talks. Highlights of my day: PostGIS Feature Frenzy, a new geoprocessing Python package called PyGeoprocessing, just released last Thurs(!) from our colleagues down at Stanford who work on the Natural Capital Project, and a very interesting talk about AppGeo's history and future of integrating open source geospatial solutions into their business applications.
The talk by Michael Terner from AppGeo echoed my own ideas about tool development (one that is also shared by many others including ESRI) that open source, closed source and commercial ventures are not mutually exclusive and can often be leveraged in one project to maximize the benefits that each brings.
In fact, at the end of my talk yesterday on Spatial Data Analysis in Python, someone had a great comment related to this: "Everytime I start a project, I always wonder if this is going to be the one where I stay in Python all the way through..." He encouraged me to be honest about that reality and also about how Python is not always the easiest or best option.
Similarly, in his talk about the history and future of PostGIS features, Paul Ramsey from CartoDB also reflected on how PostGIS is really great for geoprocessing because it leverages the benefits of database functionality (SQL, spatial querying, indexing) but that it is not so strong at spatial data analysis that requires mathematical operations like interpolation, spatial auto-correleation, etc. He ended by saying that he is interested in expanding those capabilities but the reality is that there are so many other tools that already do that. PostGIS may never be as good at mathematical functions as those other options, and why should we expect one tool to be great at everything? I completely agree.
Turf provides you with functions like calculating buffers and areas. Other common functions include aggregation, measurement, transformation, data filtering, interpolation, joining features, and classification. The detailed explanations of these functions can be found on this page.
Turf seems like a cool tool to try out if you want to provide spatial analysis functions on your webGIS applications.
One of our collaborators on the Sonoma Vegetation Mapping Project has sent work on how web mapping and high resolution imagery has helped them do their job well. These are specific comments, but might be more generally applicable to other mapping and conservation arenas.
- Communicating with partnering agencies.
- In the past year this included both large wetland restoration projects and the transfer of ownership of several thousands of acres to new stewards.
- Articulating to potential donors the context and resources of significant properties that became available for purchase.
- There are properties that have been identified as high priority conservation areas for decades and require quick action or the opportunity to protect would pass.
- Internal communication to our own staff.
- We have been involved in the protection of over 75 properties, over 47,000 acres. At this time we own 18 properties (~6500 acres) and 41 conservation easements (~7000 acres). At this scale high quality aerial imagery is essential to the size of land we steward and effective broad understanding. The way it is served as a seamless mosaic means it is available to extremely experienced and intelligent people who find the process of searching and joining orthorectified imagery by the flight path and row cumbersome or inefficient.
- Researching properties of interest.
- Besides our own internal prioritization of parcels to protect, I understand that we receive a request a week for our organization’s attention towards some property in Sonoma. Orienting ourselves to the place always includes a map with the property boundary using the most recent and/or high quality imagery for the parcel of interest and its neighbors. This is such a regular part of our process that we created a ArcGIS Server based toolset that streamlines this research task and cartography. The imagery service we consume as the basemap for all these maps is now the 2011 imagery service. This imagery is of high enough resolution that we can count on it for both regional and parcel scale inspection to support our decisions to apply our resources.
- Orienting participants to site.
- Our On the Land Program uses the imagery in their introduction maps to help visitors on guided hikes quickly orient to the place they are visiting and start folding their experience and sense of place into their visit.
- Complementing grant applications.
- Grants are an important part of the funding for major projects we undertake. High quality imagery facilitates our ability to orient the grant reviewer and visually support the argument we are making which is that our efforts will be effective and worthy of funds that are in short supply.
- Knowing what the resources on a property are is an essential part of thoughtfully managing them.
- In one example we used the aerial imagery (only a year old at the time) as a base map for botanists to classify the vegetation communities. These botanists are not experts in GIS, but by using paper maps with high resolution prints in the field they were easily able to delineate what they were observing on the ground on features interpreted in the photo. We then scanned and confidently registered their hand annotations to the same imagery, allowing staff to digitized the polygons that represent the habitat observed. These vegetation observations are shared with Sonoma County and its efforts to map all the vegetation of Sonoma County.
- Conservation easement monitoring makes extensive use of aerial imagery.
- In some cases we catch violations of our easements that are difficult to view on the ground, for example unpermitted buildings by neighbors on the lands we protect, illegal agriculture or other encroachment. It is often used to orient new and old staff to a large property before walking their and planning for work projects that might be part of prescribed management.
- The imagery helps reinforce our efforts to communicate the challenge to preserve essential connectivity in the developed and undeveloped areas of Sonoma County.
- In the Sonoma Valley there is a wildlife corridor of great interest to us as conservation priority. Aerial imagery has been an important part of discussing large land holdings such as the Sonoma Developmental Center, existing conserved land by Sonoma Land Trust and others, and the uses of the valley for housing and agriculture.
- Celebration of the landscape cannot be forgotten.
- We often pair this high quality aerial imagery with artful nature photography. The message of the parts and their relation to the whole are succinctly and poetically made. This is essential feedback to members and donors who need to see the numbers of acres protected with their support and have the heartfelt sense of success.
We look forward to the continued use of this data and the effective way it is shared.
We hope that future imagery and other raster or elevation data can be served as well as this, it would benefit many engaged in science and conservation.
Thanks to Joseph Kinyon, GIS Manager, Sonoma Land Trust
Here is another cool one: real-time map of all the flights in the air. It looks crowded!
Happy and safe travels everyone.