Posts

DC Crime in Zeppelin Hana

HANA & Zeppelin Map DC Crime

In working on a small demonstration application with a map to showcase Volume Analytics and SAP HANA we needed to import the DC Crime data into the HANA database and visualize it in Apache Zeppelin in a leaflet map.

The data is hosted on a dc.gov site with an odd interface. You need to specify that you want the entire district and then pick a time bounded slice of data. Once it finds everything there is a small link that says “Download crime data”. Then you pick the fields you want and press the get data button.

Get DC Crime Data

Data Format

The data it delivers is in a comma delimited format as seen below.

REPORT_DAT,SHIFT,OFFENSE,METHOD,BLOCK,DISTRICT,PSA,WARD,ANC,NEIGHBORHOOD_CLUSTER,BLOCK_GROUP,CENSUS_TRACT,VOTING_PRECINCT,CCN,XBLOCK,YBLOCK,START_DATE,END_DATE
11/16/2018 4:25:34 PM,EVENING,THEFT/OTHER,OTHERS,5100 - 5199 BLOCK OF WISCONSIN AVENUE NW,2,202,3,3E,Cluster 11,001001 1,001001,Precinct 31,18194950,392758,143179,11/16/2018 3:19:45 PM,11/16/2018 4:25:42 PM
11/16/2018 9:46:50 PM,EVENING,THEFT F/AUTO,OTHERS,1400 - 1499 BLOCK OF PARK ROAD NW,3,302,1,1A,Cluster 2,002802 2,002802,Precinct 39,18194960,397068,140306,11/16/2018 3:40:02 PM,11/16/2018 4:30:14 PM

Create Table

I used the HANA studio’s import feature to import the data into a table. I defined the coordinates as DOUBLE columns.

CREATE COLUMN TABLE "CYBERFLOW"."IP_LOCATION" ("IP" VARCHAR(65),
"COUNTRY_CODE"VARCHAR(10),
"COUNTRY" VARCHAR(150),
"REGION_CODE"VARCHAR(10),
"REGION_NAME"VARCHAR(300),
"CITY" VARCHAR(300),
"ZIP" VARCHAR(50),
"LAT" DOUBLE CS_DOUBLE,
"LON" DOUBLE CS_DOUBLE,
"ISP" VARCHAR(150),
"ORG" VARCHAR(150),
"AAS" VARCHAR(150),
"SOURCE" VARCHAR(500),
"LOOKUP_DATE" DATE CS_DAYDATE) UNLOAD PRIORITY 5 AUTO MERGE

The main issue is the spatial reference system being used. It is encoded in NAD 83 Maryland. This page at spatialreference.org describes it.

The problem is that most web map tools do not support this reference system. Most web maps work best with 4326 / WGS 84. NAD 83 Maryland is a flat earth model that works best for the local Washington DC and Maryland area.

Using HANA to Convert

This tutorial helped me figure this out, which explains how to transform from one reference system to another. HANA comes with four spacial systems defined. Of course NAD 83 Maryland is not one of them. But it is not hard to add a new one. This document describes the command needed to add a reference system to HANA.

The first step is to get the well known text for NAD 83 / Maryland. It is found at spatialreference.org. And we need the coordinate range for X and Y. I pulled it from this page but I now believe that is incorrect but my dots on the map of DC seem to be correct.

This is what I ended up with. This HANA command will create the reference system that is close to what I need. I recommend consulting a geospatial expert to confirm accuracy. I made it planar because this is a flat earth model. Taking the code from the SAP HANA tutorial I modified the coordinates in the transform definition at the bottom based on the well known text.

CREATE SPATIAL REFERENCE SYSTEM "NAD 83 / Maryland" 
IDENTIFIED BY 1000026985 
LINEAR UNIT OF MEASURE "meter" 
TYPE PLANAR 
COORDINATE X BETWEEN 36655.85913986 AND 181219.09721397
COORDINATE Y BETWEEN 230987.04872254 AND 574026.61735490
DEFINITION 'PROJCS["NAD83 / Maryland",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.01745329251994328,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4269"]],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    PROJECTION["Lambert_Conformal_Conic_2SP"],
    PARAMETER["standard_parallel_1",39.45],
    PARAMETER["standard_parallel_2",38.3],
    PARAMETER["latitude_of_origin",37.66666666666666],
    PARAMETER["central_meridian",-77],
    PARAMETER["false_easting",400000],
    PARAMETER["false_northing",0],
    AUTHORITY["EPSG","26985"],
    AXIS["X",EAST],
    AXIS["Y",NORTH]]'
    TRANSFORM DEFINITION '+proj=lcc
 +lat_1=39.45 +lat_2=38.3 
 +lat_0=37.66666666666666 +lon_0=-77 +x_0=400000
 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m +no_defs';

Create HANA Table

Now we need a table that we can move the data into. I created a table with both points for both coordinate systems.

CREATE COLUMN TABLE "CYBERFLOW"."DC_CRIME_GEO" 
("REPORT_DAT" NVARCHAR(22), 
"SHIFT" NVARCHAR(20), 
"OFFENSE" NVARCHAR(100), 
"METHOD" NVARCHAR(100), 
"BLOCK" NVARCHAR(200), 
"DISTRICT" SMALLINT CS_INT, 
"PSA" SMALLINT CS_INT, 
"WARD" TINYINT CS_INT, 
"ANC" NVARCHAR(10), 
"NEIGHBORHOOD_CLUSTER" NVARCHAR(20), 
"BLOCK_GROUP"NVARCHAR(20), 
"CENSUS_TRACT"NVARCHAR(20), 
"VOTING_PRECINCT"NVARCHAR(15), 
"CCN" INTEGER CS_INT, 
"GEO_POINT" ST_POINT(1000026985) CS_POINT, 
"GEO_POINT_4326" ST_POINT(4326) CS_POINT, 
"START_DATE"NVARCHAR(22), 
"END_DATE" NVARCHAR(22)) 
UNLOAD PRIORITY 5 AUTO MERGE;

Move and Convert Data

Then I could use a select with insert statement to convert the latitude and longitude doubles into HANA ST_POINTS. By using the ST_TRANSFORM function it is possible to convert from one coordinate system to another. But you must have the coordinate system defined in HANA.

insert into "CYBERFLOW"."DC_CRIME_GEO" select "REPORT_DAT","SHIFT","OFFENSE","METHOD","BLOCK","DISTRICT","PSA","WARD","ANC","NEIGHBORHOOD_CLUSTER","BLOCK_GROUP", "CENSUS_TRACT","VOTING_PRECINCT","CCN", NEW ST_POINT( XBLOCK,YBLOCK).ST_SRID(1000026985).ST_TRANSFORM( 1000026985), NEW ST_POINT( XBLOCK,YBLOCK).ST_SRID(1000026985).ST_TRANSFORM( 4326), "START_DATE","END_DATE" from "CYBERFLOW"."DC_CRIME"

View In Zeppelin on a Map

Now that the data is in HANA in the proper coordinate system we can view it in Zeppelin with the Volume map helium plug-in.

DC Crime in Zeppelin & HANA


Query of a geographic region.

Zeppelin Maps the Hard Way

In Zeppelin Maps the Easy Way I showed how to add a map to Zeppelin with a Helium module. But what if you do not have access to the Helium NPM server to load in that module? And what if you want to add features to your Leaflet Map that are not supported in the volume-leaflet package?

This will show you how the Angular javascript library will allow you to add a map user interface to a Zeppelin paragraph.

Zeppelin Angular Leaflet Map

Zeppelin Angular Leaflet Map with Markers

First we want to get a map on the screen with markers.

In Zeppelin create a new note.

As was shown in How to Use Zeppelin With SAP HANA we create a separate paragraph to build the database connection. Please substitute in your own database driver and connection string to make it work for other databases. There are other examples where you can pull in data from a csv file and turn it into a table object.

In the next paragraph we place the spark scala code to query the database and build the markers that will be passed to the final paragraph which is built with angular.

The data query paragraph has a basic way to query a bounding box. It just looks for coordinates that are greater and less than the northwest and southeast corners of a bounding box.

var sql1 = "select comments desc, lat, lng from EVENT_VIEW "
if (box.length > 0) {
var coords = box.split(",")
sql1 = sql1 + " where lng > " + coords(0).toFloat + " and lat > " + coords(1).toFloat + " and lng < " + coords(2).toFloat + " and lat < " + coords(3).toFloat
}

var sql = sql1 +" limit 20"
val map_pings = jdbcDF.sqlContext.sql(sql)
z.angularBind("locations", map_pings.collect()) 

The data from this query is used to make the map_pings and bind it to angular so that any angular code can reference it. Zeppelin has the ability to bind data into other languages so it can be used by different paragraphs in the same note. There are samples for other databases, json and csv files at this link.

We do not have access to the Hana proprietary functions because Zeppelin will load the data up in its own table view of the HANA table. We are using the command “createOrReplaceTempView” so that a copy of the data is not made in Zeppelin. It will just pass the data through.

Note that you should set up the HANA jdbc driver as described in How to Use Zeppelin With SAP HANA.

It is best if you set up a dependency to the HANA jdbc jar in the Spark interpreter. Go to the Zeppelin settings menu.

Zeppelin Settings Menu

Zeppelin Settings Menu

Pick the Interpreter and find the Spark section and press edit.

Zeppelin Interpreter Screen

Zeppelin Interpreter Screen

Then add the path you where you have the SAP HANA jdbc driver called ngdbc.jar installed.

Configure HANA jdbc in Spark Interpreter

Configure HANA jdbc in Spark Interpreter

First Paragraph

%spark
import org.apache.spark.sql._
val driver ="com.sap.db.jdbc.Driver"
val url="jdbc:sap://11.1.88.110:30015/tri"
val database   = "database schema"   
val username   = "username for the database"
val password   = "the Password for the database"
val table_view = "event_view"
var box=""
val jdbcDF = sqlContext.read.format("jdbc").option("driver",driver)
                                           .option("url",url)
                                           .option("databaseName", database)
                                           .option("dbtable", "event_view")
                                           .option("user", username)
                                           .option("password",password)
                                           .option("dbtable", table_view).load()
jdbcDF.createOrReplaceTempView("event_view")

Second Paragraph

%spark

var box = "20.214843750000004,1.9332268264771233,42.36328125000001,29.6880527498568";
var sql1 = "select comments desc, lat, lng from EVENT_VIEW "
if (box.length > 0) {
    var coords = box.split(",")
    sql1 = sql1 + " where lng  > " + coords(0).toFloat + " and lat > " +  
        coords(1).toFloat + " and lng < " + coords(2).toFloat + " and lat < " +
        coords(3).toFloat
}
var sql = sql1 +" limit 20" 

val map_pings = jdbcDF.sqlContext.sql(sql)
z.angularBind("locations", map_pings.collect())
z.angularBind("paragraph", z.getInterpreterContext().getParagraphId())
// get the paragraph id of the the angular paragraph and put it below
z.run("20171127-081000_380354042")

Third Paragraph

In the third paragraph we add the angular code with the %angular directive. Note the for each loop section where it builds the markers and adds them to the map.

%angular 
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/0.7.5/leaflet.css" />
.
<div id="map" style="height: 300px; width: 100%"></div>
<script type="text/javascript">
function initMap() {
    var element = $('#textbox');
    var map = L.map('map').setView([30.00, -30.00], 3);
   
    L.tileLayer('http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
    var geoMarkers = L.layerGroup().addTo(map);
    
    var el = angular.element($('#map').parent('.ng-scope'));
    var $scope = el.scope().compiledScope;
   
    angular.element(el).ready(function() {
        window.locationWatcher = $scope.$watch('locations', function(newValue, oldValue) {
            //geoMarkers.clearLayers();
            angular.forEach(newValue, function(event) {
                if (event)
                  var marker = L.marker([event.values[1], event.values[2]]).bindPopup(event.values[0]).addTo(geoMarkers);
            });
        })
    });
}
if (window.locationWatcher) { window.locationWatcher(); }

// ensure we only load the script once, seems to cause issues otherwise
if (window.L) {
    initMap();
} else {
    console.log('Loading Leaflet library');
    var sc = document.createElement('script');
    sc.type = 'text/javascript';
    sc.src = 'https://cdnjs.cloudflare.com/ajax/libs/leaflet/0.7.5/leaflet.js';
    sc.onerror = function(err) { alert(err); }
    document.getElementsByTagName('head')[0].appendChild(sc);
}
</script>
<p>Testing the Map</p>

<form class="form-inline">
  <div class="form-group">
    <input id="textbox" ng-model="box" data-ng-change="z.runParagraph(paragraph);"></input>
    <label for="paragraphId">Paragraph Id: </label>
    <input type="text" class="form-control" id="paragraphId" placeholder="Paragraph Id ..." ng-model="paragraph"></input>
  </div>
  <button type="submit" class="btn btn-primary" ng-click="z.runParagraph(paragraph)"> Run Paragraph</button>
</form>

Now when you run the three paragraphs in order it should produce a map with markers on it.

The next step is to add a way to query the database by drawing a box on the screen. Into the scala / spark code we add a variable for the bounding box with the z.angularBind() command. Then a watcher is made to see when this variable changes so the new value can be used to run the query.

Modify Second Paragraph

%spark
z.angularBind("box", box)
// Get the bounding box
z.angularWatch("box", (oldValue: Object, newValue: Object) => {
    println(s"value changed from $oldValue to $newValue")
    box = newValue.asInstanceOf[String]
})

var sql1 = "select comments desc, lat, lng from EVENT_VIEW "
if (box.length > 0) {
    var coords = box.split(",")
    sql1 = sql1 + " where lng  > " + coords(0).toFloat + " and lat > " +  coords(1).toFloat + " and lng < " + coords(2).toFloat + " and lat < " +  coords(3).toFloat
}
var sql = sql1 +" limit 20" 

val map_pings = jdbcDF.sqlContext.sql(sql)
z.angularBind("locations", map_pings.collect())
z.angularBind("paragraph", z.getInterpreterContext().getParagraphId())
z.run("20171127-081000_380354042") // put the paragraph id for your angular paragraph here

To the angular section we need to add in an additional leaflet library called leaflet.draw. This is done by adding an additional css link and a javascript script. Then the draw controls are added as shown in the code below.

Modify the Third Paragraph

%angular 
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/0.7.5/leaflet.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet.draw/0.4.13/leaflet.draw.css" />
.
<script src='https://cdnjs.cloudflare.com/ajax/libs/leaflet.draw/0.4.13/leaflet.draw.js'></script>
<div id="map" style="height: 300px; width: 100%"></div>

<script type="text/javascript">
function initMap() {
    var element = $('#textbox');
    var map = L.map('map').setView([30.00, -30.00], 3);
   
    L.tileLayer('http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
    var geoMarkers = L.layerGroup().addTo(map);
    var drawnItems = new L.FeatureGroup();
    
    map.addLayer(drawnItems);
    
    var drawControl = new L.Control.Draw({
        draw: {
             polygon: false,
             marker: false,
             polyline: false
        },
        edit: {
            featureGroup: drawnItems
        }
    });
    map.addControl(drawControl);
    
    map.on('draw:created', function (e) {
        var type = e.layerType;
        var layer = e.layer;
        drawnItems.addLayer(layer);
        element.val(layer.getBounds().toBBoxString());
        map.fitBounds(layer.getBounds());
        window.setTimeout(function(){
           //Triggers Angular to do its thing with changed model values
           element.trigger('input');
        }, 500);
    });
    
    var el = angular.element($('#map').parent('.ng-scope'));
    var $scope = el.scope().compiledScope;
   
    angular.element(el).ready(function() {
        window.locationWatcher = $scope.$watch('locations', function(newValue, oldValue) {
            $scope.latlng = [];
            angular.forEach(newValue, function(event) {
                if (event)
                  var marker = L.marker([event.values[1], event.values[2]]).bindPopup(event.values[0]).addTo(geoMarkers);
                  $scope.latlng.push(L.latLng(event.values[1], event.values[2]));
            });
            var bounds = L.latLngBounds($scope.latlng)
            map.fitBounds(bounds)
        })
    });

}

if (window.locationWatcher) { window.locationWatcher(); }

// ensure we only load the script once, seems to cause issues otherwise
if (window.L) {
    initMap();
} else {
    console.log('Loading Leaflet library');
    var sc = document.createElement('script');
    sc.type = 'text/javascript';
    sc.src = 'https://cdnjs.cloudflare.com/ajax/libs/leaflet/0.7.5/leaflet.js';
    sc.onerror = function(err) { alert(err); }
    document.getElementsByTagName('head')[0].appendChild(sc);
    s2.onload = initMap;
}
</script>
<p>Testing the Map</p>

<form class="form-inline">
  <div class="form-group">
    <input id="textbox" ng-model="box" data-ng-change="z.runParagraph(paragraph);"></input>
    <label for="paragraphId">Paragraph Id: </label>
    <input type="text" class="form-control" id="paragraphId" placeholder="Paragraph Id ..." ng-model="paragraph"></input>
  </div>
  <button type="submit" class="btn btn-primary" ng-click="z.runParagraph(paragraph)"> Run Paragraph</button>
</form>

There are some important features to mention here that took some investigation to figure out.

Within Zeppelin I was unable to get the box being drawn to be visible. So instead drawing a box will the map to zoom to the area selected by utilizing this code:
element.val(layer.getBounds().toBBoxString());
map.fitBounds(layer.getBounds());

To make the map zoom back to the area after the query is run this code is triggered.

$scope.latlng.push(L.latLng(event.values[1], event.values[2]))
...
var bounds = L.latLngBounds($scope.latlng)
map.fitBounds(bounds)

To trigger the spark / scala paragraph to run after drawing a box this code causes it to run the query paragraph: data-ng-change=”z.runParagraph(paragraph_id);”

<input id="textbox" ng-model="box" data-ng-change="z.runParagraph(paragraph);"></input>

The html form at the bottom is what holds and binds the data back and forth between the paragraphs. It is visible for debugging at the moment.

Query of a geographic region with Zeppelin

Query of a geographic region

Please let us know how it works out for you. Hopefully this will help you add maps to your Zeppelin notebook. I am sure there are many other better ways to accomplish this feature set but this is the first way I was able to get it all to work together.

Demo of the interface:

You can contact us using twitter at @volumeint.

Some code borrowed from: https://gist.github.com/granturing/a09aed4a302a7367be92 and https://zeppelin.apache.org/docs/latest/displaysystem/front-end-angular.html

Mapping an Epidemic

Mapping an Epidemic

This map changed the way we see the world and the way we study science, nature, and disease.

In August of 1854, cholera was ravaging the Soho neighborhood of London where John Snow) was a doctor. People were fleeing the area as they thought cholera was spread by gasses in the air or, as they called it, “bad air.”

Just as there is disinformation today about Ebola being airborne, the experts of that time thought most disease was spread in the air. There was no concept that disease might be in the water. They had no idea that bacteria even existed.

John had worked as a doctor in a major outbreak of cholera in a mine. But despite working in close quarters with the miners, he never contracted the disease. He wondered why the air did not affect him.

This inspired him to write a paper on why he believed cholera was spread through water and bodily fluids. The experts at the time did not accept his theory; they continued to believe cholera was caused by the odors emitted by rotting waste.

In the Soho outbreak in August 1854, John Snow saw a chance to further prove his theory. He went door to door keeping a tally of deaths at each home. This was only part of his quest to find evidence to prove the source of the plagues of the day.

He had been collecting statistical information, personal interviews, and other research for many years. He added this information to his paper, “On the Mode of Communication of Cholera.” The paper and his work in researching and collecting evidence founded the science of epidemiology.

One of the most innovative features was plotting data using a map; it was the first published use of dots on a map supporting a scientific conclusion. Each of the bars on John Snow’s map represents one death. Using this visual technique, he could illustrate that the deaths were centered around a point and further investigate and interview people in the area. He could also find anomalies and outliers such as deaths far from the concentration and areas with no deaths.

Epicenter Pump and Brewery

He found through personal interviews and mapping the data that the workers in the brewery (in the epicenter of the epidemic) were not dying. The owner of the brewery said that the workers were given free beer, and he thought that they never drank water at all. In fact, there was a deep well in the brewery used in the beer. In other cases, John Snow found that addresses with low deaths had their own personal well.

He also investigated the outlying incidents through interviews: some worked in the area of the pump or walked by it on the way to school. One woman who got sick had the water brought to her by a wagon each day because she liked the taste of that particular well water. One person he talked to even said the water smelled like sewage and did not drink it, but his servant did and came down with a case of cholera.

The incidents highlighted the area around a public pump on Broad Street. Using his data, he convinced the local authorities to have the pump handle removed.

The most innovative feature of the map is that it changed the way we use maps. The idea that data could be visualized to prove a fact was very new.

John Snow’s map of the service areas of two water companies

John Snow also produced another map showing which water companies supplied water in London. This map showed that the water company which stopped using water from the Thames had a lower death rate due to cholera. The map allowed John Snow to provide further evidence of disease spread through water and what could be done to fix the issue.

This is similar to the Ebola outbreak of today where tracking the disease is important. John Snow’s idea of collecting data in the field and mapping it lives on in maps like those from HealthMap, which show the spread of the Ebola virus.

Data Exploration via Map

Today, we use data driven maps as a powerful tool for all sorts of reasons. But it all started with John Snow.

(For an interesting take on this event and other historical technology that changed the way we live today, watch the “Clean“ episode of the How We Got to Now series on PBS.)

To learn more about Volume Labs and Volume Integration, please follow us on Twitter @volumeint and check out our website.

10

10+ Surprising Geospatial Technologies

Data Organized on Map

I’ve spent years in the geospatial arena, so I’m a bit of a geospatial technology geek. But now it seems like the rest of the world is increasingly interested in this technology too.

You may remember the old latitude and longitude numbers that you learned about in school. Perhaps they didn’t seem very useful or relevant to life at the time, but these coordinates are now tracked constantly with our various GPS enabled gadgets. It’s becoming increasingly common to use coordinates to define the location of data collected, a person, landmark, and more. We can add even further accuracy by recording elevation and point in time.

I would like to describe some of the components that fall under the umbrella of geospatial technology. You might find some surprises!

Equipment

First, let’s discuss some of the tools used to collect geospatial data.

1. GPS

Global Positioning System (GPS) technology is the software and equipment needed to provide the location of things on the planet. This is most often done with the use of special satellites but is often augmented by other methods like WiFi signals. There are even technologies in use that determine location by looking at the stars.

2. Field Sensors

Field sensors are electronic devices that are placed to collect information about weather, soil, or other environmental conditions. These data collecting devices could be anything from a camera to a cell phone. During collection, the data is tagged with geospatial information, so the location of the event is known and can be mapped.

Overhead Imagery

My next geospatial category is overhead imagery. This includes all the imagery from aircrafts and satellites.

3. Visual Overhead Imagery

Visual overhead imagery includes what you see in Google Maps and Google Earth when you use the satellite function. This imagery could be collected via satellite or aircraft, and the technology used involves cameras, aircraft, satellites, global positioning systems, altimeters, and microwave transmission equipment. Today, even video is collected overhead by Planet Labs.

If you don’t own an airplane or satellite, can you collect visual overhead imagery? Yes! It doesn’t have to be expensive. Some hobbyists and students are cutting their teeth on low-cost imagery collection using kites and balloons.

Balloon mapping of Lake Borgne, Louisiana (Cartographer: Stewart Long/publiclab.org)

4. Hyperspectral Overhead Imagery

Hyperspectral refers to the waves of light that are beyond human sight. Engineers have developed sensors that can gather these waves from space, but it can also be done from aircraft. The data is then transformed into a visual representation through analysis and processing to create hyperspectral overhead imagery.

This type of geospatial technology has some surprising uses. Over at the US Geological Survey (USGS), they have used hyperspectral overhead imagery collected via satellite to detect the presence of arsenic in the leaves of ferns. Further analysis led them to aid in locating arsine gas canisters buried in Washington, DC. For more information, check out the full dissertation entitled _Remote Sensing Investigations of Furgative Soil Arsenic and its Effects on Vegetation Reflectance_.

5. LIDAR

Light Detection and Ranging (LIDAR) is a technology that uses an airborne system to measure distance by shining a laser to the ground and measuring the reflected light. This yields a very accurate contour of the earth’s surface as shown in the image of the Three Sisters below.

LIDAR image of the Three Sisters volcanic peaks in Oregon (DOGAMI)

LIDAR can also measure objects on the ground such as trees and houses. This type of data is used to determine elevation and is often used when processing other imagery to improve accuracy.

How do autonomous vehicles “see” where they are going and what is in the way? LIDAR, of course! Plus, it’s even used in various industries to make 3D models of buildings and topography.

Processing

So now that we collected all this imagery, how do we use it?

6. Imagery Processing Systems

The overhead imagery produced from satellites and aircraft is not perfect for human viewing in raw form. So we use imagery processing systems to help automate the manipulation of images and data collected. This collection of computer systems makes the images and data useful to us.

Most images are taken from an angle and must be adjusted or warped. Imagery processing systems assign each pixel a geographic coordinate and an elevation. This is done by combining GPS data that was collected with each click of the camera.

Often this process is called orthorectification. To see a simplified illustration, take a look at this orthorectification animation from Satellite Imaging Corporation.

7. Geospatial Mapping

Geospatial mapping is the process and technology involved in placing information on a map. It is often the final stage of geospatial processing.

Mapping combines data from many sources and layers it onto a map, so conclusions can be drawn about the data. There are different degrees of accuracy required in this process. For some applications, showing data in an approximate relation to each other is sufficient. But other applications, like construction and military exercises, require specialized software and equipment to be as precise as possible.

In an earlier post, I wrote about creating maps with D3. The goal was to build a heat map to display the count of documents for each place name as shown in the image below.

Data Organized on Map

Applications

Let’s explore the some of the applications of all this geospatial technology.

8. Geospatial Marketing

Geospatial marketing is the concept of using geospatial tools and the collection of location information to improve marketing to customers. This is often a subset of geospatial mapping, but this application combines data about customers’ locations. This can help determine where to place a store or how many customers purchase from a particular location. For example, companies can use data about where people typically go after a ballgame to determine where advertisements should be placed.

Another widespread application of geospatial data in marketing is using the IP addresses gained from customers browsing websites and viewing advertisements. These IP addresses can be geographically located, sometimes as specifically as a person’s house, and then used to target advertisements or redesign a website.

9. Location-Aware Applications

Location-aware applications are a category of technologies that are cognizant of their location and provide feedback based that location. In fact, if an IP address can be tied to a location, almost any application can be location-aware.

With the advent of smart phones, location-aware applications have become even more common. Of course, your phone’s mapping application can display your location on a map.

There are also smartphone apps that will trigger events or actions on a phone when you cross into a geospatial area. Some examples are Geofencer and PhoneWeaver.

Additionally, the cameras on smart phones can collect the location of the phone when taking a picture. This is imbedded within the picture and can be used by Facebook, Picasa, Photoshop, and other photo software to display locale information on a map. (You may want to disable this feature if you would rather not have people know where you live.)

10. Internet of Things

The Internet of Things (IoT) is the category of technology that includes electronic objects that connect to the internet and transmit their location. This is a broad and emerging area of geospatial technology that will add even more location data to the world.

IoT could contain objects like cars, fire alarms, energy savings devices like Nest and Neurio, fitness tracking bands like the ones from Jawbone or Nike, and more. For these IoT applications and devices to work optimally, they need to know your location and combine it with other information sensed around them.

Nike+ FuelBand (Peter Parkes/flickr.com)

11. Geospatial Virtual Reality

Virtual reality that makes use of geospatial data is another emerging category. This technology will allow for an immersive experience in realistic geospatial models.

Geospatial virtual reality incorporates all of the technologies listed above to put people into the middle of simulated real-word environments. It’s already been implemented with new hardware like the Oculus Rift, which is a virtual reality headset that enables players to step inside their favorite games and virtual worlds.

Oculus Rift (Sebastian Stabinger/commons.wikimedia.org)

Show Me the Data!

At the base of all of this technology is data. Increasingly, we have to invent more ways to store geospatial data in order for it to be processed and analyzed. The next steps of geospatial technologies involve attaching geospatial information to all data collection and then processing and filtering the massive amounts of data, which is known as big data.

This is my list of surprising geospatial technologies that matter today. It started out as a top 10 list, but evolved to 11 because I just couldn’t leave out geospatial virtual reality. It’s so cool! Feel free to add your suggestions of geospatial technologies in the comments below or as a pingback.