Surveying, Mapping and GIS

Exploring all aspects of mapping and geography, from field data collection, to mapping and analysis, to integration, applications development and enterprise architecture...

  • Geospatial Technology, End to End...

    Exploring all aspects of mapping and geography, from field data collection, to mapping and analysis, to integration, applications development, enterprise architecture and policy

EPA ARRA Mapper

Posted by Dave Smith On 12/02/2010 09:20:00 PM 15 comments

Since joining EPA I've been engaged in a wide variety of projects and efforts - one which we are currently getting out the door is an upgraded mapper for EPA projects funded by the American Recovery and Reinvestment Act (ARRA), otherwise known as Stimulus or the Recovery Act.

The major categories of projects receiving EPA funding via ARRA include Superfund Hazardous Waste Cleanup, Leaking Underground Storage Tanks, Clean Water State Revolving Fund (typically wastewater treatment), Drinking Water State Revolving Fund (potable water), National Clean Diesel Campaign, and Brownfields.

 The idea was to provide more granular data across the various programs where EPA has been getting ARRA funding to projects.


The mapper reports on a quarterly basis, in concert with the ARRA reporting requirements, and was built on the ESRI Flex API. As a quick overview, it shows statewide figures, as choropleth map, with summary tables:

Visitors can click on the menu to view awards by program category, or to view all awards, for example:


The pushpins indicating awards can then be selected, and info boxes will pop up with the details. In the example below, we asked the mapper to show "Chelsea, MA" and turned on Clean Diesel awards, and clicking on the map pin, we get the goods, two awards for Chelsea (note, spelling of "Collabrative" comes directly from the database):


This application should improve transparency, with the direct intent of showing tangible benefit to users in showing what's going on right at the community level. As for lessons learned, the technology was far less of a challenge than the learning curve of how government works, and navigating my way through various EPA offices and stakeholders and gaining their acceptance and participation. My many thanks to all those who helped out.

FEA, CPIC, LoB, and Strategic Alignment...

Posted by Dave Smith On 11/14/2010 08:59:00 AM 7 comments

Since joining a federal agency and becoming a national program manager of a federal IT investment, I have been navigating various things like Capital Planning and Investment Control (CPIC), Line of Business (LoB) initiatives, Federal Enterprise Architecture (FEA), and other pieces – there is certainly no shortage of process and documentation required throughout the lifecycle as systems are planned, designed, built, maintained and decommissioned. However, for all of its compliance documentation, metrics and matrices, I still think there are a number of core disconnects.

CPIC and LoB initiatives should inform on investments, and align investments. One area where I see inadequate trackability is in mission support. Each and every IT investment should be able to map to a matrix of mission drivers – such as Agency Strategic Plan elements, such as stated priorities and core initiatives within the agency, such as specific laws and mandates which the Agency is charged with carrying out. In turn, these mappings can be aggregated and examined for alignment. If for example, if the mission objective is to assess the impact of a specific activity on a population, then there is now opportunity to understand how many IT investments relate to that assessment, and one can then get any potentially disparate and disconnected activities aligned and harmonized, to leverage each, take advantage of opportunities to share things like data, models and infrastructure instead of having stovepiped activities where each party reinvents the wheel independently of the next.

Additionally, functional components should be mapped to as well. For example, data requirements, modeling requirements, geodata hosting and web services needs, and so on. These can help to inform on infrastructure investments – for example, being able to build a robust, shared, scalable environment with load balancing and fault tolerance instead of having a series of fragile, disconnected stovepipes with no scalability or fault tolerance. It can help toward paradigm shifts like leveraging cloud capacity and other types of things which can provide cost savings – which can then hopefully be driven toward innovation and new development, rather than more stovepipes and reinventing of the wheel.

As I have a background which straddles many disciplines, when I hear “Enterprise Architecture” it conceptually still goes back to old-school, bricks-and-mortar architecture, where a building is built from a blueprint.

Consider the enterprise as a house. You have several different rooms in it, serving different functions, yet for it to function effectively as a house, the work of the architect is to draft up the elements that bring it all together into a functional, cohesive whole. That means, the structural members to support the second floor as it is placed on the first floor, that means the stairs and corridors needed to connect the rooms, that means the placement of the rooms, to give them doors and windows to daylight where needed, the arrangement of them relative to each other and to the corridors and stairs, to ensure good flows where needed, the wiring to provide light, power and communications, the plumbing to bring water to and from where it’s needed, the HVAC systems to regulate heat and cold, and so on.

Yet, for all the talk in IT EA communities, most organizations largely still function as a series of disconnected, disjointed rooms. The EA effort should serve as the master blueprint. It needs to be informed by those who need each room, but in turn also needs to inform on how everything connects and how things flow from one room to the next, and where the wiring and plumbing is, and how to connect things and create meaningful flows, relationships and functionalities. For the developer, Enterprise Architecture should inform Solution Architecture, and where gaps are identified, that should in turn go back and inform Enterprise Architecture. The loops need to be closed. All of these things, FEA, CPIC, LoB and others, need to move beyond paperwork and compliance exercises, to becoming more robustly informed and cohesive, serving as the master blueprints and roadmaps.

In terms of metrics to gauge success, the best metrics would be those which demonstrate that alignment on mission, function and coordination have been achieved.

MyPropertyInfo

Posted by Dave Smith On 9/10/2010 10:54:00 AM 5 comments

As I have been ramping up on EPA's Facility Registry System over the last couple of months since coming on board with EPA, I have also had the opportunity to work on a number of other projects - one recent one that's rolled out is MyPropertyInfo.

The most truly fun thing about working in EPA's Office of Environmental Information is that they are involved in a lot of collaborative, cross-cutting efforts, so I get exposed to a lot of different things across the agency. As an example of this, in working with EPA's Freedom of Information Act (FOIA) officer Larry Gottesman and FOIA staff, they were pursuing an idea of greater accessibility toward reducing FOIA requests, such as in the case of common requests for data which actually is already being published by EPA, but which may be scattered across separate locations in the agency.

One example of this is MyPropertyInfo - http://epa.gov/myproperty/

Here, we sought to address frequently-asked questions about properties. This type of basic background and screening is highly useful and important to bankers, realtors, prospective buyers, developers and others who deal in real estate and properties - yet, to gather all of the relevant information about a property, one might have to visit multiple sites across EPA, or to submit a FOIA request and wait to have EPA gather the data from those disparate sources. So what we did in the case of MyPropertyInfo is quickly roll out a tool that basically just gathers that existing content in one place, and additionally provide it in printer-friendly form.

Thought it was essentially just screen-scraping (as we do not directly control some of the source reporting systems), it was nonetheless a quick and effective way of getting questions answered.  Moving foward, it again demonstrates also that using approaches that can provide easily integratable content like web services in addition to traditional HTML reports, content can be even more elegantly repurposed and reused in a variety of effective ways to answer business questions - with web services associated with the reporting engines, the widgets and iPhone apps for these types of applications will virtually build themselves.  For example, real estate sites like Zillow.com would also be able to dynamically pull environmental profile information about properties of interest to prospective buyers - hopefully a vision for the future at EPA.

Here is some additional perspective on MyPropertyInfo as posted to EPA's Greenversations blog by the FOIA office's Wendy Schumacher:  http://blog.epa.gov/blog/2010/08/30/my-property-info/

Traffic Congrestion, LBS and ITS

Posted by Dave Smith On 8/14/2010 12:33:00 PM 10 comments

For something along a different theme, given my extremely late (1:30AM) arrival back in Pennsylvania last night, Intelligent Transportation Systems (ITS) and Congestion Management.


Previously, I had been involved in a few projects involving Intelligent Transportation Systems - and yet, it still amazes me how far behind we are in terms of even basic approaches.  Last night, I was stuck in traffic due to an unfortunate accident ahead on the roadway.  My immediate observations:  A state trooper was sitting on the side of the road, with his mandate to alert drivers and monitor the end of the queue for problems.  However, where he was situated, as so often happens, was past the last exit available where motorists could get off the interstate and find an alternative.

Had he been situated ahead of that last exit, I and so many other motorists with onboard GPS could quite easily have hit our "DETOUR" buttons and navigated around the congestion rather than end up in the midst of it.  But instead, we end up stuck, and the congestion and queue only grows and grows.  Poor congestion management.

Secondly, a police cruiser sitting on the side of the road certainly alerts drivers of something.  However, it doesn't give any specificity whatsoever.  Perhaps, it was just that the trooper stopped a speeder, et cetera.  Often they are sitting well behind the congestion queue, and sometimes it's not immediately evident that there is congestion ahead.  Opportunity for informing motorists is lost, and the situation is not mitigated and managed as well as it could be.


It would seem to me that there are any number of relatively simple ways to address and mitigate congestion as a result of an accident or other similar traffic event - we certainly have ample technologies available.  For example, a portable variable message board that could rapidly be deployed by troopers (as in the photo), or other alerts.  There are numerous Variable Message Sign (VMS) boards along interstate corridors, yet amazingly, to this day, they still are largely uncoordinated, where messaging is not propagated along the corridor across district or state boundaries.  Highway officials still seem to not recognize that roadways are functionally networks, that internal administrative boundaries are not appropriate barriers as far as motorists are concerned.

It would also seem that protocols like CAP messaging, GeoRSS and others could and should be leveraged, and combined with very simple, wireless digital broadcast technologies, aligned to highway advisory radio beacon broadcasts, to provide simple, low-cost means of transmitting location-based information to in-dash receivers, GPS units and so on.  Certainly some such systems exist, however via subscription, or at additional cost to the price of the receivers, and so on - however, perhaps a better business model for such as broadcast system could operate via public-private partnerships, where operators of hotels, restaurants and amenities could fund the system by providing basic information about available attractions and amenities when there are no highway incidents.  A perfect case for Location-based service (LBS) technologies.  This does not have to be a costly, complicated thing.  We already have all of the ingredients and have had them for several years.

Locational Data Policy and Tools

Posted by Dave Smith On 8/07/2010 11:22:00 AM 2 comments

As I've posted previously, one of the newest hats I now wear is that I'm now the national program manager for EPA's Facility Registry System (FRS), where I am collecting and managing locational data for 2.9 million unique sites and facilities across states, tribes, and territories - I'm certainly excited about being able to contribute some good ideas toward enhancing its' capabilities, holdings, and collaborating and integrating with others across government.

A large part of this is data aggregated from other sources, such as data collected and maintained by state and tribal partners, EPA program offices and others, and then shared out via such means as EPA's Exchange Network. Historically, FRS has done what it can to improve data quality on the back end, by providing a locational record which aggregates up from the disparate underlying records, with layers of standardization, validation, verification and correction algorithms, as well as working with a national network of data stewards. This has iteratively resulted in vast improvements to the data, correcting common issues such as reversed latitude and longitude values, omitted signs in longitudes, partial or erroneous address fields and so on.

However, it still remains that there remain some issues with the data, with the weakness being in how data is collected, imposing limits on what kinds of backend correction can be performed. In most cases, data is captured via basic text fields. The further upstream that the data can be vetted and validated, the better, in particular, right at the point of capture, for example instances where facility operators themselves enter the data.

So, here is the notion - a toolbox of plug-and-play web services and reusable code to replace the basic free-text field, which allows real-time parsing and verification of data being entered. Part of that may involve using licensed commercial APIs to help with address verification and disambiguation, for example, the Bing Maps capability to deal with an incomplete address or one with a typo, such as "1200 Contitutoin, Wash DC" the web services would try to match these and return "Did you mean 1200 Constitution Avenue NW, Washington, DC?"


Between suggesting an alternative which attempts to correct partial and/or incorrect addresses, and providing an aerial photo as a visual cue for verification, it improves the likeihood that the user is going to doublecheck their entry and either accept the suggested alternative or type in a corrected address, along with having the visual verification in the aerial photo. Notionally, if the aerial photo view shows a big open field where there should be a large plastics plant, they would stop and wonder, and perhaps doublecheck the address they had entered.

That's certainly a good first step, and is something I'm currently looking to promote on the short term. In talking to some of my EPA stakeholders, they are very interested in this, and I will look at developing some easy-to-integrate code for them to use.

But, to think more long-range, let's take that further - from the large universe of facilities that I deal with, not all things populating "address" fields are conventional street addresses. For example, remote mining activities in western states might instead be represented on the PLSS system, such as "Sec 17, Twp 6W Range 11N", or rural areas might simply use "Mile 7.5 on FM 1325 S of Anytown" or "PR 155 13.5km west of Anytown".

Again, perhaps there are ways to improve this, a longer-term discussion, but certainly the ingredients exist. A first step might be to look at developing guidance on consistent ways to have folks enter this type of data, for example "Sec 17, Twp 6W Range 11N" versus S17-T6W-R11N, along with developing parsers that can understand and standardize the possible permutations that might be entered, including entry of section and meridian info, e.g. NW1/4 SW1/4 SE1/4 SEC 22 T2S R3E MDM for an entry that includes drilldown into quarter sections to identify a 10-acre parcel, also referencing the Mount Diablo Meridian.


Currently, there isn't any truly standardized way of entering and managing these, but perhaps there is a role in the surveying community toward standardized nomenclature to assist in database searching and indexing.  Coincident with this is potential collaborative development of ways to approach parsing and interpreting nonstandardized entries, along with leveraging existing PLSS data and  geocoders built toward translating these into locational extents, such as a bounding box, along with provisioning it with appropriate record-level metadata describing elements such as method of derivation and accuracy estimate.

In concert with this, obviously, should be an effort toward providing linkages to actual field survey polygonal data, as appropriate if it's a parcel-oriented effort (for example, for superfund site cleanup and brownfields), and where such data is available.

Similarly, one could collaboratively develop guidance and parsers for dealing with the route-oriented elements, for example "Mile 7.5 on FM 1325 S of Anytown" or "PR 155 13.5km west of Anytown" toward standardizing these types of fields as well - for example, whether or not to disambiguate or expand FM as "Farm to Market Route", what order to place elements consistently.

And then, one would want to leverage routing software to measure the distance along the given route from the given POI, toward providing a roughly-geocoded locational value to get in the ballpark.  And again, one would want a web service that does this to return any appropriate metadata on source, error and so on.

PLSS locations, mileage-along-a-route locations, and things like this are just a sampling of the universe of possibilities.  And as I point out, there are bits and pieces of tools that can do some of these things, but they are currently scattered and uncoordinated, and community-oriented, collaborative efforts can help to pull some of these together.

Atop these, as mentioned above one could also provide additional pieces, such as tools for visual verification, at the most basic level, or, if collection mandates permit, tools to allow the user to drop a pushpin on an aerial photo feature, drag a bounding box, or digitize a rough boundary - (and most ultimately of course, a means of entering and/or uploading survey data for field-located monumentation points, boundary topology, and record description data).

From a federal perspective, EPA is certainly not the only agency that needs some of these types of tools, and EPA is certainly not the only agency that needs internal policy and/or best practices guidance on how to deal with how these types of values are best represented in databases.  It would make sense, from an Enterprise Architecture perspective, for the federal community to collaborate, along with state, tribal and local governments.  Similarly, I would think that there are a lot of non-profits, academia and private sector entities that have a big stake in locational data improvement that could benefit from improved data that would be facilitated by such tools, along with benefitting from such tools for data collection themselves.

For my part, I will try to do what I can toward leading the charge on these, and to leverage any existing efforts already out there.  Additionally, given the capabilities that FRS has, I am looking to continue to integrate across internal stakeholders as well as external agencies toward being able to aggregate, link and reshare, with a process where data is iteratively improved and upgraded collectively.

I'd certainly be interested in getting thoughts, ideas and perspectives from others on this.

Federal Geospatial Platform

Posted by Dave Smith On 6/25/2010 01:31:00 PM 0 comments




"In 2010 and 2011, Federal data managers for geospatial data will move to a portfolio management approach, creating a Geospatial Platform to support Geospatial One-Stop, place-based initiatives, and other potential future programs. This transformation will be facilitated by improving the governance framework to address the requirements of State, local and tribal agencies, Administration policy, and agency mission objectives. Investments will be prioritized based on business needs. The Geospatial Platform will explore opportunities for increased collaboration with Data.gov, with an emphasis on reuse of architectural standards and technology, ultimately increasing access to geospatial data."
- President’s Budget, Fiscal Year 2011
Earlier this week, I had the opportunity to attend one day of the multi-day National Geospatial Advisory Committee (NGAC) meeting in Shepherdstown, WV - Though I had attended a few of their meetings previously, the thing that I was most interested in for this particular meeting is discussion of the emergent Geospatial Platform.  As the NGAC is a FACA committee, non-member involvement and participation is limited, though was able to get to get a sense of things...   The intent has a lot of potentialities, such as being able to provide more robust visualization and geospatial tool capabilities such as geocoding services, to augment ongoing Data.Gov efforts, as well as support ongoing FGDC initiatives.

The sense I got is that thre are many pieces still undefined, many things not yet prioritized, and so on, and discussion of options and alternatives was a big focus of the discussion that I sat in on - and one big challenge yet ahead is refinement of governance structure - as to how its' strategic vision and board of directors would be structured and at what level of government, where its' managing partner and operational/tactical component would reside.  These things are being discussed in iterative fashion, as they develop their roadmap.

Some interesting thoughts and ideas, not new, but again being raised - of the need for a strategic ownership stake in geospatial data at the White House level, e.g. a Federal GIO to reside within OMB, who would have the authority and teeth to centrally direct and manage geospatial policy in executive agencies at the federal level.  Another question, at the operations level, of commoditized geospatial technology platforms which could be shared across all federal agencies - a federal geocoding service, for example, or a geospatial data services hosting platform where individual stewards in agencies which might not have their own robust geospatial infrastructure could upload data and register it, apply symbologies, et cetera to be served as fed-wide robust, shared OGC web services, and so on.

What shape this will take is yet to be determined, and it sounds like the plan is to firm up the roadmap internally within the NGAC and then circulate it more broadly when v3 of the Roadmap is completed.

More about the Geospatial Platform:


Other coverage, items and discussion:

Think Globally, Code Locally

Posted by Dave Smith On 5/23/2010 10:35:00 AM 5 comments

One of those key philosophical notions that circulates among many stewardship and advocacy groups is to think globally, act locally - The same notion can be applied in best practices in solutions architecture, requirements development, and coding principles, where it applies to so many things - in terms of true enterprise architecture and integration approaches, in terms of data stewardship, and for public sector agencies, in terms of transparency and open government.

Each and every day, folks craft queries for accessing complex data, they write business logic for analysis and so on - and more often than not, they then wrap it up with a nice glossy user interface.  MVC and many other best practices applied, check...  But here is where another key opportunity to really open things up is lost.

Let's take the example of state or federal agencies - many of them are doing great things, in terms of publishing datasets (Open Government IntitiativeData.gov, Geospatial One-Stop, State Cartographers' Offices and Clearinghouses, and so on).  They are doing great things in terms of reporting tools, web queries and lookups, and mapping and visualization tools.  But...  As wonderful as all of these efforts are, imagine how powerful it would be if these could be linked and integrated, from agency to agency?  Imagine, for example, applications pulling in data from NOAA, NASA, EPA and other agencies?  Imagine state environmental agencies pulling in information from federal environmental agencies - the power of open government and transparency begins to multiply and become more and more powerful by orders of magnitude.

So what are we talking about?  Web services and APIs.  Not even anything complex, it could be as simple as REST and JSON, as in the space of public-facing applications, for a vast majority of data-centric or analytical applications, we are generally talking about read-only, non-transactional, nonsensitive data.  Why web services or APIs?   Isn't something like publishing datasets to Data.gov enough?  Publishing datasets to someplace like Data.gov is fine - but consider this - if someone downloads your data and integrates it into their own standalone application, what control do you then have over it?  Precious little.  When you refresh your database afterward and post a new dataset, is there any assurance that they will come back and get a fresh copy and keep up to date, or do you begin to incur a liability of your data being out of sync as it is represented in someone else's public-facing application?  Absolutely.

If, on the other hand, you post an easy-to-integrate RESTful service or API, you make it far easier to have your data current.

Criticisms?  Yes, there are many, but we can look at those as well - some might push back and say, "is there sufficient bandwidth and infrastructure for hosting those web services?"  The mere fact of web services in and of themselves generally incur no more new bandwidth or infrastructure demand than the public-facing web applications that you are already building.  So if that's really a concern, then maybe you shouldn't be building ANY applications.  The argument is generally a red herring.

Security concerns?  Same deal - you are already exposing the data and business logic via your web applications - if you think the web service poses new security concerns, then perhaps you haven't adequately thought through your EXISTING security.  So another red herring.

"They will come and take all our data!" - Again, why would they, if they know they can grab it dynamically and that you will be assured of their reliance on them by continually posting data refreshes?

What about poorly-behaved queries against your web service that might bring your database to its' knees?  Same thing - those are things that really should be handled in the logic for your existing application.  Better yet, think ahead a bit more and provide options - query filters, such as by a data attribute or domain, bounding boxes for geodata, and so on.

So - in conclusion, this type of thinking can up developers' approaches to how they deal with security, queries and filters, and so on, and generally tighten things up - but in the meantime, also provide a tremendous opportunity for data sharing and integration, whether within an agency, company or office, fostering collaboration across departments and programs, or outside of one's own organization, fostering collaboration across agencies and levels of government, across industry, academia and so on.

An individual organization can try all they like to anticipate all of the public's thoughts, needs, interests, they can try to walk in their stakeholders' footsteps - but ultimately, one can never fully anticipate all possibilities, and there are a lot of bright people out there with many untapped ideas - so why not open it up and facilitate that larger collective brainstorm of ideas by providing services ready to integrate?  Let's take things like the Open Government Initiative and other good things ongoing in government and elsewhere to the next level - dynamic, services and resources oriented architecture...

U.S. Environmental Protection Agency...

Posted by Dave Smith On 4/05/2010 05:52:00 PM 4 comments


My big news to report:  After 6 years at Synergist Technology Group, Inc., I am changing directions.  I have accepted a position at the U.S. Environmental Protection Agency's Office of Environmental Information.  I will be starting at USEPA the week of April 26th, 2010.

I greatly enjoyed the work I was doing at Synergist, having been involved in everything from business development to writing code and everything in between, supporting geospatial, mapping and integration efforts to support diverse business areas from environmental protection to emergency response to transportation to military and intelligence applications.  I enjoyed the diversity.

I have worked in private sector, consulting for over 20 years, but all throughout this, have always been solutions-focused, and have always had a keen interest in supporting the core mission of the agencies that I worked with - though, the downside of private sector consulting is always in the vagaries of its' occasionally fragmentary nature, with competetive bidding and teaming arrangements often dynamic and changing.

So, coming into Federal service brings a whole gamut of new things.  In some ways, I will get to dig deep into some of my core passions of environmental protection, geospatial technology and integration - and I look forward to this. However, it will surely bring many other challenges along with it - but I am up for any challenge.

My new role will, among other things, involve support to EPA's Facility Registry System (FRS), which is a repository for facilities data for 2.5 million unique facilities regulated under a variety of environmental laws and statutes. Herein, I see opportunity for further geo-enablement of Agency business processes, along with continued integration potential, to put hooks and eyes into a variety of other processes for ever-improving analytical, reporting and querying capabilities, cross-media, cross-agency, cross-domain, cross-dimension, as well as facilitating further partnership-building, between local, state, and federal government, tribes, academia, industry and the public, and supporting data publishing, OGI and transparency initiatives.

I anticipate I will also be pulled into a variety of other geospatial efforts across EPA as well... In many ways, I will essentially continue to do many of the things I already did - only, from the other side of the fence. In terms of physical logistics, I have already essentially been commuting to Washington DC for 6 years - though, now having a permanent duty station there, I can now consider getting a place to stay, as opposed to living out of hotels and bouncing between various locales as I have been.

So - mixed emotions - I will certainly miss working with Synergist, but likewise, there is a great team at EPA, and I am sure I will have a great time there.

ESRI and Volunteered Geographic Information

Posted by Dave Smith On 2/20/2010 10:36:00 AM 7 comments

Once again, I greatly enjoyed this year's ESRI Federal User Conference - I was able to make it to several sessions Thursday and Friday... Perhaps will post more on this, as time permits.


As he has done before, Jack Dangermond solicited feedback and questions in his FedUC wrapup following Recovery Accountability and Transparency Board chairman Earl E. Devaney's excellent talk - I was happy to pitch in and asked him my own question about ESRI's vision for volunteered geographic information / crowdsourcing / participatory GIS...

This got Jack Dangermond excited, it seems he has been thinking about the concept, though even with his response, we need to get beyond the initial technical hurdles - and in talking to some other ESRI folks during and after the conference, I am happy to hear that some other ESRI staffers are thinking about this as well.

The thing is this: It's not really about ESRI tools importing OpenStreetMap, GeoRSS or Twitter feeds, and so on. Mere import and merge with your own data is really just a tiny part of it.

The true power of VGI is in its dynamic nature - for example, in the case of Haiti response, there were dozens of volunteers all providing concurrent updates, imports and edits to OpenStreetMap, as well as hundreds consuming the map data. The state of the map changed, sometimes radically, from hour to hour, and often even from minute to minute. As one example, an individual on the ground in Haiti sent an SMS message to Ushahidi with GPS coordinates for two locales where supplies could be airdropped or landed via improvised helicopter landing zone. The maps were blank in those two areas. Yet, within minutes, I and other OSM mappers pulled up the declassified DMA maps, DigitalGlobe and other imagery that had been donated by various providers, and sketched in the roads, trails, streams, buildings and other culture and planimetrics for those communities in need.

Leveraging those dynamic updates is one key piece to making the most of VGI. That means, going beyond import, to being able to consume and integrate the data on the fly.

The second piece is that VGI is not a one-way street. To use Haiti again as an example, dozens of disparate agencies are all using OpenStreetMap - several are in turn actively contributing back as well. Each is then building up on the work of the others, and the efforts of each resource leverages successive investments of the prior effort. This is particularly useful for resource-constrained organizations and volunteer efforts. As an example of this, as a member of Engineers Without Borders, I have been trying to promote adoption of OpenStreetMap for mapping efforts - e.g. one effort providing potable water can then dovetail into another organization's efforts to do health assessments, and so on.

As to cultural and organizational resistance to crowdsourcing, accuracy and reliability - that can be handled via record-level metadata. The double-edged sword of OpenStreetMap is in its use of key/value pairs for attribute data, as opposed to rigidly structured tables and columns, which on one hand can lead to folksonomies with inconsistent tagging, but on the other hand can handle rapid, flexible, ad-hoc changes to accomodate new needs, as well as allow complex representations via a collection of tags - which in turn can then also be cross-walked to other agencies' data models for interaction and ETL. While OpenStreetMap, on its' face, leaves much up to the individual contributors, best practices can and should be implemented. All edits are tracked in OpenStreetMap, which provides some basic metadata as to who and when, however more robust means of reverting adverse changes would be useful. Similarly, best practices are generally communicated via wiki, such as adding tags for source (e.g. digitized from DigitalGlobe imagery, with date). One of my comments in my followup to Jack Dangermond was that some of the governance/user guidance can be put directly into the tools, such as via JOSM, Merkaartor or Potlatch presets and templates.

Some may push back and suggest that shared community platforms like OpenStreetMap lack accuracy or reliability. The beauty of it is that if you don't like it, you can fix it.

And another goal... Dynamic integration of community platforms like OpenStreetMap beyond just base mapping and visualization, to be incorporated into modeling and analysis, via crosswalks and semantic interoperability.

I am happy to see this is something of interest to Jack Dangermond and ESRI, and hope that the bidirectionality and dynamic nature of VGI are fully embraced down the road.

Live Blogging from FedUC, Day 1 (er... not really)

Posted by Dave Smith On 2/17/2010 06:51:00 PM 2 comments

This will be as close as I get to live blogging from the ESRI Federal User Conference, Day 1.


Problem is, I missed it.

Unfortunately, as my day went, I was tied up in a meeting in Baltimore and didn't get back to DC until the tail end of the reception.

So that should make this the shortest wrapup of FedUC Day 1.

I heard there was some tasty sushi at the reception, but by the time I arrived, it was already gone... However, having not had anything to eat since morning, I did inhale a handful of various hors d'ouvres and a Heineken...

To catch up, I did monitor a few assorted tidbits via Twitter throughout the day- some of these struck me:
  • Buzzword: VGI. Volunteered Geographic Information. a/k/a crowdsourcing. All pointing to the OpenStreetMap paradigm- Given the phenomenal success of OpenStreetMap in supporting base mapping for humanitarian efforts in Haiti, along with the fluid and adaptable nature of its key/value pair model, which was utilized for tagging crisis-related features, such as landslides, collapsed buildings, road obstacles, refugee camps and so on - how can one NOT talk about OpenStreetMap? Great to hear about OpenStreetMap being used in ArcGIS. But... the pundit in me asks, how transparent is this? Is it "importing OSM data"? That's great for many applications. But what of emergency response applications? Given the dynamic nature of OpenStreetMap, "import" might not cut it. How about direct, native OSM support? That I need to investigate further. But then, comes the other, far more important piece of VGI - the participatory piece? Can/will ArcGIS 10 support direct editing of OSM? And outside of OSM, how much robust ArcGIS 10 capability for participatory GIS exists right out of the box?

  • Offshoot: Citizen-centric science. I understand Audubon demonstrated eBird - interesting - a year ago, I contributed a proof-of-concept to the eBio conference in London, demonstrating harnessing social media such as Twitter to allow citizen science participation to allow folks to record observations of various species in the wild, toward such ends as assessing biodiversity, invasive species, and so on. I tied this in with web services such as GBIF's lookup services to, for example, translate between common names and scientific name, and so on. This, in turn, tied in to other sources such as Flickr, and combined, wherever possible, with available geographic information, for providing feeds and display in, for example Bing Maps or other platforms. Great to see this coming along... I'd be interested in looking at the Audubon effort more closely, along with further exploring the model for vetting and validating inputs.

  • The Cloud... Evidently a big focus on cloud hosting, the new partnership between ESRI and Amazon, and "rent ArcGIS Server by the hour". I think this is an interesting model and have been harping on this for years. Geodata services hosting at this point is nothing esoteric, and could/should essentially be commoditized *cough* GeoServer *cough*. But... where the less-well-charted and more-interesting territory still lies is not in just serving up data. ArcGIS Server is frankly often too much tool to waste on just serving up map layers and tiles. Where I see the opportunity for ArcGIS server is in true ANALYSIS, MODELING and so on. Web-based geoanalytical services and geoprocessing services. We need a good model, and some good strategic thinking in the community of how the long-range picture of all of that will look.

  • ArcGIS for iPhone... That was another feature of ArcGIS 10 noted by some who attended the Plenary... Sounds great, but... I want to know more about how it operates, and more importantly, how customizable, configurable, how many features and functionalities it supports. Hopefully someone can shed some light there...
Even for showing up at the last minute, I briefly ran into @jfrancica, @donatsafe, @MikeHardy and @sturich from Pen Bay Media and got to say hi to them, as well as various other friends - I know there are plenty more friends, tweeple and geobloggers in town this week - I also did see @pbissett and @cageyjames from afar - on locating the @weogeo booth, it still had a few people milling about, so I unfortunately didn't get to chat...

More to follow... Tomorrow, my luck should be better, and I plan to be able to stay for the whole day - and for #geoglobaldomination afterward. Hopefully others will be posting their recaps, observations and scuttlebutt as well... At this point, the handful of hors d'oevres and beer in my otherwise empty tummy are looking for company.

ESRI Federal User Conference 2010

Posted by Dave Smith On 2/15/2010 09:15:00 AM 0 comments

I am looking forward to spending this week in Washington DC, despite any further threats of snow (1"-3" expected tonight)... Living in Northeastern Pennsylvania, 1 to 3 inches of snow is no big thing. But the 30+ that hit the Washington area last week are astounding. Hopefully, much of the prior accumulation is now under control.


Despite the snow, I laugh in the face of the snow and am still anticipating a decent turnout at the 2010 ESRI Federal User Conference (FedUC). At present, it sounds like a lot of my own federal colleagues are still planning on attending, as are various friends, geobloggers, geotweeple and so on... It has rapidly become the east-coast version of the San Diego ESRI International User Conference, with solid attendance not just by feds, but by a wide variety of others as well. The agenda, again a mix of technical GIS topics and where GIS is being used in a wide variety of business domains, along with a collection of special interest group meetings. I will generally be following an environmental science track, along with a few excursions into other areas.

It is being held once again at the Walter E. Washington Convention Center in Washington, DC, February 17-19th, although I will be arriving today, for various meetings around Washington today and tomorrow, and a few throughout the FedUC as well. Free for feds to attend, relatively cheap for others...

I hope to be able to post here and there from the FedUC, depending on connectivity and power availability (my workhorse laptop no longer holds a charge as it used to) - as well as the periodic blips from Twitter...

And if you are in the DC area, but are NOT attending the ESRI Federal User Conference, then here's a definite to keep an eye out for: #geoglobaldomination - essentially, just an ad-hoc, twitter-organized, vendor-neutral, platform-agnostic gathering of geospatial folks getting together over a few beers to discuss esoterics and idiosyncrasies of the geospatial business...

It will be fun!

Pennsylvania Geospatial Coordination Council

Posted by Dave Smith On 1/18/2010 09:09:00 PM 1 comments

Last week, I attended a discussion session sponsored by the Pennsylvania Mapping and GIS Consortium (PaMAGIC) and facilitated by John Palatiello, regarding standing up a Geospatial Coordination Council as a formal governmental advisory body for Pennsylvania. This is not a new concept, it is something that PaMAGIC has been pursuing for several years, it has been discussed several times prior at the PA GIS Conference, and has been introduced in the legislature several times, in various incarnations. It was last introduced in 2007-2008 by Representative Russ Fairchild as House Bill 1304.

The discussion session was held at the Pennsylvania State Association of Township Supervisors' office just outside Harrisburg, and roughly 60 people attended in person and by phone - an excellent cross section of state government, county and local government, private sector, utilities and others. I did not, however, see any representation by the Pennsylvania Society of Land Surveyors or Pennsylvania Society of Professional Engineers. Similarly, I do not see a spot designated on the proposed Council for these organizations.

I have posted two documents - one is a summary of the meeting, the second is the newest version of the proposed bill - the GIS community is invited to provide comment, by the end of this week, to PaMAGIC President Glenn McNichol, gmcnichol -at- dvrpc.org.


What does give me more optimism in this iteration is that there appears to be additional championing of this cause coming from the Governor's office. It is anticipated that Rep. Fairchild will be introducing the bill again, and I am hoping that the geospatial community will be able to rally around the effort, hopefully PaMAGIC will spearhead a good communications campaign with John Palatiello's help.

Mapping for Haiti Earthquake Response

Posted by Dave Smith On 1/18/2010 04:30:00 PM 2 comments


For the last few days I have been working on efforts to help Haiti earthquake response - part of this is mapping Haiti via OpenStreetMap. Currently, useful maps of Haiti are few and far between, in some case there are detailed maps for parts of the country and cities like Jacmel, but which are print maps (no digital edition exists), in many cases many decades old and outdated, in other instances the only maps which exist are small-scale, with limited detail. Commercial map platforms like Bing Maps, Google Maps and Yahoo vary greatly in their detail as well, and update cycles are slow - however, here, OpenStreetMap has been able to rapidly respond and provide very speedy and robust updates, to include capturing data about collapsed buildings, and so on.

Mikel Maron captures the speed and effectiveness of OpenStreetMap with these two screenshots of Port-Au-Prince, just before the earthquake and just after:

before:


after:


I would encourage any others who have time to contribute to get involved in this effort as well - editing can be done directly in OpenStreetMap via the 'edit' tab, which opens a web-based tool called Potlatch - or, a number of other tools are available as well. (You will need to register an account in order to edit - feel free to connect with me there as well - http://www.openstreetmap.org/user/Dave%20Smith). The OpenStreetMap WikiProject Haiti page provides a lot of good information and frequent updates. A number of data sources have been assembled, together with fresh post-earthquake imagery generously donated by companies like DigitalGlobe.

For folks who are interested in more robust tools, I primarily use Merkaartor which runs best on Windows platforms - others favor JOSM which is Java-based and runs on any platform supporting Java.

With either of these, you should be able to use the download function to navigate to an area of Haiti (select a relatively small area) and then download the OSM data

Some have wrestled with getting the various available imagery and map services to work properly in JOSM and Merkaartor - they are both a bit clunky about accessing WMS servers - I can offer some of my tips gleaned from a little debugging using Fiddler2: For Merkaartor, use Tools -> WMS Servers Editor and create a new entry with the following URL: http://maps.geography.uc.edu/cgi-bin/mapserv?map=/home/cgn/public_html/maps/mapfiles/haiti.map&version=1.1.0&SERVICE=WMS&REQUEST=GetMap? and show capabilities to access the DigitalGlobe CrisisEvent imagery (DG_crisis_event_service), select EPSG:4326 (only option available) and image/png and you should be able to go from there. If there is no image layer showing, go to Layers -> Add new image layer, and then right-click on the newly added image layer and select your newly-added WMS server for DigitalGlobe.

Digitization tasks are fairly intuitive - tools allow you to draw points, lines and polygons, as well as to create relations which allow multiple entities to be grouped together - however the absolute key to success is in proper use of tags to provide attribute values for any entities being created. These are generally intuitive as well, e.g. tag of highway, value of residential to turn a generic line into a residential road. It also helps to look at tagging of existing features, and to familiarize yourself with the features list - the OpenStreetMap wiki is searchable and quite useful. Finally, as terms of using the DigitalGlobe imagery, any data entered using their imagery should also be tagged with source=digitalglobe.

One of the other projects I have been working on is development of a set of tags that would be useful for emergency responders, relief and aid efforts - for this, I have started a wiki page http://wiki.openstreetmap.org/index.php?title=Humanitarian_OSM_Tags- any comments and thoughts can be added to the wiki page.


--- Update: Jan 20 ---

To update, I wanted to also include reference to the video tutorial that Kate Chapman put together on mapping for Haiti, to help demystify the tools:


Thanks to Kate and all the other great folks contributing to the Haiti mapping efforts...

The "Antiquated" County Surveyor?

Posted by Dave Smith On 1/07/2010 04:51:00 PM 5 comments

In Pennsylvania, the position of County Surveyor was abolished decades ago. Elsewhere, some states do still have County Surveyors - yet where still in use, it has in some instances become politicized, or may have duties which are either murky, inconsistent from one place to another, or even a position without any duties. Many consider the position "antiquated".


I, however, think that perhaps the notion of County Surveyor needs to be revisited. Not just in a traditional sense of basic survey support functions, such as are performed by County Engineers, but also toward sound boundary data stewardship.

For example, many of my colleagues have struggled with finding accurate information about municipal boundaries, corners, information on annexations, and so on. These need sound stewardship. Here in Pennsylvania I have heard plenty of stories about vital records associated with municipal boundaries getting lost, falling into disrepair, and so on - without any sound stewardship or curation. I have heard plenty of stories about disputes and nebulous municipal boundary locations, with disparities into hundreds of feet, which in turn can impact property owners and a whole host of administrative bodies, with regard to taxation, school districts and so on.

Further, with the emergence of GIS for tax assessment, there is also a role for county surveyors - in collaboratively developing a robust cadastral framework together with the county tax assessor, county GIS department, municipal counterparts, private sector surveyors and engineers, and so on.

Here, the County Surveyor could be responsible for reviewing submitted plans, for working with other local surveyors toward getting surveyed parcels, subdivisions, rights-of-ways and so on tied to a consistent system of monumentation, via GPS and other means, and so on, tying deeds and land records data to GIS data - toward iteratively refining the cadastre toward providing ever-increasing degrees of accuracy and reliability. Similarly, the County Surveyor could work with his counterparts in adjoining counties collaboratively toward building and bolstering regional, and ultimately statewide partnerships such as PAMAP here in Pennsylvania, which was a collaborative effort toward leveraging investments for obtaining aerial imagery - similar approaches can be taken toward gathering a host of other core datasets and mapping, such as LIDAR data, hydrology, roads, utilities, buildings and structures, and so on.

This model exists in various forms, to varying degrees, in various locales - however generally in far more of a disconnected and ad-hoc fashion, with varying degrees of "antiquated" holdovers in culture, and varying degrees of forward-looking approach, such as those I advocate here. As to whether it should be an elected, appointed or other position - that's, however another matter which I don't particularly care to get into today.

I for one think it's worth looking at again...

(Photo: Benjamin W. Harrison, County Surveyor of Hamilton County, Ohio 1892-1902)

Search