-
Paper 100 - Session title: Open Data and Tools Continuation
12:00 rasdaman: the Spatio-Temporal Datacube Reference Implementation
Baumann, Peter (1,2); Misev, Dimitar (1,2); Merticariu, Vlad (1,2) 1: Jacobs University, Germany; 2: rasdaman GmbH, Germany
Show abstract
Datacubes increasingly are accepted as a promising paradigm that simplifies user access and allows for better scaling server architectures. By aggregating all scenes pertaining to a particular satellite instrument over space and time, a single multi-dimensional object is created which can be sliced and diced, aggregated and combined through open standards interfaces.
One particularly promising technology for implementing datacubes are Array Databases, a field coined by the open-source rasdaman ("raster data manager") array engine. Based on a formal data model, the rasdaman query language enables declarative queries (which typically are generated through visual clients, transparent to the user). This language forms the blueprint for the OGC WCPS geo datacube query language standard as well as the ISO SQL MDA ("Multi-Dimensional Arrays") candidate standard. Today, rasdaman is both OGC and INSPIRE official reference implementation for the WCS/WCPS datacube standards. Operational services exist on dozens of Terabytes of Earth and Planetary remote sensing data. In Fall 2017, a rasdaman instance will go on board the ESA OPS-SAT satellite, effectively transforming it into an on-board, queryable datacube processor.
We present the open-source rasdaman datacube tool, its concepts, architecture, as well as sample services using it, including newly formed SMEs grounding their service model on the use of rasdaman.
Presentation
[Authors] [ Overview programme]
-
Paper 146 - Session title: Open Data and Tools Continuation
12:45 SIMOcean: Maritime Open Data and Services Platform for Portuguese Institutions
Almeida, Nuno Miguel (1); Grosso, Nuno (1); Deus, Ricardo (2); Oliveira, Paulo (2); Almeida, Sara (3); Alves, Margarida (3) 1: DEIMOS, Portugal; 2: IPMA, Portugal; 3: Instituto Hidrográfico, Portugal
Show abstract
The existence of an integrated management of Portuguese marine environment is crucial to monitor a wide range of interdependent domains. A system like this assimilates data and information from different thematic areas, ranging from ocean and atmosphere state variables to higher level datasets describing human activities and related environmental, social and economic impacts.
These datasets are collected by a wide number of public and private institutions with very diverse purposes leading to dataset duplication, inexistence of common data and metadata standards across organizations, and the propagation of closed information systems with different implementation solutions. This lack of coordination and visibility hinders the marine management, monitoring and vigilance capabilities, not only by making it more difficult to access, or even be aware of, the existence of certain datasets, but also by minimizing the ability to create added value products or services through dataset integration from different sources. The adoption of an Open Data philosophy will bring significant benefits by reducing the cost of information exchange and data integration, promoting the extensive use of this data.
SIMOcean (System for Integrated Monitoring of the Ocean), co-funded by the EEA Grants Programme, is integrated in the initiative of the Portuguese Government to develop a set of coordinated systems providing access to national marine data. These systems aim to improve the Portuguese marine management capabilities, aggregating different data, including specific human activities datasets (vessel traffic, fishing records), and environment variables. Those datasets, currently scattered among different departments of the Portuguese Meteorological (IPMA) and the Navy's Hydrographic (IH) Institutes, are brought together in the SIMOcean system that exploit this data through the following flagship services: 1) Characterisation of Fishing Areas; 2) Wave Alerts for Sea Ports; and 3) Diagnostic of Meteo-Oceanographic Fields. The specifications of these services were driven by end users such as Civil Protection Authorities, Port Authorities and Fishing Associations, where these new products are expected to create a significant positive impact in their operations.
SIMOcean is based on open source GIS interoperable solutions, compliant with OGC standards and INSPIRE directive. The Catalogue solution (based on ckan) uses a INSPIRE compliant metadata profile for marine environment developed by SNIMAR project, the guidelines provided by the directive 2013/37/EU and the Goldbook provided by the European Data portal. The system is deployed on a scalable infrastructure, providing authorised entities a single access point for data catalogue, visualisation, processing and deployment of value added services. It will be presented the challenges faced during the different phases of the project through different perspectives, Data Providers, System Integrators and End Users showing what was needed to put in place to have an attractive product for the community.
Presentation
[Authors] [ Overview programme]
-
Paper 159 - Session title: Open Data and Tools Continuation
11:45 Crowd-driven Tools for the Calibration and Validation of Earth Observation Products
Moorthy, Inian (1); See, Linda (1); Fritz, Steffen (1); McCallum, Ian (1); Perger, Christoph (1); Duerauer, Martina (1); Dresel, Christopher (1); Sturn, Tobias (1); Karner, Mathias (1); Schepaschenko, Dmitry (1); Lesiv, Myroslava (1); Danylo, Olha (1); Laso Bayas, Juan Carlos (1); Salk, Carl (1); Maus, Victor (1); Fraisl, Dilek (1); Domian, Dahlia (1); Mathieu, Pierre Philippe (2) 1: International Institute for Applied Systems Analysis, Laxenburg, Austria; 2: European Space Agency, ESRIN, Frascati, Italy
Show abstract
In recent years there has been a rapid diffusion in open access Earth Observation (EO) data available at global scales to help scientists address planetary challenges including climate change, food security and disaster management. For example, since 2016 the European Space Agency (ESA), via its Sentinel-2 satellites, has been providing frequent (5 day repeat cycle) and fine-grained (10 meter resolution) optical imagery for open and public use. As such, the EO community is faced with the need to design methods for transforming this abundance of EO data into well-validated environmental monitoring products. To help facilitate the training and validation of these products (i.e. land cover, land use), several crowd-driven tools that engage stakeholders (within and outside the scientific community) in various tasks, including satellite image interpretation, and online interactive mapping, have been developed. This paper will highlight the new results and potential of a series of such tools developed at the International Institute for Applied Systems Analysis (IIASA), namely the Geo-Wiki engagement platform, the LACO-Wiki validation tool, and Picture Pile, a mobile application for rapid image assessment and change detection. Through various thematic data collection campaigns, these tools have helped to collect citizen-observed information to improve global maps of cropland and agricultural field size, to validate various land cover products and to create post natural disaster damage assessment maps. Furthermore, Picture Pile is designed as a generic and flexible tool that is customizable to many different domains and research avenues that require interpreted satellite images as a data resource. Such tools, in combination with the recent emergence of Citizen Observatories (i.e. LandSense, GROW, GroundTruth 2.0, SCENT funded by Horizon2020), present clear opportunities to integrate citizen-driven observations with established authoritative data sources to further extend GEOSS and Copernicus capacities, and support comprehensive environmental monitoring systems. In addition, these applications have considerable potential in lowering expenditure costs on in-situ data collection and current calibration/validation approaches within the processing chain of environmental monitoring activities both within and beyond Europe.
Presentation
[Authors] [ Overview programme]
-
Paper 164 - Session title: Open Data and Tools Continuation
11:30 Open Data, Open Protocols and Formats, Open-Source Software: WebWorldWind
Voumard, Yann (1); Sacramento, Paulo (1); Collins, Paul David (2); Hogan, Patrick (2) 1: Solenix Deutschland GmbH, Germany; 2: National Aeronautics and Space Administration (NASA), USA
Show abstract
WebWorldWind is an open-source software library for building 3D virtual globes on the web. The joint development team from ESA and NASA working on the project strives to make open data easily accessible and manageable in a 3D environment. For achieving this goal, the library relies heavily on highly-recognised open protocols, such as those standardised and promoted by the Open Geospatial Consortium (WMS/WMTS, WFS, WCS, OpenSearch for Earth Observation, etc.), and formats (Shapefiles, JPEG, PNG, GeoTIFF, GeoJSON, KML, Collada, etc.) for bringing data to the globe and interpreting them.
However, implementing the right protocols and formats is only the first step towards truly supporting a wider community in the development of applications using open data. The application developers need access to the data in a meaningful and customisable way. These principles are at the core of the architecture of WebWorldWind both via its public interfaces and its extensibility.
The open-source nature of the project is, in itself, a contribution to an application development landscape based on openness. In fact, interested developers can get invaluable insight into the workings of the framework simply by examining its code in detail. This is useful not only to understand the concepts and how to best leverage the framework, but also to contribute to its evolution and improvement over time.
This presentation will first review some of the architectural concepts of WebWorldWind in support of open data and protocols. The current state of the art in terms of WebWorldWind features will then be presented. Concrete applications developed by the European community will come to exemplify the applicability of the developed concepts. Finally, a roadmap of improvements and new functionality to be implemented will be shown as a means to gather feedback from the community.
Presentation
[Authors] [ Overview programme]
-
Paper 179 - Session title: Open Data and Tools Continuation
12:15 Geo-Scientific Platform as a Service - Tools and Solutions for Efficient Access to and Analysis of Oceanographic Data
Hansen, Morten Wergeland; Korosov, Anton; Vines, Aleksander Nansen Environmental and Remote Sensing Center, Norway
Show abstract
Existing international and Norwegian infrastructure projects, e.g., ESA GlobCurrent, NRC NorDataNet, NRC NM C and NRC NORMAP, provide open data access through the OPeNDAP protocol following the conventions for CF (Climate and Forecast) metadata, designed to promote the processing and sharing of files created with the NetCDF application programming interface (API). This approach is now also being implemented in the Norwegian Sentinel Data Hub (satellittdata.no) to provide satellite EO data to the user community. Simultaneously with providing simplified and unified data access, these projects also seek to establish and use common standards for use and discovery metadata allowing development of standardized tools for data search and (subset) web streaming to perform actual scientific analysis.
A combination of software tools and actual data access, which we call the Geo-Scientific Platform as a Service (SPaaS), takes advantage of these opportunities to harmonize and streamline the search, retrieval and analysis of integrated satellite and auxiliary observations of the oceans in a seamless system. The core part of the Geo-SPaaS is a metadata catalog to store granular metadata describing the structure, location and content of available satellite, model, and in situ datasets. Data analysis tools include software for visualization, interactive in-depth analysis, and server-based processing chains. The Geo-SPaaS components are integrated in virtual machines (e.g., VirtualBox, VMware), of which provisioning and deployment are automatized using existing state-of the-art open-source tools (e.g., Vagrant, Ansible, Git, Python). The open-source code for scientific tools and virtual machine configurations is available on GitHub (https://github.com/nansencenter/), and is coupled to an online continuous integration system (Travis CI).
The Geo-SPaaS enables researchers to more quickly develop and test scientific algorithms, which then can be operationalized on server systems (Geo-SPaaS nodes).
Presentation
[Authors] [ Overview programme]
-
Paper 180 - Session title: Open Data and Tools Continuation
11:15 dtwSat: An R Package for Land Cover Classification Using Satellite Image Time Series
Maus, Victor International Institute for Applied Systems Analysis, Austria
Show abstract
Open access to satellite data has boosted the development of new approaches to quantify and understand Earth's changes. The large spatiotemporal availability of satellite imagery, for example, has improved our capability to map and monitor land use and land cover changes over vast areas. Given the open availability of large image data sets, the Earth Observation community would get much benefit from methods that are openly available, reproducible and comparable. This paper presents the R package dtwSat, which provides an implementation of the Time-Weighted Dynamic Time Warping (TWDTW) method for land cover mapping using sequences of multi-band satellite images. Methods based on Dynamic Time Warping (DTW) are suitable to handle irregularly sampled and out-of-phase time series, which is frequently the case of those from remote sensing. TWDTW algorithm has achieved significant results using MODIS, Landsat, and Sentinel-2 time series to classify natural vegetation and crop types in different regions. Using existing R packages as building blocks dtwSat supports the full cycle of land cover classification using satellite time series, ranging from selecting temporal patterns to visualizing and assessing the results. To handle the satellite images, dtwSat uses well-known data structures from the R package raster, which offers the option to work with large raster data sets stored on disk instead of loading into memory (RAM) at once. The current version of the dtwSat package provides pixel-based time series classification, i.e., each time series is processed independently from each other, and therefore, the code is easily parallelizable. dtwSat is open source and distributed under a GNU General Public License GPL (≥ 2). A binary version is available from the Comprehensive R Archive Network (https://cran.r-project.org/web/packages/dtwSat) and the development version from GitHub (https://github.com/vwmaus/dtwSat). Future versions of the package envisage new features to reduce border effects, increase spatial homogeneity (i.e., reduce the called 'salt and pepper effect') and improve the temporal consistency of land cover transitions. dtwSat makes it straightforward to apply and compare the TWDTW approach with other methods, contributing to rapid advance automated and semi-automated methods to analyze satellite time series.
Presentation
[Authors] [ Overview programme]
-
Paper 181 - Session title: Open Data and Tools Continuation
12:30 Foreshore Assessment with Space Technology (FAST): Services To Support Nature-Based Solutions For Coastal Flood And Erosion Risk Reduction
Morris, Edward Peter (1); Peralta, Gloria (1); Dijkstra, Jasper (2); Evans, Ben (3); Oteman, Bas (4); van der Meulen, Myra (2); Scrieciu, Albert (5); Hendriksen, Gerrit (2); Gomez-Enri, Jesus (1); Benevante, Javier (1); Smith, Geoff (6); Bouma, Tjeerd (4); Stanica, Adrian (5); Möller, Iris (3); van der Wal, Daphne (4); van Wesenbeeck, Bregje (2); de Vries, Mindert (2) 1: Universidad de Cádiz, Spain; 2: Deltares, Netherlands; 3: University of Cambridge, UK; 4: Royal Netherlands Institute for Sea Research (NIOZ), Netherlands; 5: National Institute for Research and Development of Marine Geology and Geoecology (GeoEcoMar), Romania; 6: Spectro-natura, UK
Show abstract
The FAST project (EU-FP-SPACE 607131, www.fast-space-project.eu) has developed the MI-SAFE package, which provides Earth Observation (EO) and Open Source (OS) modeling services to support Nature-Based Solutions (NBS) for coastal flood and erosion risk reduction. Developed by an international consortium of experts and relying on the European Copernicus Program, MI-SAFE aims to deliver smart, data-driven services that will facilitate the widespread use of ecosystem-engineering concepts in coastal protection, improving cost-efficiency and the well-being of coastal communities.
The unique features of MI-SAFE include; a combination of global coastal coverage and high-resolution local analysis that can provide flood-hazard related parameters in data-sparse regions, a transparent and verifiable scientific basis implemented using Open Source (OS) tools supported by the Open Earth software community, automated coupling of EO, sea-level, wave and vegetation modeling, delivery of data products as Open Geospatial Consortium (OGC) data streams, and a versatile, modular structure with the capacity to include advanced functionality driven by specific-user requirements.
The package is made up of three key components: (1) The MI-SAFE viewer (http://fast.openearth.eu/), a user-friendly on-line viewer that provides easy access to data products and services; (2) OGC data streams, a web catalog service that provides Open Access (OA) geospatial data layers; and (3) OS Modeling, using a specifically calibrated version of XBEACH that includes the effects of coastal vegetation on wave attenuation.
Services are provided at three different levels, which represent differences in the spatial scale, quality of input data and predictions, and complexity of the modeling approach: Educational, an overview of potential wave attenuation by vegetation at the global scale; Expert, high-resolution data with a specifically calibrated model at case-study sites; and Advanced, on-demand, tailor-made solutions potentially linked to other domains (such as inundation and damage modeling) to be used in rapid assessment and design settings. Educational and Expert services are OA, whereas Advanced services (including consultation, training, development of new functionalities, and insitu calibration/validation) are offered by the consortium on request; with the idea that these will offset the package's running costs, leading to self-sufficiency.
Here we present an overview of the MI-SAFE package and demonstrate some of its key functionalities, discuss scientific and technical aspects of sustainably meeting end-users requirements, and invite feedback from the audience about the future development of the package.
[Authors] [ Overview programme]
-
Paper 188 - Session title: Open Data and Tools Continuation
13:00 Elastic Search to Geo Search: Delivering web-based search tools for large volume, heterogeneous airborne, in-situ and satellite-based observations
Stephens, Ag (1); Smith, Richard (1); Garland, Wendy (1); Parton, Graham (1); Kershaw, Philip (2) 1: NCAS / Centre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory; 2: NCEO / Centre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory
Show abstract
We describe the development of web-based graphical map-based tools for interactive search and discovery of large volume heterogenous airborne, in-situ and satellite-based observations along with the underpinning technologies to index and catalogue the associated metadata. This work is in the context of a multi-petabyte archive of environmental science datasets managed by STFC's Centre for Environmental Data Analysis (CEDA). With data holdings of over 250 million files it is essential that CEDA provides an effective set of tools to facilitate data discovery and access.
Requirements for a web-based map-interface originated with the EUropean Facility for Airborne Research (EUFAR2) project and UK Facility for Airborne Atmospheric Measurements (FAAM) and the need to expose the associated airborne in-situ and remote sensing research flight data more readily to the user community. The EFF indexes temporal, spatial, file-system and miscellaneous properties in a catalogue implemented using ElasticSearch. This is challenging given the variety of data types and formats, each requiring its own specific adaptor to parse metadata content and incorporate into the ElasticSearch index. The EFF front-end is a lightweight web-app in which Google Maps provides a map-interface and a thin JavaScript layer manages the interactions between the spatio-temporal search and Elasticsearch's RESTful API.
Building on the success of the EFF, CEDA has recently developed its Satellite Data Finder (SDF) tool. Whilst the EFF holds 10s of thousands of flight records the SDF needed to scale to millions of satellite scenes. Starting with the Sentinel missions, the fundamentals of the EFF were extended to increase the capability of the tool and in particular, revise the interface to effectively manage and present the large volume of data resulting from search queries. Additional filters were added to allow searching by satellite and usability testing was used to inform and improve the design making incremental improvements to deliver a more intuitive interface.
Scaling to these data volumes has also presented challenges with the back-end, ElasticSearch was scaled up to an 8-node virtual cluster (8 x 16GB RAM) to allow both test and production indexes to be accommodated with 100GB set aside for the storage of metadata indexes. In addition, indexing of product metadata has taken advantage of CEDA's JASMIN (jasmin.ac.uk) batch compute cluster, Lotus, to effectively scan large volumes of products in parallel.
The finished version of the SDF provides a simple web-tool that allows users to quickly narrow down their search to an area and period of interest and access the matching data. In the wider context, many environmental datasets, such as the ESA Sentinel missions, are publicly available and have the potential to be exploited across the academic, public and commercial sectors. By providing interactive web-search tools CEDA is increasing the usage and value of its data holdings. Looking forward, the challenge is now to integrate the rest of the CEDA archive under these interfaces.
Presentation
[Authors] [ Overview programme]
