Platforms

In the coming years, new EO missions will offer a large amount of new data useful for water resource management purposes. These data shall be accessed, processed, analysed and visualized for users and service providers but sometimes these users and providers do not want to invest ICT resources or do not have skills for handling these large volumes of data. The Hydrology Thematic Exploitation Platform (TEP) is an ESA project that aims at providing hydrology services such as flood mapping/forecasting, water quality and level monitoring, hydrology modeling and small water bodies mapping based on EO data. In addition to that, Hydrology TEP will also provide a collaborative framework where scientific users, river basin organisations and service providers could rapidly and easily access to a large number of EO data, integrate their own data and tools (in-situ data, socioeconomic data, analysis tools...) and process their processors (service prototypes, hydrological models, meteorological models) within a user-friendly environment. In summary, Hydrology TEP project wants to build a community and offer to this community an ensemble of services to facilitate and simplify their daily work.

The Forestry Thematic Exploitation Platform (F‑TEP) aims at improving information on forest resources for better forest management from economic and ecological viewpoints. Presently only a handful of countries have good information on their forest resources. Forest information is often uncertain or outdated due to insufficient staff and funding to acquire data on forests, due to limited computing resources, software tools, and limited expertise to utilize available data.

The European Copernicus program and other open data sources offer an unprecedented opportunity to acquire up-to-date information on forests from local to global extents. However, the huge data volumes require powerful computing resources and easy-to-use software before the data volumes can be refined to useful information.

We are building the F-TEP to offer anybody an opportunity to obtain relevant information on forest from any global location. The F-TEP is meant to serve actors from government administrators to scientists and commercial sector for which the F-TEP can become a global marketplace for their products.

F-TEP is organized as a web portal with functionalities meant from inexperienced to expert users. It offers an access to free Copernicus data, publicly available reference data, digital elevation models as well as to existing spatial products on forest resources. An access to commercial data and services will also be available. It will have scalable computing infrastructure and representative software for data value adding.

For the novice users the platform has push button type functionalities and for the scientists and other experts it offers an opportunity to upload own algorithms and reference data in addition to using versatile software available on the platform. As users grow with expertise and confidence and want to pursue more complex tasks, they can proceed to unlock a new range of functionality. With this model, F‑TEP can act as a learning platform as well.

The strategy for user support in F-TEP combines design for easiness of use, good textual and visual support material, strong support for communication and collaboration between peer users, and a responsive helpdesk.

The project, funded by the European Space Agency is one of the six parallel Thematic Exploitation Platform projects. It started in March 2015 and will last for 30 months. However, the purpose is to make the F-TEP a permanent service and continuously expand its capability to help forestry users. In the conference we are able to demonstrate an initial implementation of the F-TEP and the approach in two extensive-area demonstrations in Mexico and Finland.

The upcoming suite of Sentinel satellites in combination with their free and open access data policy will open new perspectives for establishing a spatially and temporally detailed monitoring of the Earth’s surface. The Sentinel fleet will provide a so-far unique coverage with Earth observation (EO) data and new possibilities with respect to the implementation of innovative methodologies, techniques and geo-information products and services. However, the capability to effectively and efficiently access, process, analyze and distribute the mass data streams from the Sentinels and high-level information products derived from them poses a key challenge. This is also true with respect to the necessity of flexibly adapting the processing and analysis procedures to new or changing user requirements and technical developments. Hence, the implementation of operational, modular and highly automated processing chains, embedded in powerful hard- and software environments and linked with effective distribution functionalities, is of central importance.

This contribution introduces the TEP Urban project that aims at the utilization of modern information technology functionalities and services to bridge the gap between the technology-driven EO sector and the information needs of environmental science, planning, and policy. Key components of such a system are currently developed in the TEP Urban project. This includes the implementation of an open, web-based platform employing distributed high-level computing infrastructures (Platform as a Service – PaaS) as well as providing key functionalities for i) high-performance access to thematic data (Information as a Service – InaaS), ii) modular and generic state-of-the art pre-processing, analysis, and visualization (Software as a Service – SaaS), iii) customized development and dissemination of algorithms, products and services, and iv) networking and communication. These services and functionalities are supposed to enable any interested user to easily exploit and generate thematic information on the status and development of the environment based on EO data and technologies.

The TEP Urban platform is supposed to initiate a step change in the use of EO data by providing an open and participatory platform based on modern ICT technologies and services that enables any interested user to easily exploit and generate thematic information on the status and development of the built environment.

ESA's Thematic Exploitation Platform (TEP) initiative explores a new concept for EO data exploitation. Instead of downloading the data, users are invited to perform processing remotely. Remote processing is especially interesting in view of the continuous increase of volume of available EO data. Providing processing resources closer to the data avoids classical bottlenecks associated to data processing, storage and especially data transfer capabilities.

The TEPs go a step further and provide users with an environment to develop and validate new applications, and to share these applications with other users. Applications can range from simple post-processing (graph generation, statistical filtering...) to complex fusion, extraction or classification algorithms. Sharing applications with other users brings about the benefits of collaborative science, supporting reproducible research, peer-reviewing and collaborative improvement, benchmarking and round-robins. It should also foster the creation of community of thematic users, increase the diffusion of EO data, and promote multi-disciplinary approaches.

While on demand remote procesing is not necessarily a new concept, most implementations are either too constrained or require programming against a specific API which is unfamiliar to many EO scientists. In fact there is also a wealth of pre-existing code which would be too complex, expensive or cumbersome to adapt. The solution proposed by the Coastal TEP, in addition to providing general building blocks allows users to provide their own processing software, while making their integration as simple and as efficient as possible, without the need to learn a specific new technology.

We will report on the implementation of our solution, and showcase the advantages (such as data locality, parallel computation, etc) of such a solution by running through a typical workflow, such as computing time series of average Sea Surface Temperatures and chlorophyll concentration over a given area of interest, as well as some custom processing examples.

Earth observations from satellites produce vast amounts of data and are playing an increasingly important role, especially with the new Sentinel missions, as a regular and reliable high-quality, free and open data source for scientific, public sector and increasingly for commercial activities. The latest developments in Information and Communication Technology (ICT) facilitate the handling of large volumes of data and, most importantly, have started to modify the expectations that organisations have on new services development and on support to Earth Observation (EO) data exploitation, now aiming at chains of value-creation where openness and accountability of data products play a key role. To achieve this, particular effort has to be dedicated to the assessment of solutions for the authoring and portability of EO processing algorithms to cloud infrastructures. Moving the processing to where the data is also implies to optimize the connectivity services of the data centres, with new discovery and processing methods.

The Geohazards Thematic Exploitation Platform (GEP) is an ESA funded R&D activity to exploit the benefits of these new techniques for large scale on demand and systematic processing of EO data in the Geohazards domain. It supports the geohazards community by creating an Exploitation Platform with new models of collaboration where data providers, users and processors produce and deliver scientific and commercial information products in the Cloud. The focus is on the geohazards domain but with wider implications covering the next generation Ground Segment capabilities, and new ICT solutions to maximize the exploitation of EO data from past and future missions. 

The GEP is creating an ecosystem of partnerships for data, applications and ICT resources. With already 30+ early adopters for a validation phase initiated in 2015 and continued over 2016, it defines a new paradigm for EO data exploitation and valorisation based on several open science principles, where users bring-in applications, domain experts bring processors, and ICT providers bring processing power, in order to scale-out the creation and spread of value-added products with scientific and/or commercial value. 

Since October 2015 the GEP includes six new partnerships that bring new applications and new end-users. Each Pilot on the Platform is focused on either integrating an application or running on-demand processing using applications available on the platform. The GEP has already secured funding to expand the user base, and will gradually reach a total of 60 users by 2017.

Within GEP, the Open Science practice stands on reproducible, citable, discoverable and shareable data, applications and results. The GEP community can make applications and research outputs available in a citable, shareable and reproducible manner. The Platform’s integration of Cloud services like Github, Zenodo and Figshare provides enhanced object, code and document libraries, so the community can upload files in a number of supported formats, and attribute them a Digital Object Identifier (DOI) used to uniquely identify a research asset. 

In this presentation, we will describe the GEP consortium effort to leverage the Open Science principles and deliver a significant contribution to the community objectives of the CEOS WG on Disasters.

Since more than 20 years, “Earth Observation” (EO) satellites developed or operated by ESA have provided a wealth of data. In the coming years, the Sentinel missions, along with the Copernicus Contributing Missions as well as Earth Explorers and other, Third Party missions will provide routine monitoring of our environment at the global scale, thereby delivering an unprecedented amount of data.

While the availability of the growing volume of environmental data from space represents a unique opportunity for science, general R&D, and applications, it also poses a major challenge to achieve its full potential in terms of data exploitation. Firstly, because the emergence of large volumes of data raises new issues in terms of discovery, access, exploitation, and visualization of “Big Data”, with profound implications on how users do “data-intensive” Earth Science. Secondly, because the inherent growing diversity and complexity of data and users, need to cooperate to make sense of a wealth of data of different nature (e.g. EO, in-situ, model), structure, format and error.

In this context, the University of Strasbourg supports the A²S 'Alsace Aval Sentinel' programme which aims at building an operative processing environment to maximize the exploitation of EO data from current and future missions. This initative is also part of the national data repository initiatives (THEIA Land, ForM@Ter) designed to foster the use of satellite images for the monitoring of land surfaces and the solid earth, through regional and thematic scientific competence centres.

This presentation focuses on the presentation of the distributed processing services which are currently being implemented on high performance computing facilities. The innovative processing methods being implemented consist of:

-       a chain for the quantification of earth surface deformation from several processes (tectonics, landslides, glaciers) and making use optical and SAR images

-       a chain for the detection and mapping of surface waters (water bodies and floods)

-       a chain for the quantification of urban (and more generally impermeable surfaces) developments.

-       on-demand image mining tools for the rapid detection of annual, seasonnal or event-based changes

The system is designed for both near-real time routine processing of S1 and S2 images within one day after acquisition, and on-demand processing of image archives (past and current satellite missions) for specific purposes, such as rapid mapping of natural disasters. The processing chains are currently designed for processing large areas of 1000 x 1000 km.

The scientific and industrial communities are being confronted with a strong increase of Earth Observation (EO) satellite missions and related data. This is in particular the case for the Atmospheric Sciences communities, with the upcoming Copernicus Sentinel-5 Precursor, Sentinel-4, -5 and -3, and ESA’s Earth Explorers scientific satellites ADM-Aeolus and EarthCARE.

The challenge is not only to manage the large volume of data generated by each mission / sensor, but to process and analyze the data streams. Creating synergies among the different datasets will be key to exploit the full potential of the available information. Integrating EO data with ground based observations and numerical models, is the basis for a new data exploitation paradigm which opens new research and commercial opportunities.

As a preparation activity supporting scientific data exploitation for Earth Explorer and Sentinel atmospheric missions, ESA is funding the technology study and prototype implementation of the “Technology and Atmospheric Mission Platform” (TAMP). Services and tools are developed along use cases defined with users from different scientific and operational fields and implemented according to their requirements to ensure acceptance of TAMP by the atmospheric community.

The TAMP test-bed environment offers data access, visualization, processing and analysis services for the “data triangle” consisting of (1) EO satellite products, (2) model data provided by chemical weather forecast, and (3) reference / validation data sets.

The implementation pursues the “virtual workspace” concept: all resources (data, processing, visualization, collaboration tools) are provided as “remote services”, accessible through a standard web browser, to avoid the download of big data volumes and for allowing utilization of provided infrastructure for computation, analysis and sharing of results.

This work presents the TAMP platform (concepts, implementation): use cases for (a) SO2 volcanic emissions monitoring, (b) global stratospheric Ozone, and (c) tropospheric aerosol from satellite, model and ground measurements (stations, lidar) are presented to demonstrate the effectiveness of the tool for users’ daily work. 

The interdisciplinary research consortium of the “GeoMultiSens” project focuses on developing an open source, scalable and modular Big Data system that combines data from different sensors and analyzes data in the petabyte range (1015 Byte). The most important modules are: (1) data acquisition, (2) pre-processing and homogenization, (3) storage, (4) analysis, and (5) visual exploration. The data acquisition module enables users to specify a region and time interval of interest, to identify the available remote sensing scenes in different data archives, to assess how these scenes are distributed in space and time, and to decide which scenes to use for a specific analysis. The homogenization module uses novel and state-of-the-art algorithms that combine the selected remote sensing scenes from different sensors into a common data set. The data storage module optimises storage and processing of petabytes of data in a parallel and failure tolerant manner. The core technology of the data storage module is XtreemFS (http://www.xtreemfs.org). The analysis module implements image classification and time series analysis algorithms. The visual exploration module supports users in assessing the analysis results. All modules are adapted to a map-reduce processing scheme to allow a very fast information retrieval and parallel computing within the processing system Flink (http://flink.apache.org). Finally, a Visual Analytics approach integrates the individual modules and provides a visual interface to each step in the analysis pipeline.

The Big Data system “GeoMultiSens” will store and process remotely sensed data from space-borne multispectral sensors of high and medium spatial resolution such as Sentinel-2, Landsat 5/7/8, Spot 1-6, ASTER, ALOS AVNIR-2 and RapidEye. Our poster presents the overall scientific concept of the Big Data system “GeoMultiSens” and technical details of the most important modules. We discuss scientific challenges of the Big Data system “GeoMultiSens” and present our ideas to address these scientific challenges. A short video demonstrating the challenges and contributions of GeoMultiSens can be found here: http://www.geomultisens.gfz-potsdam.de