TEP - Exploitation Platforms

Nowadays most EO services are still based on downloading source data to the local environment, before performing basic preprocessing (e.g. compositing, mosaicking, atmospheric correction) and starting with specific analyses relevant for the service. Structured availability of data within a cloud environment makes it possible to automate tasks and run them faster. However, this work-process is still either too slow or the required computing power/bandwidth too high to be scalable to regional or even global scales. GeoVille and Sinergise have joined forces to solve this problem by connecting a standard scripting environment with Sentinel Hub's web services enabling innovative EO value added services (EO-VAS). EO-VAS will create a scalable on-demand service, which actually will generate and deliver EO products to users worldwide in a cost-effective manner. To demonstrate the approach, we have designed an initial service allowing for multi-temporal land classification in an extremely efficient manner. It takes spectral indices, such as for example the maximum NDVI pixel value, from a multi-temporal image stack with multiple observations and creates clusters based on the spatio-temporal variability of the index. This allows investigating seasonal vegetation variations and to discriminate various land cover types, such as cropland (high variation of vegetation throughout the year), bare soil (no variation, low vegetation index value) and grassland/forest (low variation, high vegetation index value). More detailed and accurate classifications are possible by introducing additional indices, derived metrics and region-specific thresholds. The Sentinel Hub web service is used to create the spectral indices for any chosen area and season. This comes as an input to a clustering procedure. As there is practically no need to store the data of intermediate steps, the service can be provided in real-time. When a request comes, it can be globally run without large IT resources, limited only by "regional EO knowhow" with regard to different spectral index patterns in various regions. A side effect of this simplicity is the given flexibility - it is possible to change the "rule engine" and parameters and see the result in a matter of seconds. This opens the possibility to develop and evolve services in a very fast and efficient manner. The main technological novelty lies in a cost-effective approach, which uses resources only when somebody is asking for results, thus making the services more affordable. The functional novelty is expressed through its connecting characteristics - connecting end-user communities with developers, who can build their services on top of the initial tool-set and commercializing them through the platform without the need for large investments in processing and storage infrastructure or data processing developments.

ESA Hydrology Exploitation Platform (HEP) is a project aim at facilitating discover, access, process, visualisation and sharing of water information derived from Earth Observation data such as flood mapping, water level, water quality and the assimilation of these data into hydrological models. ESA HEP team will show the existing features and capabilities of this HEP through an on-line demonstration. ESA HEP services could be accessed through https://hydrology-tep.eo.esa.int/#!thematic.

The Forestry Thematic Exploitation Platform (Forestry TEP) has recently entered the pre-operational phase. This means that the principal functionalities are in place and a higher number of users can start using it. At the time of writing of the abstract, the database of interested organizations has more than fifty members from academia, research organizations, private sector, public administration, and NGO’s. The user pool is growing when marketing of the platform is now intensifying. User interest in Forestry TEP has exceeded all the expectations. The enthusiasm clearly shows that such one-stop-shop for Earth Observation services in forestry has a strong demand in the user community. The users are seeing the benefits of working on the cloud instead of having to download and install data and software on their own computing facilities that may be incapable to process the vast volumes of Copernicus data. Presently six thematic processing services are available on the Forestry TEP - for computing vegetation indices and for mapping of land and forest cover, biomass and change. Through a voting system available on the Forestry TEP web site, a user can affect the prioritization of future applications. In addition, a user can utilize all the features of the Sentinel Application Platform SNAP, Monteverdi/Orfeo toolbox, and an open source Geographic Information System QGIS. The user can search the data on the Forestry TEP Platform, switch to access remotely any of the above-mentioned software tools, process data using these tools, save the result to the platform, and continue processing there or download the result to his own computer. Relevant images from the search can be saved to a Data Basket where they can be found also for the future processing in another session. The user can upload his own reference data to train the land cover or biomass models or he can generate the data interactively with QGIS on the platform, for instance. Additionally, via a web-based developer interface, it is possible for service providers to develop their existing or completely new applications to be run on the platform. Applications and result products can be shared to designated users or openly to everybody. In the operational phase, there will be a possibility also to sell products and offer application software as a service on the platform. Already now, data privacy and security functionality is in place. The platform presently offers access to Sentinel-1, Sentinel-2, and Landsat-8 data. More data types, including commercial data will be introduced according to user interest. Users can search over a geographic region not only for the available satellite images but also for existing products that have been computed and shared by other users. The Forestry TEP is being developed by VTT Technical Research Centre of Finland as the coordinator, application and user specialist, CGI IT UK as the system developer and integrator, Science and Technology Facilities Council (STFC UK) as the principal data access and infrastructure provider, and Spacebel (BE) and Arbonaut (FI) as application and service experts. In addition to using the Climate and Environmental Monitoring from Space (CEMS) facility and cooperative ground segment of STFC, Sentinel data can experimentally be downloaded from the Earth Observation Innovative Platform Testbed Poland (IPT) facility. It is expected that for the future operational phase to start in 2018, the Copernicus Data and Information Access Services (DIAS) will offer a potential infrastructure solution for Forestry TEP. https://forestry-tep.eo.esa.int/

The volume and variety of Earth Observation data available for the polar regions is growing rapidly, providing the opportunity for ever more complex investigations and exploitation in support of polar communities. The European Space Agency (ESA) and other satellite operators have been at the forefront of collecting, analysing, processing and disseminating new data and information from EO satellites. With this increase in volume of data and range of uses, there are new challenges in order to fully utilise and exploit this capacity. ESA has established a series of Thematic Exploitation Platforms (TEP) which will provide the necessary collaborative environment which will deliver the resources and capabilities required for users’ exploitation work. The TEP projects address present challenges and opportunities in scientific data exploitation and operational applications by collocating data, processing capabilities and ICT infrastructure, providing a complete cloud based work-environment for users. The Polar TEP addresses the requirements and challenges of the diverse polar user community. This presentation will describe the Polar TEP concept and the range of potential uses it will support. We will present details of the working environment where users can bring their algorithms, applications and development activities directly to the data. It will also cover the rich set of polar themed EO and complimentary datasets, relevant toolboxes and processing capabilities, plus functionality to allow deployment of user defined workflows and processing environments. The PTEP is currently in a pre-operations phase and will increasingly be used by real users. We will report on progress made in providing platform access to early users. We will also outline the ongoing pilot project which will demonstrate the potential of Polar TEP to investigate current and future iceberg risk in Baffin Bay. The pilot project will integrate a diverse set of data, processors and models to allow users to investigate linkages between iceberg trajectories, changes in ice sheet velocity, glacier calving rates and ocean circulation. The integration of these components and toolsets will allow Polar TEP users to easily investigate questions about changing iceberg populations in support of regional climate change studies, evaluate risk assessments and inform infrastructure and ship routing decisions.

The upcoming suite of Sentinel satellites in combination with their free and open access data policy will open new perspectives for establishing a spatially and temporally detailed monitoring of the Earth’s surface. However, the capability to effectively and efficiently access, process, analyze and distribute the mass data streams from the Sentinels and high-level information products derived from them poses a key challenge. This is also true with respect to the necessity of flexibly adapting the processing and analysis procedures to new or changing user requirements and technical developments. Hence, the implementation of operational, modular and highly automated processing chains, embedded in powerful hard- and software environments and linked with effective distribution functionalities, is of central importance. This contribution introduces the TEP Urban platform that aims at the utilization of modern information technology functionalities and services to bridge the gap between the technology-driven EO sector and the information needs of environmental science, planning, and policy. Key components of the system are an open, web-based portal connected to distributed high-level computing infrastructures and providing key functionalities for i) high-performance data access and processing, ii) modular and generic state-of-the art pre-processing, analysis, and visualization, iii) customized development and sharing of algorithms, products and services, and iv) networking and communication. These services and functionalities are supposed to enable any interested user to easily exploit and generate thematic information on the status and development of the environment based on EO data and technologies. The TEP Urban platform is supposed to initiate a step change in the use of EO data by providing an open and participatory platform based on modern ICT technologies and services that enable any interested user to easily exploit and generate thematic information on the status and development of the built environment. So far more than 240 institutions from 41 countries have requested U-TEP data and system access.

Today, massive processing resources are often reserved to a very limited number of users mostly because of the human investment needed to start using the resources. Users will generally be required to: · Specify their hardware requirements and reserve the resources · Learn the APIs to access the system · modify, package or sometimes rewrite their prototype code in order to make it compatible with the system This not only raises the entry bar to start using the system, but it also tends to lock forever the software to a specific platform. In the frame of the Coastal Thematic Exploitation Platform (CTEP), we have developed a radically different approach based on the idea that one should be able to integrate a software on the platform with a few clicks. The only constraint on the software being that all output files should be placed in a specific folder, the exact same code can be run on a stand-alone linux machine and on the CTEP cluster. This makes integration much simpler and facilitates maintenance of the software. Our solution relies on a number of specific software tools: · An original python-based implementation of a WPS server (WISPY). WISPY is managing a dynamical database of processing services which can be created, modified or deleted on-the-fly. · A processor integration mechanism and user interface. The interface allows users to define the WPS service (name, abstract, input parameters) and upload the software packaged as archive file. The CTEP automatically creates a container with the software and records in the WPS database. The interface also allows the user to share the processing service with other users. · Built-in pre- and post-processing steps to retrieve input files from the data archive, store processing results in the user data storage area, and register georeferenced images as layers for visualization. These automatic steps avoids the need to learn specific APIs. In addition, effort has been made to facilitate the development: · The CTEP provides interactive remote desktops based on noVNC for development and prototyping · A large range of pre-installed python libraries for python developers · Systematic recording of execution logs on the cluster for debugging · On-line access to SNAP and QGIS for in-depth analysis of Sentinel or geospatial data. We will describe the technical solution and report on the first experience of the system on the integration of processing services for coastal environment monitoring. As expected, simple software can be integrated on the platform in one hour typically (including debugging time), with little if any learning curve required.

In line with the paradigm shift in Earth Observation of “Bringing the users to the data”, ESA provides collaborative, virtual work environments giving access to EO data and tools, processors, and ICT resources through coherent interfaces. These coherent interfaces are categorized thematically, tailored to the related user communities and named Thematic Exploitation Platforms (TEP). The Food Security Thematic Exploitation Platform (FS-TEP) is the youngest out of seven TEPs and is developed in an agile mode in close coordination with its users. It will provide a “one stop platform” for the extraction of information from EO data for services in the food security sector mainly in Europe & Africa, allowing both access to EO data and processing of these data sets. Thereby it will foster smart, data-intensive agricultural and aquacultural applications in the scientific, private and public domain. The FS-TEP builds on a large and heterogeneous user community, spanning from application developers in agriculture to aquaculture, from small-scale farmers to agricultural industry, from public science to the finance and insurance sectors, from local and national administration to international agencies. To meet the requirements of these groups, the FS-TEP will provide different frontend interfaces. Service pilots will demonstrate the platform’s ability to support agriculture and aquaculture with tailored EO based information services. The project team developing the FS-TEP and implementing pilot services during a 30 months period (started in April 2017) is led by Vista GmbH, Germany, supported by CGI Italy, VITO, Belgium, and Hatfield Consultants, Canada. It is funded by ESA under contract number 4000120074/17/I-EF.