Open Data and Tools

We present a customized version of GGIS (Quantum GIS) integrating OTB (Orfeo Toolbox) software that allows the processing of Sentinel-1 IW time series acquisitions in GRD products format. The processing is dedicated to radar non specialists that are concerned with vegetation monitoring such as Land cover estimation, or deforestation and forest degradation areas detection.

Compared to its optical counterpart, the community of Synthetic Aperture Radar (SAR) data users for land applications is still small. One major reason for that originates from the differences in the acquisition principle and the underlying physics of the imaging process. For non-experts, this results in difficulties of applying the correct processing steps as well as interpreting the non-intuitive backscatter image composites. On the other hand, the free and open access to Sentinel-1 data widened the community of interested users and paves the way for the integration of SAR data into operational monitoring systems. In order to ease the use and access to analysis-ready SAR data for wide-area land applications, the Food and Agriculture Organization of the United Nations develops the Open SAR Toolkit (OST) under the SEPAL project. OST includes fully automated pre-processing routines that are mainly build on top of the Sentinel Application Platform (SNAP) and other freely available open-source software such as GDAL, Orfeo Toolbox and Python. The simple and intuitive GUI is based on the R Shiny package and is accessed via a web-browser. This allows to employ OST also on cloud-platforms, as in the case of SEPAL (see abstract of Lindquist et al.). For the moment, supported data sets are the ALOS Kyoto & Carbon mosaics and Sentinel-1 products. The former is freely available for non-commercial use, and OST eases the access and preparation of the data tiles provided by JAXA. For Sentinel-1, data inventory and download routines, as well as a GRD to RTC processor allows for the rapid generation of radiometrically terrain corrected (RTC) imagery that is ready for subsequent analysis tasks such as land cover classification. More advanced and processing intensive data products, such as time-series and timescan imagery, can be easily produced as well, in a fully automatic manner. Ultimately, mosaicking generates seamless wide-area data sets. Alongside the processing routines, accompanying demos and capacity building material provide the user a gentle entry into the world of radar remote sensing for land applications and refer to a wealth of relevant literature for a more profound study of the subject. The presentation includes nationwide, wall-to-wall Sentinel-1 timescan and time-series mosaics, that have been combined with optical and ALOS K&C data for biomass and land cover mapping.

Hardware and software requirements for advanced Interferometric Synthetic Aperture Radar (InSAR) processing of long stacks of satellite SAR imagery to generate land deformation time series are notably increasing in the current era of ‘big SAR data’ (e.g. Cigna 2015). The volume and length of interferometric SAR data stacks are growing exponentially (e.g. every 6 days a new Sentinel-1 IW scene is acquired for all land areas of Europe, along each pass, ascending and descending), and with them InSAR processing workloads and demands. A vast component of SAR data handling, initial manipulation and specialised InSAR processing can be delegated to remote systems and virtual environments. For instance, ESA’s Grid-Processing On Demand (G-POD; http://gpod.eo.esa.int/) platform for EO applications offers an environment where SAR data can be processed using high-performance and sizeable computing resources. In this paper, a number of InSAR processing trials that were carried out using G-POD and its Land Information Service ‘InSAR SBAS’ will be presented. The latter is based on the automated Parallel Small BAseline Subset (P-SBAS) processing chain developed at CNR-IREA (Casu et al. 2014), and allows both the generation of interferograms and the multi-temporal analysis with extraction of land deformation time series for coherent targets. Hundreds of SAR scenes at medium resolution acquired by ESA’s ERS-1/2 and ENVISAT missions and made freely available through ESA’s Virtual Archive 4 (http://eo-virtual-archive4.esa.int/) were used to run the trials. Processing with P-SBAS was conducted through the user-friendly G-POD web portal, which allowed selection of input data, setting of processing parameters, thresholds and options, and effective monitoring of the full processing chain from remote, as well as downloading of the generated results for subsequent visualisation in Google Earth and uploading into GIS platforms for interpretation and analysis. Case studies that will be presented include major cities of Europe such as Rome, Naples and London, as well as in Mexico and the Middle-East, where natural hazards (e.g. land subsidence, volcanic activity) combine with anthropogenic factors (e.g. groundwater abstraction). An analysis of processing times to derive time series for each case study (less than half a day) and the precision (up to mm) and geological validation of the retrieved results proves the value of open platforms and tools such as the P-SBAS in G-POD, to handle demanding InSAR workloads and help the InSAR community (even non-experts) to freely generate InSAR products from open SAR data. - Casu F. et al. 2014. SBAS DInSAR parallel processing for deformation time series computation. IEEE JSTARS 7(8), 3285-3296, doi: 10.1109/JSTARS.2014.2322671 - Cigna F. 2015. Getting ready for the generation of a nationwide ground motion product for Great Britain using SAR data stacks: feasibility, data volumes and perspectives. Proc. IGARSS 2015, 1464-1467, doi: 10.1109/IGARSS.2015.7326055

The EnMAP-Box is designed to process imaging spectroscopy data and particularly developed to handle data from the upcoming EnMAP (Environmental Mapping and Analysis Program) sensor. It serves as a platform for sharing and distributing algorithms and methods among scientists and potential end-users. Starting with version 3.0 the EnMAP-Box is designed as a free and open source (FOSS) Python plug-in for the geographic information system QGIS, which is also FOSS. The two main goals are to provide (i) state-of-the-art applications for the processing of high dimensional spectral and temporal remote sensing data and (ii) a graphical user interface (GUI) that enhances the GIS oriented visualization capabilities in QGIS by applications for visualization and exploration of multi-band remote sensing data. Therefore, the algorithms provided in the EnMAP-Box will be of high value for many other, especially multi- and hyperspectral EO missions. The EnMAP-Box plug-inbridges and combines efficiently all advantages of QGIS (e.g. for visualisation, vector data processing), packages like GDAL (for data IO or working with virtual raster files) and abundant libraries for Python (e.g. SciKits learn for EO data classification and pyqtgraph for fast and interactive chart drawing). The plug-in consists of a (i) graphical user interface for hyperspectral data visualisation and e.g. spectral library management, (ii) a set of algorithms, and (iii) a high-level abstraction application programming interface (EnMAP API). The EnMAP-Box can be started from QGIS or stand-alone, and is planned to become registered in the QGIS plug-in repository. The algorithms are integrated in the QGIS processing framework, thus they may be used in the QGIS graphical model builder and chained with algorithms provided by other plugins. The EnMAP-Box API allows easy domain specific workflows with high level data types and operators, which allows the integration of a multitude of powerful algorithms with only few lines of code. In our presentation we will illustrate the concept of the EnMAP-Box and how it efficiently integrates various FOSS components. Based on this we will address the value and possibilities of this concept for integration e.g. with the Sentinel-2 toolbox.

SNAP is a software platform and a set of open source software toolboxes developed by ESA for the research and exploitation of data from the Sentinel missions, as well as other 3rd party high and medium resolution optical and SAR missions. The software consists of the SNAP (SeNtinel Applications Platform) software platform, for which EO toolboxes can be added as modules. Currently the major toolboxes running on SNAP are the Sentinel-1/2/3 toolboxes. In addition the user can install other toolboxes like the PROBA-V Toolbox, SMOS-Box Kit and the radarsat-2 polarimetric Toolkit. Each toolbox contains collection of specific processing tools, data product readers and writers that are added to the “generic” raster data analysis and visualisation tools offered by the SNAP platform. Functionalities of the Sentinel-1 toolbox include basic and advanced SAR image post-processing tools, as well as interferometric and polarimetric functionality. The toolbox is fully compatible with TOPSAR mode of Sentinel-1 data for which a complete TOPSAR Interferometric processing chains are possible. Further interferometric functionality is provided via bridges to 3rd party tools like SNAPHU (for phase unwrapping) and StaMPS (for Persistent Scatterer Interferometry (PSI)). The Toolbox also provides tools for exploitation of data including speckle filters, polarimetric decompositions, and classifiers. The software supports most other civilian spaceborne SAR missions including RADARSAT-2, TerraSAR-X/TanDEM-X, ALOS-1&2, Cosmo-Skymed, Kompsat-5, ERS-1&2 and ENVISAT. Sentinel-2 toolbox is a multi-mission toolbox, already providing support for Sentinel-2, RapidEye, Deimos, SPOT 1 to SPOT 5 datasets. In terms of processing algorithms the software includes tools specific to the Sentinel-2 mission, such as atmospheric correction module Sen2Cor, multitemporal synthesis processor (L3), biophysical products processor (L2B), deforestration detector, along with a set of more generic tools for high resolution optical data. Sentinel-3 Toolbox consists of a rich set of visualisation, analysis and processing tools for the exploitation of OLCI and SLSTR data from Sentinel-3 mission. It also supports the ESA missions Envisat (MERIS & AATSR), ERS (ATSR), SMOS as well as third party data from MODIS (Aqua and Terra), Landsat (TM), ALOS (AVNIR & PRISM) and others.The SNAP platform can be used interactively via the Graphical User Interface (GUI), or via the command-line enabling high-throughput processing on computer-clusters or clouds. SNAP has been extremely well received by the EO user communities with over 30000 active users in June 2017.

In 2014 ESA released the first version of the open source Sentinels Application Platform, SNAP, with 3 toolboxes providing instrument specific support for Sentinel 1, 2 and 3. Today, SNAP 5 is the current version. SNAP has evolved significantly, e.g. through the support of uncertainties or the multi-resolution data model, and the Sentinel toolboxes have been amended with various new supporting tools. New instrument toolboxes have been added for Radarsat, SMOS and Proba-V. SNAP has more than 10.000 downloads. SNAP is a platform that offers users various ways to use it, to adapt it to individual needs and to extend it. Most ESA Sentinel data users are familiar with SNAP Desktop, the interactive GUI application. A steadily growing user community is exploring and enjoying the possibilities offered by the Graph Processing Framework which allows users to connect different processing steps ("processing chains"), accible via the graphical graph builder and the batch processor; the python and Java API to write own scripts / programmes to implement own algorithmic ideas the SNAP forum to share experiences, get support for difficult questions, and to get in touch with the SNAP developers In this presentation we will showcast 3 examples of public contributions to SNAP that highlight the above elements of a modern community platform for developing scientific application in an open science environment: OLCI calibration modification (vicarious calibration) using graphs MusenAlp - lake surface temperature retrieval implemented in python GlobWetlands Africa Toolbox - QGIS using SNAP Key for all user interactions is the forum, and the SNAP forum which has several thouthands of threads. The issue tracker, on the contrary, is less popular although it a much better means for directly influencing the further development of the software. We will present both community tools, experiences and pros and cons. SNAP, and its precursors, the ENVISAT BEAM and NEST toolboxes, are a microcosms in itself which allow to study the performance of community tools when working in the very specific domain of Earth Observation Science and Applications. In our conclusion we will elaborate on the lessons-learned from more than 10 years of open source EO toolbox development, and hope to stimulate discussions on evolution and further improvements.

The ESA Atmospheric Toolbox (BEAT) is one of the ESA earth observation end user toolboxes. Since its inception in 2002 it has brought data reading, analysis and visualization support for products from a wide range of atmospheric missions such as ENVISAT, ERS, Metop, Aura, ACE, Odin, GOSAT, NPP-Suomi, Atmospheric CII, and Sentinel-5P. The toolbox consists of three main components: CODA, HARP, and VISAN. CODA is the core data reading interface allowing direct product access to a wide range of product formats using a single interface. HARP is the data harmonization and inter-comparison interface which brings all data to a harmonized data format and then provides operations such as filtering, collocation, unit conversion, quantity conversion (e.g. vmr to nd), vertical integration/smoothing, and binning/regridding. VISAN is a python-based basic analysis and visualization environment with advanced interactive plotting for 2D and geographical data. All components of the toolbox are freely available, cross-platform, and fully open source.