Methods of Working with HDF-EOS Files
The main methods of working with HDF-EOS files are through the following: a range of browsing and visualization software available both in the public domain (free) and for a cost from commercial vendors; a set of command line utilities; a set of programming interfaces and callable routines within the HDF and HDF-EOS libraries; through the HDF Configuration Record (HCR), and using the SDP Toolkit. Further detail on each method is given below:
As HDF and HDF-EOS has evolved, the most common way of looking and working with the files is through previously created software (free or commercial) and utilities. However, many users still wish to write their own routines and code to read and write data in HDF/HDF-EOS. For this reason, we will still detail the High-level APIs of the HDF-EOS library, although not to the extent that is/was done with the HDF counterpart. The utilities and software will be briefly summarized but the information provided at the directed web sites will be much more complete then could be supplied here. The latter method (SDP Toolkit and Toolkit_MTD) is more appropriate for Data centers/computer facilities and uses other ECS software together with the HDF-EOS library to mainly focus on providing metadata (core, inventory, and structual) for HDF-EOS files and datasets. Since the HDF-EOS library was developed to be a stand alone library, available separately, that allows EOS data end users to work with all aspects of HDF-EOS data, the SDP Toolkit will not be focused upon in great detail for this tutorial. The use of HCR to provide structual metadata for HDF-EOS files will also be briefly discussed.
HDF browsing and visualization tools
There are both publicly available (free) and commercial software packages that can be used to work with HDF-EOS files.In general, the scientific data within HDF-EOS files can be read by any tool that works with standard HDF files. However, what can not be accessed by all tools dealing with HDF and using the HDF library is the geolocation information. It has also been noted at the yearly ESDIS sponsored HDF-EOS Workshops that the development of software tools to work with HDF-EOS files, and especially through the HDF-EOS library, has been lacking (as of September 2000). However, many of the tools in the previous section have recently been modified to work with HDF-EOS data, even if only through the HDF library. A quick summary and the internet link of some of the more useful and commonly-used software that can work with HDF-EOS data is provided below. The user is directed to the software web sites for more detailed inforamtion on how to obtain and use the packages.
One method of working with HDF or HDF-EOS files is through command line utilities during a terminal session. Many of the command line utilities mentioned in the previous sections for use with HDF data sets may also be used on HDF-EOS objects. However, an additional set of utilities have been developed strictly for HDF-EOS. These utilities and detailed information may be downloaded from the HDF-EOS Standards and Tool Information Center page. A listing and brief description of some of these utilities is provided below:
Application Programming Interfaces (APIs) are specifically tailored for each type of data supported by the HDF or HDF-EOS library. These APIs are callable routines which will allow the user to access, read and write files specifically for the type of data they are interested in. The call of these APIs and associated routines occur in either a C or FORTRAN program with the main programming being a set of call statements that access, open, operate (read, write, etc.), and terminate. All of the rest is taken care of by the interface itself.
The HDF-EOS library provides APIs and associated routines for all the data types supported by HDF-EOS.The following is a list and short description of the various APIs with the C and FORTRAN prefaces for each interface given in parenthesis:
HDF-EOS APIsFurther detail on the various HDF-EOS APIs and how to use them is provided in Section 10 - HDF-EOS APIs.
As it stands today, HDF and HDF-EOS have evolved to where individual users have chosen different ways to work with HDF-EOS data sets or files. HDF-EOS, as described previously, is basically a subset or extenstion of the base HDF library and that HDF-EOS objects are HDF data objects with added geolocation information. Some users choose to work totally with the HDF-EOS APIs (but the HDF Libray must be compiled and linked first) to write both the scientific data itself as well as the geolocation information (latitude, lingitude, time, etc..) that makes it a swath or grid file. Many more HDF/HDF-EOS users utilize both the HDF and HDF-EOS libraries/APIs in creating a type of hybrid HDF file where the grid, swath, or point APIs are used to write the scientific data while the HDF APIs (i.e., SD and VS APIs) are used to provide the geolocation and temporal information for the swaths and grids.
The HDF Configuration Record (HCR) and the HDF-EOS library (HCR)
The HDF Configuration Record (HCR) was developed by NCSA and RSC as a formal description of HDF-EOS data that provides users of the data with a high-level description of the structure or configuration of the different types of objects in an HDF-EOS file. While ECS core metadata and product-specific metadata can be provided using the HDF-EOS library, the HCR can be used to provide the user with information concerning the layout or structure of the data files. This is known as structual metadata.
The nice feature of HCR is that it provides this "structual metadata" in a standard, fairly easy to read (by all) syntax called Object Description Language (ODL). The HCR and ODL will not be discussed in detail in this tutorial. Please check the NCSA HCR Page for more information on HCR. A primer on ODL, provided by NCSA, can be found here.
The SDP Toolkit (and Toolkit_MTD) and the HDF-EOS library
The SDP Toolkit is a collection of software and library functions developed by RSC for the ECS Project. Included in this collection of software is the HDF and HDF-EOS libraries. The SDP Toolkit was mainly developed for EOS data production developers and scientists at Science Computing Facilities (SCF) and Distributed Active Archive Systems (DAACs). The main purpose of the SDP Toolkit is to provide an interface between the instrument processing software used by EOS Science teams and and the production system environment.
In this production system, the HDF and HDF-EOS libraries are used together with the SDP Toolkit for metadata handling, time/date conversion, process control, memory management, error/status reporting and message handling. The metadata tools provided by the SDP Toolkit allow the user(s) to write ECS inventory and granule metadata into HDF-EOS files.
To obtain more information on the SDP Toolkit and to be able to download the software itself, please go to the SDP Toolkit Home Page or via anonymous ftp at the EDHS server, under the sdptk and hdfeos directories. The needed group password can be obtained be e-mailing pgstlkit@eos.hitc.com.
Most of the material in the SDP Toolkit is of little use to most EOS data users. More beneficial to the individual data provider,users, and software developers is Toolkit_MTD, which is a much smaller subset of the SDP Toolkit that only contains the parts and tools of the SDP Toolkit dealing with metadata and time/date conversions. The user is directed to a Toolkit_MTD White Paper and a Toolkit_MTD powerpoint presentation for a detailed description and overview of all the tools and operations provided by Toolkit_MTD.
Freely available software that permits the viewing and browsing HDF-EOS files have been developed by various institutes, data centers, and businesses. The following are just a few of the tools that have been developed to work with HDF-EOS data sets (often within HDF files) and are still available at no cost in the public domain:
Commercial Software
Below is a partial list of some of the more powerful and more commonly used software packages for working with HDF-EOS files or HDF-EOS objects/datasets within HDF files that are available for a price from commercial vendors: