What types of data does HDF-EOS support?
Which version of HDF and HDF-EOS should I use?
Where can I get additional and detailed information on HDF-EOS?
As mentioned in Section 2 - An Introduction to HDF, HDF as developed by NCSA is the common data format selected by NASA and ESDIS as the format standard for data exchange of EOS data. The HDF software library provides the application routines and code to work with (create, read, write, etc...) various data types in HDF.
As it stands, however, many of the data products and data types that result from the EOS missions (current and future) are not fully supported by the base HDF library nor map directly to NCSA/HDF data types. These data types mostly include geolocation data types, or data sets with associated geolocation or temporal information. Examples of these types of data sets include point data sets from field observation studies, data sets from the swaths of satellite borne sensors, and data sets on gridded map projections. These type of measurements or data, called Point, Swath, and Grid data sets, are major components and products of the EOS missions, which began in late 1999 with the launch of TERRA, and will continue with the future launches of AQUA (2001) and AURA. As a result, the HDF-EOS (Hierarchical Data Format-Earth Observing System) software library was created for the EOSDIS Core System (ECS) by the Raytheon Systems Company (RSC) to fully and optimally support these types of data.
The HDF-EOS library is not a stand alone software package, rather it is an extension of the base HDF library that requires the installation of the base HDF library and the use of its' basic function calls as an underlying basis to successfully work with HDF-EOS data types like Point, Swath, and Grid. Therefore, in all actuality, HDF-EOS data sets or objects are basically HDF objects with the added feature of being able to support geolocation information. This geolocation information is included in the HDF files through the automatic inclusion of structual metadata.
In particular, HDF-EOS allows for the subsetting of point, swath, and gridded data sets in terms of time and geolocation (Latitude, longitude, altitude) and not just in rows and columns ( i and j) like in HDF. One other feature that specifically defines an HDF-EOS data set is the mandatory inclusion of ECS Core metadata. This is not required in ordinary HDF files, although it may be used as an option. It should also be noted that an HDF file may include both HDF AND HDF-EOS data objects and that HDF-EOS data sets can indeed be read by HDF tools and routines but with the geolocation information not accessible.
The HDF-EOS format shares many of the same important features of HDF (i.e., self-description) as mentioned in Section 2 - An Introduction to HDF. In addition, HDF-EOS provides the following:
What types of data does HDF-EOS support?
As of Summer 2000, the current version of the HDF-EOS library (HDF-EOS 2.6) supports three types of geolocation data types:
Similar to HDF, each data type is supported by a collection of software called an Application Programming Interface (API) that allows the user to work with the data, particularly in reference to both time and space. The Swath, Point, and Grid APIs will be discussed later in the tutorial.
Which version of HDF and HDF-EOS should I use?
The most current version or release of HDF and HDF-EOS is the best place to begin. As of July 2000, the current version of the HDF library is HDF 4.1r3. The current version of the HDF-EOS library, HDF-EOS2.6, is built upon and based on this version of HDF (HDF4.1).
As mentioned previously, a new version of HDF, called HDF5, has also recently been developed to address the shortcomings of HDF4. This new HDF library, which will eventually become the new standard over time, includes simpler source codes, more consistent and fewer data models, and the ability to work with large data sets (> 2GB). A new version of the HDF-EOS Library, called HDF-EOS3.0, has been developed which is totally based on the new HDF5 Library and will have an entirely different functionality. The HDF-EOS3.0 Library will will soon officially become "avaialble".
Data producers and instrument teams for the TERRA and AQUA sensors are using the HDF4 Library (and, thus, HDF-EOS 2.6 if HDF-EOS is used) to create many data sets. Users of this data should continue to use the HDF4/HDF-EOS2.6 libraries. However, as time evolves, data sets created by or together with the HDF5 Library will employ routines from HDF-EOS3.0. Tool developers and data providers have been working with all the various HDF and HDF-EOS libraries and have found that, due to naming convention conflicts, both the HDF-EOS2.6 and HDF-EOS3,0 libraries can not be opened and accessed at the same time. Only one can be used.
One of the bigger differences between HDF-EOS2.6 and HDF-EOS3.0 is that with HDF-EOS2.6 the user must access the structual metadata to map geolocation data with the scientific data while this dependency has been been renoved in HDF-EOS3.0 as the structual Metadata/geolocation information will be contained within HDF5 objects.
Where can I get additional and detailed information on HDF-EOS?
There are several locations where the user may find detailed documentation and information on HDF-EOS. One of the most helpful sites for HDF-EOS documentation is the ECS Data Handling System (EDHS). Doing a quick search for "HDF-EOS" at this location, the user will find links to download many HDF-EOS documets, including the following:
Another informative site that contains detailed HDF-EOS documentation is the HDF-EOS Standards and Tools Information Center home page. The HDF-EOS Information Resources page at this site also provides links to the latest documentation on HDF-EOS including:
In addition to containing HDF-EOS documentation, this site also provides links to or information on HDF-EOS source code, how to order an HDF-EOS data sampler, and the latest version of the HDF-EOS FAQ (Frequently Asked Questions). This site also provides links to documentation on the HDF Configuration Record (HCR).
The NCSA HCR Page provides complete documentation on HCR. The HCR is a free set of software tools created by NCSA that provides an interface for the user to create, read or write HDF (including HDF-EOS) files and is a formal description of the HDF-EOS data. The HCR will be discussed in later sections.
An HDF-EOS Data Sampler CD can be ordered from the HDF-EOS Standards and Tools Information Center web site. In addition, some of the HDF-EOS data contained on the CD sampler may also be viewed or downloaded from the DIAL server at NASA Goddard Space FLight Center.
With the launch of TERRA in late 1999, as well as the data being generated by Landsat-7 and TRMM, earth science data in HDF/HDF-EOS format is being archived at various data centers and Distributed Active Archive Centers (DAACs). One central location or site that provides user access to EOS and other types of data archived at many of these centers is the EOS Data Gateway (EDG). The user may also wish to check the individual data centers such as the EROS Data Center (EDC) for Landsat-7, ASTER, and MODIS-Land data; the Langley Atmospheric Sciences Data Center for MISR and MOPITT data products; the National Snow and Ice Data Center (NSIDC) for MODIS ice-related products; and the Goddard Space Flight Center Earth Science (GES) Data Center for MODIS atmospheric and ocean data products. Sample MODIS products may also be obtained from GSFC. The above-mentioned DAACs can also provide the user with HDF and HDF-EOS reader software to work with the data.
As mentioned earlier, many of the files containing HDF-EOS objects such as swaths and grids have been placed together with HDF objects in "hybrid" HDF files using the HDF library rather then using the HDF-EOS library to create pure HDF-EOS files with only HDF-EOS objects.