An HDF and HDF-EOS Tutorial for Beginners: EOSDIS Users and Small Data Providers
The NASA ESDIS project selected the Hierarchical Data Format (HDF) as the common data format of choice for standard product exchange and distribution. As developed by the National Center for Supercomputer Applications (NCSA), the HDF format is supported by a collection of software routines and applications needed to work with data sets in HDF. This set of software, referred to as the HDF library, is available in the public domain. To facilitate the exchange of data and data products generated by current (TERRA) and upcoming Earth Observing System (EOS) missions (such as AQUA and AURA), as well as satellite missions such as TRMM, a sub-library or library extension of HDF, called HDF-EOS, has been developed for the EOSDIS Core System (ECS) by the Raytheon Systems Corporation (RSC). HDF-EOS was developed to specifically deal with the types of satellite and field campaign data products that are and will be routinely generated by EOS missions but which are not fully supported by the base HDF library.
While there are many advantages to the use of HDF and HDF-EOS, a key to their success as a common data format and software library may be dependent upon expanding the general user and science communities' awareness, knowledge, and comfort with HDF and HDF-EOS. In particular, it is the individual investigators, academia (students through researchers), the educational community, and the general public that many times do not have the required knowledge, nor the resources to commit to obtaining this knowledge, to work with HDF and HDF-EOS files.
In response to this need, the NASA ESDIS project has funded the creation of this on-line tutorial geared towards HDF and HDF-EOS beginners. The purpose of this tutorial is to provide the non-expert, particularly potential future users and producers of EOS data, with the necessary information to enable one to learn about HDF/HDF-EOS and to successfully browse, read and write data sets in HDF format. This information will include, but not be limited to, sections on the basics of HDF/HDF-EOS and HDF/HDF-EOS files, the required software/hardware, the various ways of working with the files, a review of HDF and HDF-EOS commands and operations, and instructions and advice for writing programs to work with HDF and HDF-EOS.
A portion of the information presented here can also be found in much further detail scattered throughout various other literature and several of the excellent reference guides and manuals (more on this in Section 2- An Introduction to HDF and Section 3- An Introduction to HDF-EOS) written by NCSA for HDF and by RSC for HDF-EOS. However, the goal of this tutorial is to present, in a concise and easy to understand form, only the information needed to help the HDF novice to browse, read and write basic HDF/HDF-EOS files. More importantly, as they are certainly linked, it is the goal of this tutorial to combine the information on both HDF and HDF-EOS under one encompassing document. Furthermore, the HDF library has been designed to work with many different types of data (arrays, images, etc.) and to carry out both simple and complex operations on data sets. As a teaching tool, this tutorial will concentrate on only several selected data types supported by HDF (images, palettes, scientific data arrays) and HDF-EOS (Point, grid, and swath) libraries, and the basic operations such as reading, writing, and browsing entire data sets. The user will be directed to the some of the most critical documentation to learn about all the various operations and data types supported by HDF and HDF-EOS.
Other excellent HDF tutorials have been developed by NCSA and GSFC. The current tutorial contains much more of the basic information of HDF and is geared to the HDF beginner or novice. Though there are many places of overlap between the tutorials, they seem to compliment each other in providing information for all types of HDF users.
The current version of this HDF and HDF-EOS tutorial concentrates on the latest release of the HDF library (HDF 4.1r3 as of July 2000) and HDF-EOS sub-library (HDF-EOS 2.6 as of June 2000), and how to use them for browsing, reading and writing HDF files containing scientific data arrays, raster images with associated palettes; and point, swath, and gridded HDF-EOS data sets.
In addition to HDF4, a second version of HDF, called HDF5, has also been developed. This new library was designed to address the main drawbacks of HDF4, particularly the inability to deal with large data sets. Both the HDF4 and HDF5 Libraries are currently being used by data producers and being supported by NCSA, although all future development of the HDF library will be on HDF5.
As of the summer of 2000, investigators and science teams were making individual decisions regarding which HDF format/library (4 or 5) to use for their data. Data providers and instrument teams from the current TERRA platform (MODIS, MISR, etc..) and the upcoming AQUA platform (2001) are using the HDF4 library. Partly for this reason, as well as the fact that there has been little commercial tool development for HDF5, this tutorial will be focused upon HDF4. To learn more about HDF5, the readers are directed to the NCSA HDF5 page and the outstanding HDF5 tutorial that has been developed by NCSA.
With the creation of a new HDF5 Library, there has simarlarly been the development of a new HDF-EOS sub-library which is totally based on HDF5. This is the new HDF-EOS 3.0 library, which has entirely new functionality, but it has yet to be offically accepted by ESDIS. This tutorial will focus on the contents and supporting software for the original HDF-EOS 2.6 library (based on HDF4).It should be noted that NASA and HDF-EOS have full intention of indefinitely supporting HDF4 and will expand significant effort on facilitating the interoperability and conversion between HDF4 and HDF5. This will be touched upon later on in the tutorial
In support of a contract to the NASA ESDIS project, this HDF tutorial has been designed by Simpson Weather Associates, Inc. (SWA) with the goal of teaching the novice HDF and HDF-EOS users, especially potential users of EOSDIS and both current and future EOS data products, how to use the libraries to read and write HDF or HDF-EOS files. The tutorial has been constructed in two parts. The main part of the tutorial is what we call the "Lecture" component where we present what we think is the information necessary for a novice user to learn what HDF and HDF-EOS is, what it can be used for, and how to apply it in practice. Included in this "Lecture" material is a step-by-step outline detailing what the user must do to successfully read or write an HDF/HDF-EOS file.
This "lecture" component makes up most of the tutorial. The second and much smaller component of the tutorial is a question and answer section (what we call the "Laboratory") which tests the user's knowledge of HDF, concentrating on the information needed by the novice or beginner user to work independently with the HDF library to read and write files.
We realize that the familiarity and knowledge level of the users of this tutorial will span a wide range. As a result, we think it should be up to the users to decide how they wish to learn and navigate through the tutorial. However, we do advise that those with very little or no knowledge of HDF should first preview and study the lecture material before testing themselves with the Laboratory section.
Modifications and updates to the HDF4 and HDF-EOS2 libraries, as well as user feedback, may require changes to the tutorial content. As an example, new incremental versions of HDF (HDF4.1r4) and HDF-EOS (HDF-EOS 2.7) were released in December of 2000, after the completion of the contract. The tutorial will eventually be updated to indicate any neceesary changes from these versions of HDF and HDF-EOS. Although NCSA's HDF5 tutorial is very thorough, it is also hoped that, in the future, someone will take on the task of creating a tutorial specifically aimed at beginner users that covers the new HDF5 and HDF-EOS 3.0 libraries.