Attributes and Metadata for HDF-EOS
Metadata has often been phrased as "data about data" and provides the user/reader with a description of the data files and data sets, including content, format, size, parameters, dimensions, name, etc... The reader is directed to Metadata in EOSDIS, the ECS Metadata Home Page and the ECS Metadata FAQ for detailed information on metadata and how it might apply to EOS data sets, including those to be written in HDF/HDF-EOS. In addition, the reader is also directed to Thoughts on HDF-EOS Matadata for early (1996) thoughts on the provision of metadata in HDF-EOS.
In brief, data in the EOS system must include both core metadata and product specific metadata. The core metadata has been defined by EOSDIS and is common to most EOSDIS products while the product specific metadata exists for information on individual or specific data products not contained in the Core metadata (such as instrument information). A third type of metadata, called structual metadata is used to describe the structure, layout and contents of a particular data granule or data field. This includes describing the relationship between the different data components such as the the scientific data itself and its' corresponding geolocation information. This third type of metadata is specific to HDF-EOS and is only present if the HDF-EOS library and API commands have been invoked to work with point, swath or grid structures. Below is a listing of several ways to work with metadata or attributes for HDF-EOS
Using the HDF and HDF-EOS APIs for Attributes/Metadata
As mentioned above, the stuctual metadata containing the geolocation information is created by the HDF-EOS library through the invoking of the HDF-EOS API function calls. However, this information was/is not usually intended to be accessed by the user.
One way of providing ECS metadata (Core and Product Specific) for HDF-EOS data is the use of global attibutes within the HDF-EOS and HDF APIs. All HDF APIs and the three Point, Swath, and Grid APIs of the HDF-EOS Library all provide the user with the ability of reading and writing metadata and attributes using statement calls in the user composed program. The attributes are actually blocks of text which spell out the metadata and are written in Object Description Language (ODL) with the form of:
parameter = value
is used and where
parameter is the identifier or label
value is a real, integer, or character string describing the parameter/object.
A primer on ODL, provided by NCSA, can be found here.
A deseriable feature of ODL is that it is in straight-forward text format that makes it easy to read and understand by all users. Using the ODL, metadata information can be read or provided for all the necessary dimensions, geolocation fields, data fields, paramaters, etc.. of the data set.
The following are the ODL objects most frequently used:
Object Statement where OBJECT = identifier
An End Object Statement where END_OBJECT = identifier Attribute Assignment Statement where attribute = value
As an example to show the way the ODL can be used, the following text strings can be used to write some of the metadata for a HDF-EOS swath file:
OBJECT = Swath
Name = Swath_EOS
OBJECT = DImension
Name = "Alongtrack"
Size = 35
END_OBJECT = Dimension
OBJECT = Dimension
Name = "Crosstrack"
Size = 50
END_OBJECT = Dimension
OBJECT = DimensionMap
GeoDimension = "GeoTrack"
DataDimension = Restr
Offest = 0
Increment = 1
END_OBJECT = DimensionMap
OBJECT = DataField
Name = "wind"
DataType = DFNT_FLOAT64
DimList = "GeoTrack"
END_OBJECT = DataField
OBJECT = GeoField
Name = "Time"
DataType = DFNT_FLOAT64
DimList = "GeoTrack"
END_OBJECT = GeoField
END_OBJECT = Swath
The metadata are provided in these ODL text strings and then attached to HDF or HDF-EOS files using global attributes. Either the HDF SD API or the SW and GD HDF-EOS APIs may be used to attach the metadata. As an example, the following illustrates how the SW API (similar commands for the Grid API exist) can be used to write metadata using global attributes AFTER the file has been attached:
status=swwrattr(swid, attrname,ntype,count,data) (FORTRAN)
or
status=SWwriteattr(swid, attrname,ntype,count,data) (C);
where
swid = swath id returned by the swcreate/SWcreate command
attrname= name of attribute
ntype= number type of attribute (described previously)
count= number of values in each attribute
data= attribute values (character string in ODL is an option).
The attribute name could be something like COREMETADATA.0, STRUCTMETADATA.0, or PRODUCTMETADATA.0 with the data being the ODL text string as shown above (just an example).
Besides for writing Metadata, global attributes may also be used to list, inventory, and read existing metadata or attributes through the following commands (only FORTRAN examples are provided):
status=swattrinfo(swid, attrname,ntype,count)
nattr=swinqattrs(swid, attrlist,bufsize)
status=swrdattr(swid, attrname,databuf)
where
swid = swath id returned by the swattach/SWattach command
attrname= name of attribute
ntype= number type of attribute
count= number of values in each attribute
databuf= attribute values (character string in ODL is an option)-- OUTPUT
attrlist= list of all attributes in swath
To help users, the ESDIS Project office has created several utilities that enable the users to work with HDF-EOS metadata. The metadmp utility and ufm (user friendly metadata) translator allows the user to, respectively, extract metadata (in ODL format) from HDF-EOS or EOS data granules and to thranslate the metadata ODL into a more easily read HTML format.
Using the SDP Toolkit or Toolkit_MTD
Another way of reading or writing metadata for HDF-EOS is through the use of the HDF-EOS library and the SDP Toolkit or Toolkit_MTD. As mentioned in Section 7 - Methods of Working with HDF-EOS, the SDP Toolkit/Toolkit_MTD contians the HDF and HDF-EOS libraries for download and were mainly developed for EOS data producers, DAACs, and Science Computing Facilities with one of the main objectives being the handling of ECS metadata. Together with the HDF and HDF-Eos libraries, the commands of the Toolkits are used to allow the user to write (and read) ECS core and product specific metadata into a Metadata Configuration File (.ncf or .met). This file accompanies the HDF-EOS data file and is written in the ascii ODL foramt (described above).
To obtain more information on the SDP Toolkit and to be able to download the software itself, please go to the SDP Toolkit Home Page or via anonymous ftp at the EDHS server, under the sdptk and hdfeos directories. The needed group password can be obtained be e-mailing pgstlkit@eos.hitc.com.
Most of the material in the SDP Toolkit is of little use to most EOS data users. More beneficial to the individual data provider,users, and software developers is Toolkit_MTD, which is a much smaller subset of the SDP Toolkit that only contains the parts and tools of the SDP Toolkit dealing with metadata and time/date conversions. The user is directed to a Toolkit_MTD White Paper and a Toolkit_MTD powerpoint presentation for a detailed description and overview of all the tools and operations provided by Toolkit_MTD.
Using HCR for Attributes/Metadata
Another way of reading or writing metadata for HDF-EOS is through the use of the HDF Configuration Record (HCR). The HCR is a formal record structure that describes the structure and metadata content of an HDF-EOS file using the easy to read (or "human-readable") and standard syntax called Object Description Language (ODL). The HCR, using ODL, supports the specification of all structual metadata for point, swath and grid objects using the attribute strings in the HDF-EOS library calls.
Please check the NCSA HCR Page for more information on HCR. A primer on ODL, provided by NCSA, can be found here.
A deseriable feature of HCR (and ODL) is that it is in straight-forward text format that makes it easy to read and understand by all users. Using the HCR and ODL metadata information can be read or provided for all the necessary dimensions, geolocation fields, data fields, paramaters, etc.. of the data set.
The main question is then how and where do I provide (or read) this metadata for HDF-EOS Data Objects/Files. The first step would be to download all the HCR sofware and documentation from the NCSA HCR Page. Included in this software is a tool called hdf2hcr which allows the user to generate a HCR file that describes the EOS objects in an HDF-EOS file.