SDS problem with metadata for NetCDF file

Nov 11, 2010 at 12:27 AM

I recently downloaded the SDS library from CodePlex to use in a project I’m working on dealing with some satellite data that’s stored in NetCDF format.  Unfortunately, I’m running into a problem with the tool with the particular files I’m trying to parse.  They’re NetCDF-4 files, internally in HDF5 format with compression of the variables.  When I try to use the library APIs in code or just through sds.exe to load the file, I’m getting the following error:

From sds.exe:

 FAILED: Failed to create DataSet instance from uri c:\temp\climate\foo.nc: Problem with attribute metadata.
 Problem with attribute metadata.

Exception from code:

 Microsoft.Research.Science.Data.DataSetCreateException was unhandled
   Message=Failed to create DataSet instance from uri c:\temp\climate\foo.nc: Problem with attribute metadata.
   Source=Microsoft.Research.Science.Data
   FailedUri=c:\temp\climate\foo.nc
   StackTrace:
        at Microsoft.Research.Science.Data.Factory.DataSetFactory.Create(String uri)
        at Microsoft.Research.Science.Data.DataSet.Open(String uri)
        at ClimateFill.Program.Main(String[] args) in C:\projects\ClimateFill\ClimateFill\Program.cs:line 13
        at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
        at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
        at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
        at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
        at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
        at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
        at System.Threading.ThreadHelper.ThreadStart()
   InnerException: Microsoft.Research.Science.Data.NetCDF4.NetCDFException
        Message=Problem with attribute metadata.
        Source=Microsoft.Research.Science.Data.NetCDF4
        ResultCode=-107
        StackTrace:
             at Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet.HandleResult(Int32 resultCode)
             at Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet.InitializeFromFile(String fileName, ResourceOpenMode openMode)
             at Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet..ctor(String uri)

I can use various other tools (ncdump, etc) to look at the file, so I’m fairly certain it’s not a problem with the file itself.  I tried running the tool on older “classic format” NetCDF files, and that seems to work just fine, so I’m guessing it’s probably related to the new NetCDF-4 file with the HDF5 internal format.  As far as I can tell, that should be supported...?

A couple more things I tried.  Firstly, the netcdf4.dll that is in the same dir as sds.exe was a 32-bit dll, but I’m on a x64 machine.  I tried browsing the source code on the CodePlex site and downloaded the netcdf4.dll from Main\ext\x64 and replacing the original netcdf4.dll with that one, but it didn’t seem to make any difference.

Also, for a while, I thought that perhaps I was missing some hdf5 dlls that needed to be built separately or something (seemed to be indicated by the NetCDF .rtf file that is in the SDS core .msi), but browsing through the code, in NetCDFProvider\NetCDFDataSet.cs, there’s this comment that leads me to believe that's not the case.
    /// The provider requires unmanaged netcdf4.dll library to work.
    /// The current Scientific DataSet release runtime package includes netcdf4.dll
    /// built from Unidata's source codes for NetCDF Release 4.0.1
    /// with statically linked satellite libraries (including HDF5).

If someone could provide any sort of guidance to help me figure out what’s going on here, that would be much appreciated.  I can supply the NetCDF file I’m trying to open if that would be helpful.

Thanks!

--Jeff

Nov 11, 2010 at 10:57 AM

Hi Jeff,

First of all, the installers provided should install 64-bit libraries if your machine is x64. If you used them and they don't do that on your machine, please inform me.

Concerning NetCDF, could you supply the file to check what is wrong?

 

Regards, Dmitry.

 

Nov 11, 2010 at 6:02 PM

Ah, regarding the 64-bit libraries, I think I was confused because sds.exe is in "c:\Program Files (x86)\Microsoft Research\Scientific DataSet 1.2", and the netcdf4.dll that is in that dir is a 32-bit binary... however, the managed NetCDF assembly that's actually getting used is Microsoft.Research.Science.Data.NetCDF4.dll, which lives in the GAC, and there's a 64-bit copy of netcdf4.dll next to *that* file, so that's presumably the one that's being used.  Quite confusing, but anyway, that doesn't seem to be a problem :-)

As far as the NetCDF file that's not working, I put it up on an online file-share here: http://www.4shared.com/file/gItdV7R1/foo.html
(shouldn't require any registration or anything to get it).

Thanks,
--Jeff

Nov 18, 2010 at 9:52 PM

Any luck figuring out what's wrong with parsing that file?

Nov 19, 2010 at 11:59 AM

The problem very likely lies within the unmanaged NetCDF library: nc_open() method for the given file returns an error "Problem with attribute metadata" (which is a NetCDF4-specific error). We recently use NetCDF 4.0.1 Release version, so the issue maybe solved in the most recent version 4.1.1.

Which version of NetCDF was used to create that file?

 

Regards, Dmitry

Nov 22, 2010 at 11:39 PM

The version of NetCDF used to create these files was 4.1-snapshot2010021811 (a snapshot of the 4.1 branch before the official 4.1.1 release).  Can I just copy version 4.1.1 of netcdf4.dll over the version 4.0.1 one (alongside Microsoft.Research.Science.Data.NetCDF4.dll in the GAC) to see whether that might fix the problem?  If so, would I have to build my own copy with statically linked HDF5 libraries (as mentioned in the code comments above)?  Any pointers on doing so?

Thanks,
--Jeff

Nov 23, 2010 at 10:32 AM

Yes, that can work. If you have version 4.1.1 of netcdf4.dll either statically or dynamically linked with HDF5 libraries, you can replace the existing netcdf4.dll with newer one (and, in case of dynamic linking, also supply satellite libraries).

The problem (for us) with NetCDF Release 4.1.x is that it doesn't contain official solution/projects files for Windows, meanwhile NetCDF 4.0.1 does.

 

Regards, Dmitry.

 

Nov 23, 2010 at 6:25 PM

I played around with this for a while, and finally got it to work using pre-built binaries from ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32-bin.zip.  Those are 32-bit only, but I used corflags to change the managed program to have the 32bit-only flag (i.e. run using the 32-bit .net runtime), and it correctly parsed my file.

So, it does seem that there's something the 4.1.1 NetCDF is writing in the headers that the 4.0.1 version of NetCDF can't parse.  Is there any plan to work on getting the 4.1.x stuff building on Windows (and thus the SDS distributed with a newer version of NetCDF binaries)?  From the NetCDF website, it certainly seems like building on Windows is a supported scenario, even if there aren't official project files.  I'll play around with that myself as well when I have a chance, though for the moment, I've worked around this by simply adding an extra step in the middle to convert the .nc4 files back to an older format; that loses the compression though, multiplying the size of all the files by about 50x, so it's not a great long-term solution.

Thanks for your help with this!
--Jeff

Nov 24, 2010 at 3:54 PM

I also replaced the netcdf4.dll (of version 4.0.1) with the netcdf4.dll of version 4.1.1 and the SDS library correctly opened your file; thus, the library of version 4.0 is not obliged to read files of version 4.1.1.

We will move to the next version of NetCDF, but not in the next release that is coming soon. Unfortunately, there is no time to make proper NetCDF 4.1.1 build for Windows now.

 

Regards, Dmitry.