Using dataset library to read netcdf files

May 19, 2011 at 3:19 PM

I would like to use these libraries to read a NetCDF file provided by madis and import this into a dataset.  I'd like to do this from inside of a C# application rather than from command line.  I'm not sure if this is possible using the tool?  Can I do this?

Thanks in advance,

Linda

May 19, 2011 at 3:35 PM
Edited May 19, 2011 at 3:36 PM

Hi Linda,

Reading NetCDF using SDS is quite simple; to do this install current release, and then add a reference to the assembly Microsoft.Research.Science.Data in your C# project. Now you can open NetCDF file and read its attributes and data using DataSet class; for example, following code opens existing file "air.mon.mean.nc" and gets some metadata and data from it:

using Microsoft.Research.Science.Data;
using Microsoft.Research.Science.Data.Imperative;
...
var dataset = sds.DataSet.Open("air.mon.mean.nc?openMode=readOnly");

// Get attributes of variable "air"
short[] valid_range = dataset.GetAttr<short[]>("air", "valid_range");
floatscale = dataset.GetAttr<float>("air", "scale_factor");
floatoffset = dataset.GetAttr<float>("air", "add_offset");

// Get data of variable "lat"
float[] lat = dataset.GetData<float[]>("lat");

DataSet Imperative API reference is available here.

Read the Getting started document for a more detailed introduction. Source code for tutorials can be found here, under solution sln/CodePlexTutorials.

If you have any further questions, please do not hesitate to ask.

 

Regards, Dmitry.

 

May 19, 2011 at 3:39 PM

Dmitry,

Thanks for your prompt reply.  I have tried opening the dataset in the viewer and it just hangs.  According to the madis site, it indicates

that the data is netCDF, but the file has no extension on it.  I changed the extension to .nc, but am not sure why it hung.  It is definitely

not a csv file, so I made the assumption that .nc was the appropriate choice.  Is there a way that I can tell by editing the file whether it

is the correct format?

Thanks,

Linda

 

May 19, 2011 at 3:47 PM

Linda, if you have installed SDS on your machine, please try this: run cmd.exe and type

sds "path-to-the-file.nc?openMode=readOnly"

If this is supported NC file, its schema will be printed in the console. Otherwise it should print an error message; if you post it here, I will probably be able to suggest something.

 

Good luck,

Dmitry.

May 19, 2011 at 3:53 PM

Leon,

OK.  Here is what I got for output:

[209] typePST of type SByte (maxPSTEntries:300)
[208] namePST of type SByte (maxPSTEntries:300) (maxStaTypeLen:11)
[207] code4PST of type Int16 (maxPSTEntries:300)
[206] code3PST of type Int16 (maxPSTEntries:300)
[205] code2PST of type Int16 (maxPSTEntries:300)
[204] code1PST of type Int16 (maxPSTEntries:300)
[203] numPST of type Int16
[202] test1 of type SByte (recNum:71943) (maxLDADtestLen:51)
[201] skyCovLayerBase of type Single (recNum:71943) (maxSkyCover:6)
[200] skyCvr of type SByte (recNum:71943) (maxSkyCover:6) (maxSkyLen:8)
[199] numSkyCoverObs of type Int32 (recNum:71943)
[198] presWx of type SByte (recNum:71943) (maxWeatherLen:25)
[197] mobileElev of type Single (recNum:71943)
[196] mobileLon of type Single (recNum:71943)
[195] mobileLat of type Single (recNum:71943)
[194] waterLevel of type Single (recNum:71943)
[193] soilMoisturePercentQCD of type Single (recNum:71943) (QCcheckNum:10)
[192] soilMoisturePercentQCR of type Int32 (recNum:71943)
[191] soilMoisturePercentQCA of type Int32 (recNum:71943)
[190] soilMoisturePercentDD of type SByte (recNum:71943)
[189] soilMoisturePercent of type Single (recNum:71943)
[188] soilMoisture of type Single (recNum:71943)
[187] soilTemperatureQCD of type Single (recNum:71943) (QCcheckNum:10)
[186] soilTemperatureQCR of type Int32 (recNum:71943)
[185] soilTemperatureQCA of type Int32 (recNum:71943)
[184] soilTemperatureDD of type SByte (recNum:71943)
[183] soilTemperature of type Single (recNum:71943)
[182] fuelMoistChangeTime of type Double (recNum:71943)
[181] fuelMoisture of type Single (recNum:71943)
[180] fuelTempChangeTime of type Double (recNum:71943)
[179] fuelTemperature of type Single (recNum:71943)
[178] roadSubsurfaceTemp4 of type Single (recNum:71943)
[177] roadSubsurfaceTemp3 of type Single (recNum:71943)
[176] roadSubsurfaceTemp2 of type Single (recNum:71943)
[175] roadSubsurfaceTemp1 of type Single (recNum:71943)
[174] roadState4 of type Int16 (recNum:71943)
[173] roadState3 of type Int16 (recNum:71943)
[172] roadState2 of type Int16 (recNum:71943)
[171] roadState1 of type Int16 (recNum:71943)
[170] roadLiquidDepth4 of type Single (recNum:71943)
[169] roadLiquidDepth3 of type Single (recNum:71943)
[168] roadLiquidDepth2 of type Single (recNum:71943)
[167] roadLiquidDepth1 of type Single (recNum:71943)
[166] roadLiquidIcePercent4 of type Single (recNum:71943)
[165] roadLiquidIcePercent3 of type Single (recNum:71943)
[164] roadLiquidIcePercent2 of type Single (recNum:71943)
[163] roadLiquidIcePercent1 of type Single (recNum:71943)
[162] roadLiquidChemPercent4 of type Single (recNum:71943)
[161] roadLiquidChemPercent3 of type Single (recNum:71943)
[160] roadLiquidChemPercent2 of type Single (recNum:71943)
[159] roadLiquidChemPercent1 of type Single (recNum:71943)
[158] roadLiquidChemFactor4 of type Single (recNum:71943)
[157] roadLiquidChemFactor3 of type Single (recNum:71943)
[156] roadLiquidChemFactor2 of type Single (recNum:71943)
[155] roadLiquidChemFactor1 of type Single (recNum:71943)
[154] roadLiquidFreezeTemp4 of type Single (recNum:71943)
[153] roadLiquidFreezeTemp3 of type Single (recNum:71943)
[152] roadLiquidFreezeTemp2 of type Single (recNum:71943)
[151] roadLiquidFreezeTemp1 of type Single (recNum:71943)
[150] roadTemperature4 of type Single (recNum:71943)
[149] roadTemperature3 of type Single (recNum:71943)
[148] roadTemperature2 of type Single (recNum:71943)
[147] roadTemperature1 of type Single (recNum:71943)
[146] capPi of type Single (recNum:71943)
[145] formalError of type Single (recNum:71943)
[144] meanWeightedTemperature of type Single (recNum:71943)
[143] wetSignalDelay of type Single (recNum:71943)
[142] drySignalDelay of type Single (recNum:71943)
[141] totalSignalDelay of type Single (recNum:71943)
[140] totalColumnPWV of type Single (recNum:71943)
[139] rawMessage of type SByte (recNum:71943) (maxLDADmessageLen:512)
[138] seaSurfaceTempICR of type Int32 (recNum:71943)
[137] seaSurfaceTempICA of type Int32 (recNum:71943)
[136] seaSurfaceTempQCD of type Single (recNum:71943) (QCcheckNum:10)
[135] seaSurfaceTempQCR of type Int32 (recNum:71943)
[134] seaSurfaceTempQCA of type Int32 (recNum:71943)
[133] seaSurfaceTempDD of type SByte (recNum:71943)
[132] seaSurfaceTemp of type Single (recNum:71943)
[131] solarRadChangeTime of type Double (recNum:71943)
[130] solarRadiation of type Single (recNum:71943)
[129] timeSinceLastPcp of type Double (recNum:71943)
[128] precipIntensity of type Int16 (recNum:71943) (maxSensor:2)
[127] precipType of type Int16 (recNum:71943) (maxSensor:2)
[126] precipRateQCD of type Single (recNum:71943) (QCcheckNum:10)
[125] precipRateQCR of type Int32 (recNum:71943)
[124] precipRateQCA of type Int32 (recNum:71943)
[123] precipRateDD of type SByte (recNum:71943)
[122] precipRate of type Single (recNum:71943)
[121] precipAccumICR of type Int32 (recNum:71943)
[120] precipAccumICA of type Int32 (recNum:71943)
[119] precipAccumQCD of type Single (recNum:71943) (QCcheckNum:10)
[118] precipAccumQCR of type Int32 (recNum:71943)
[117] precipAccumQCA of type Int32 (recNum:71943)
[116] precipAccumDD of type SByte (recNum:71943)
[115] precipAccum of type Single (recNum:71943)
[114] rawPrecip of type Single (recNum:71943)
[113] visibilityICR of type Int32 (recNum:71943)
[112] visibilityICA of type Int32 (recNum:71943)
[111] visibilityQCD of type Single (recNum:71943) (QCcheckNum:10)
[110] visibilityQCR of type Int32 (recNum:71943)
[109] visibilityQCA of type Int32 (recNum:71943)
[108] visibilityDD of type SByte (recNum:71943)
[107] visibility of type Single (recNum:71943)
[106] windDirMaxQCD of type Single (recNum:71943) (QCcheckNum:10)
[105] windDirMaxQCR of type Int32 (recNum:71943)
[104] windDirMaxQCA of type Int32 (recNum:71943)
[103] windDirMaxDD of type SByte (recNum:71943)
[102] windDirMax of type Single (recNum:71943)
[101] windGustQCD of type Single (recNum:71943) (QCcheckNum:10)
[100] windGustQCR of type Int32 (recNum:71943)
[99] windGustQCA of type Int32 (recNum:71943)
[98] windGustDD of type SByte (recNum:71943)
[97] windGustChangeTime of type Double (recNum:71943)
[96] windGust of type Single (recNum:71943)
[95] windSpeedICR of type Int32 (recNum:71943)
[94] windSpeedICA of type Int32 (recNum:71943)
[93] windSpeedQCD of type Single (recNum:71943) (QCcheckNum:10)
[92] windSpeedQCR of type Int32 (recNum:71943)
[91] windSpeedQCA of type Int32 (recNum:71943)
[90] windSpeedDD of type SByte (recNum:71943)
[89] windSpeedChangeTime of type Double (recNum:71943)
[88] windSpeed of type Single (recNum:71943)
[87] windDirICR of type Int32 (recNum:71943)
[86] windDirICA of type Int32 (recNum:71943)
[85] windDirQCD of type Single (recNum:71943) (QCcheckNum:10)
[84] windDirQCR of type Int32 (recNum:71943)
[83] windDirQCA of type Int32 (recNum:71943)
[82] windDirDD of type SByte (recNum:71943)
[81] windDirChangeTime of type Double (recNum:71943)
[80] windDir of type Single (recNum:71943)
[79] altimeterQCD of type Single (recNum:71943) (QCcheckNum:10)
[78] altimeterQCR of type Int32 (recNum:71943)
[77] altimeterQCA of type Int32 (recNum:71943)
[76] altimeterDD of type SByte (recNum:71943)
[75] altimeter of type Single (recNum:71943)
[74] pressChange3HourICR of type Int32 (recNum:71943)
[73] pressChange3HourICA of type Int32 (recNum:71943)
[72] pressChange3HourQCD of type Single (recNum:71943) (QCcheckNum:10)
[71] pressChange3HourQCR of type Int32 (recNum:71943)
[70] pressChange3HourQCA of type Int32 (recNum:71943)
[69] pressChange3HourDD of type SByte (recNum:71943)
[68] pressChange3Hour of type Single (recNum:71943)
[67] pressChangeChar of type Int16 (recNum:71943)
[66] seaLevelPressureICR of type Int32 (recNum:71943)
[65] seaLevelPressureICA of type Int32 (recNum:71943)
[64] seaLevelPressureQCD of type Single (recNum:71943) (QCcheckNum:10)
[63] seaLevelPressureQCR of type Int32 (recNum:71943)
[62] seaLevelPressureQCA of type Int32 (recNum:71943)
[61] seaLevelPressureDD of type SByte (recNum:71943)
[60] seaLevelPressure of type Single (recNum:71943)
[59] stationPressureICR of type Int32 (recNum:71943)
[58] stationPressureICA of type Int32 (recNum:71943)
[57] stationPressureQCD of type Single (recNum:71943) (QCcheckNum:10)
[56] stationPressureQCR of type Int32 (recNum:71943)
[55] stationPressureQCA of type Int32 (recNum:71943)
[54] stationPressureDD of type SByte (recNum:71943)
[53] stationPressChangeTime of type Double (recNum:71943)
[52] stationPressure of type Single (recNum:71943)
[51] relHumidityQCD of type Single (recNum:71943) (QCcheckNum:10)
[50] relHumidityQCR of type Int32 (recNum:71943)
[49] relHumidityQCA of type Int32 (recNum:71943)
[48] relHumidityDD of type SByte (recNum:71943)
[47] rhChangeTime of type Double (recNum:71943)
[46] relHumidity of type Single (recNum:71943)
[45] dewpointICR of type Int32 (recNum:71943)
[44] dewpointICA of type Int32 (recNum:71943)
[43] dewpointQCD of type Single (recNum:71943) (QCcheckNum:10)
[42] dewpointQCR of type Int32 (recNum:71943)
[41] dewpointQCA of type Int32 (recNum:71943)
[40] dewpointDD of type SByte (recNum:71943)
[39] dewpoint of type Single (recNum:71943)
[38] temperatureICR of type Int32 (recNum:71943)
[37] temperatureICA of type Int32 (recNum:71943)
[36] temperatureQCD of type Single (recNum:71943) (QCcheckNum:10)
[35] temperatureQCR of type Int32 (recNum:71943)
[34] temperatureQCA of type Int32 (recNum:71943)
[33] temperatureDD of type SByte (recNum:71943)
[32] tempChangeTime of type Double (recNum:71943)
[31] temperature of type Single (recNum:71943)
[30] receivedTime of type Double (recNum:71943)
[29] reportTime of type Double (recNum:71943)
[28] observationTime of type Double (recNum:71943)
[27] elevation of type Single (recNum:71943)
[26] longitude of type Single (recNum:71943)
[25] latitude of type Single (recNum:71943)
[24] ICT of type SByte (ICcheckNum:55) (ICcheckNameLen:72)
[23] QCT of type SByte (QCcheckNum:10) (QCcheckNameLen:60)
[22] filterSetNum of type Int32
[21] dataProvider of type SByte (recNum:71943) (maxStaTypeLen:11)
[20] stationType of type SByte (recNum:71943) (maxStaTypeLen:11)
[19] numericWMOid of type Int32 (recNum:71943)
[18] homeWFO of type SByte (recNum:71943) (maxHomeWFOlen:4)
[17] stationName of type SByte (recNum:71943) (maxNameLength:51)
[16] handbook5Id of type SByte (recNum:71943) (maxStaIdLen:6)
[15] stationId of type SByte (recNum:71943) (maxStaIdLen:6)
[14] providerId of type SByte (recNum:71943) (maxProviderIdLen:12)
[13] secondsStage3 of type Int32 (recNum:71943)
[12] secondsStage1_2 of type Int32 (recNum:71943)
[11] lastInBin of type Int32 (nInventoryBins:24)
[10] firstInBin of type Int32 (nInventoryBins:24)
[9] isOverflow of type Int32 (recNum:71943)
[8] firstOverflow of type Int32
[7] globalInventory of type Int32
[6] inventory of type Int32 (maxStaticIds:30000)
[5] prevRecord of type Int32 (recNum:71943)
[4] invTime of type Int32 (recNum:71943)
[3] lastRecord of type Int32 (maxStaticIds:30000)
[2] staticIds of type SByte (maxStaticIds:30000) (totalIdLen:24)
[1] nStaticIds of type Int32

Thanks again,

Linda

 

May 19, 2011 at 4:07 PM

sorry for calling you 'leon', Dmitry.  I've got two conversations going on here at once.  Does that data look correct?

Linda

May 19, 2011 at 4:18 PM

That's ok. Yes, the file is a correct NetCDF file. The problem is in the DatasetViewer application: it hangs because it starts to explore which visualizations it can suggest and it takes very long time for such large amount of variables.

Your next actions depend on what you actually wish; if you just want to use these data from C# program, just use SDS API as I referred in the first reply - it will work properly. Else, if you need visualization of the data, you should first significantly decrease a number of variables. For this you would need to write an SDS-based program filtering variables in a dataset. If you need the latter, please inform me and I will provide you a sample program for this.

Dmitry

May 19, 2011 at 4:40 PM

Dmitry,

I am able to open the file successfully now in code.  However, I don't understand how the data is laid out and I haven't been able to find any source that gives a good description of how the netcdf files are formatted.  From the schema I sent earlier, how would I pull out all of the latitudes, for example.  (Latitude looks like it's the 25th element of an array)?

Thanks so much,

Linda

May 19, 2011 at 5:15 PM

Dmitry,

You can ignore my last question.  I figured it out.  This DLL is awesome.  I just have one more question.  I can't extract data when the datatype is SByte.  Other datatypes are working correctly, but all of the data that is of type SByte is returning the following error:  Cannot unambiguously identify a variable in the data set.  The statement I'm using to get the data is: 

SByte [] providerId = dataset.GetData<SByte[]>("providerId" );

 

Any ideas?

Thanks again,

Linda

May 19, 2011 at 9:14 PM

Dimitry,

I'm all set.  I needed to reference the field by index instead of name.  I don't see any duplicate names in the file, but it seems to be able to work properly if I reference the data by index.

Thanks for all of your help,

Linda

May 20, 2011 at 8:24 AM
lfarr wrote:

Other datatypes are working correctly, but all of the data that is of type SByte is returning the following error:  Cannot unambiguously identify a variable in the data set.  The statement I'm using to get the data is: 

 

SByte [] providerId = dataset.GetData<SByte[]>("providerId" );

Hi Linda,

I'm glad that you've succeeded with the library. The problem with accessing variable "providerId" by name caused by the wrong type specification: the variable is two-dimensional. That's why dataset couldn't find the variable; instead of

SByte [] providerId = dataset.GetData<SByte[]>("providerId" );

you should have written

SByte [,] providerId = dataset.GetData<SByte[,]>("providerId" );

Rank of the variable is seen from the output of the sds you've sent me before:

[14] providerId of type SByte (recNum:71943) (maxProviderIdLen:12)

Each line of the output is a single variable summary (a variable is an array with metadata) that shows the Variable ID (e.g., 14), Name (providerId), TypeOfData (SByte), and shape. Variable shape includes one dimension for vectors, two dimensions for matrices, and so on. Variable summary shows both name and length for each of the dimensions (e.g. providerID depends on recNum and maxProviderIdLen  and has lengths 71943 x 12). More about sds utility you can find here.

 

Regards, Dmitry.

May 20, 2011 at 2:12 PM

Dmitry,

I don't know how I missed that.  I must have looked at that schema a hundred times.  Ideally, I'd like to filter out much of this data based on one of the other fields.  For example,  I'd like to be able to filter based on station type and then end up with a multi-dimensional array indexed on stationid with its associated variables (i.e. longitude, latitude), etc.  So far I've been able to pull all the data into individual arrays, but not sure how to get it into a matrix.  Are there methods that I can use to do such a thing?

Thanks,

Linda

 

May 20, 2011 at 3:48 PM

Linda, to give an advice or a code sample I need more information about the file. I see that stationType variable is two-dimensional; it depends on recNum and maxStaTypeLen.

Then what is

stationType[i] 

which is hence an array of SByte? Are these bytes determine a station type?

Do you need to filter, e.g., only those temperatures which have certain station type, i.e. we take temperature[i] for such 0<=i<recNum so that stationType[i] == _station_type_?

May 20, 2011 at 6:15 PM

Dmitry,

I just received the schema from NOAA.  It seems that each record number represents stationid, provider name and time of weather reading.  So the 71,943 must be a place holder for a unique record.  So ideally, I want to end up with a matrix like so:

stationid   station_name  provider  latitude  longitude  temperature...

I'm unsure how to best do this using the library.  Getting each variable individually using GetData() and then using the same index to get each would probably work, but I'm unsure if this is the best way to do this using the tool.  I have the schema, but it's too big to post.  Do you have an email I can send it to?

Thanks,

Linda