Accessing the slice by function in .NET

Feb 11, 2011 at 5:00 PM

Hi

I've just started using the SDS components and very impressed. I can open a netcdf file in VS2010 but I don't know how to access the slice by function that is available in the executable version of the Dataset viewer. Can you confirm how I access this dfunction and tell it what variable to slice by?

Chris

Feb 16, 2011 at 1:11 PM

I also just started using it.... i am doing something to implement the slices since documentation did not help.

but take a look at it and please provide me with some comments

http://147.102.85.132/Public/SDSnetCDF.rar
I also want to create a new NetCDF file from scratch ...  any chance that anyone can give a nice complete sample ? 

Feb 16, 2011 at 6:45 PM

Hi

Thanks for the reply. I've had a look at your code and you are obviously a more accomplanished developer than I and you are more familiar with the SDS objects which I have just come across. Your code works well and does exactly what I wanted to achieve! I will explore in more detail and let you know my comments.

 It is a shame that there is not more documentation as the library seems like it could be really useful for handling NETCDF files in .NET which is what I want to do.

Feb 17, 2011 at 8:46 AM

Hi thanks but this is simply not true, i was experimenting.

I am currently reading in detail more about the netcdf format (i am also intrested in CF-1.0 ) and how to create a new file with sds without errors. I am a confused about coordinate variables.

I also agree that the documentation is very little for what this tool can do.

Any other resources or documentation would be usefull. Keep in touch in case we are alone!

 

Feb 17, 2011 at 2:50 PM
Edited Feb 17, 2011 at 3:01 PM

Hello,

Simplest way to get a slice form a DataSet is to use Imperative Extensions API and its method DataSet.GetData:

using Microsoft.Research.Science.Data.Imperative;
...
// Opens a data set:
var ds = DataSet.Open("test.csv?openMode=open");        
// Gets a value from a scalar variable of type double with given name "myvar":
double[,,] val3d = ds.GetData<double[,,]>("myvar"); // Gets all data.
// To get a 1d-slice by reducing first two dimensions:
double[] val1d = ds.GetData<double[]>("myvar", 
     DataSet.ReduceDim(0), // First index is 0 and reduced from output array
     DataSet.ReduceDim(10), // Second index is 10 and reduced from output array
     DataSet.FromToEnd(0)); // Takes all elements along this dimension

See more details here.

Another (more complicated) way how you can get a slice as an array from a DataSet in your .NET code is to use DataSet.StrideVariable method. For example:

using (DataSet target = DataSet.Open("data.nc?openMode=open")) 
{     
     Variable<int> v2d = target.AddVariable<int>("test2d", new string[]{"x", "y"});
     v2d.PutData(new int[,] {{ 0, 1, 2}, {4, 5, 6}, {7, 8, 9}}); // v2d is a 2d-variable    
     
     // If we want to get a slice, we can add a special "virtual" variable using StrideVariable method,
     // and we specify source variable, origin vector, stride ("0" means "remove the dimension"), count which is how many
     // elements to take ("0" means "as much as available"):
     Variable<int> sv2d1 = target.StrideVariable(v2d, new int[,] { 1, 0 }, new int[] { 0, 1 }, new int[] { 1, 0 }, "slicedv2d");

     // Dataset target now has new variable named "slicedv2d"   
     // Dimension "x" is removed and sv2d1 is one-dimensional and contains {4,5,6}:
    Assert.IsTrue(Compare(sv2d1.GetData(), new int[] { 4, 5, 6})); // true
}

 See Scientific DataSet Reference.chm reference documentation installed in Start menu for details.

 Does it resolve your issue?

 Regards, Dmitry.

Feb 17, 2011 at 6:04 PM

Hi Dmitry,

I used  the first method in the sample above.

However the second one also is very intresting... which method is faster ?

I am going to use the library for loading and handling  netCDF "CF-1.0 compatible" files (http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/cf-conventions.html)

I am also intresting in netCDF CF-1.0 file creation. I guess it is possible to do this but some aspects like "Grid Mappings" and "Projections" maybe will become a problem

I just found out how to  add metadata to the file (not variables). I am also thinking to add support (not me alone) to dotspatial ( dotspatial.codeplex.com ) to use this library to read and create netCDF files.

Thanks for the support.

Angelos

Feb 17, 2011 at 6:54 PM
Thanks dimity, I will explore some more and the code samples and links are very useful. Do you know what the longer term plans are for this library - is it still active in terms of development and will it be updated?

Angelos, interested to hear you are also using DotSpatial. i have been using this with some hydrological modelling code I have developed in .net and I came across sds as I was looking to process climate inputs and output model results in netcdf and also view such files efficiently from my code as there will be a lot of outputs. A formal link between this and dot spatial could be very powerful. You can do some netcdf stuff with esri's Arcgis but an open source gis-netcdf approach that can be tailored to specific needs would be very versatile especially with all the climate change projections that will come out under cmip5 that everyone will want to process etc.
Feb 18, 2011 at 6:11 PM

A small step for others maybe but I've now managed to slice and dice my data as required using the code samples you both supplied - many thanks.

For the record the vb.net code below was my test code that allowed me to process and visualise the timeslice.

Dim Timestep As Integer
        Dim Timeslice(,) As Short
        Dim LatSlice() As Single
        Dim Lonslice() As Single

        Dim LongRange, LatRange As Microsoft.Research.Science.Data.Range

        Using v = DataSet.Open("EOBSrrDec2010.nc?openMode=open")

            'Test slice to try
            Timestep = 1
            LongRange = DataSet.Range(115, 175)
            LatRange = DataSet.Range(95, 140)

            'Slice data
            Timeslice = v.GetData(Of Short(,))(3, DataSet.ReduceDim(Timestep), LatRange, LongRange)
            Lonslice = v.GetData(Of Single())(2, LongRange)
            LatSlice = v.GetData(Of Single())(1, LatRange)

            'Create a dataset for display purposes
            Dim v2 As DataSet
            v2 = DataSet.Open("msds:memory?&name=test2")

            'Map dimensions to appropriate variables
            v2.Add(Of Single())("longitude", Lonslice, "Londim")
            v2.Add(Of Single())("latitude", LatSlice, "Latdim")
            v2.Add(Of Short(,))("Daily rainfall", Timeslice, {"Latdim", "Londim"})

            'Copy metadata for original source
            For x As Integer = 0 To v2.Variables.Count - 1
                For Each y In v.Variables(x + 1).Metadata
                    v2.PutAttr(x, y.Key, y.Value)
                Next y
            Next x

            'Process missing datavalues
            v2.Variables(0).MissingValue = v.Variables(1).Metadata("_FillValue")

            'Commit data to dataset
            v2.Commit()

            'Now display timeslice
            DataSetViewerControl1.DataSet = v2
            DataSetViewerControl1.IsXRulerVisible = True
            DataSetViewerControl1.IsYRulerVisible = True
        End Using


Mar 9, 2011 at 10:25 AM

Any ideas on how to find the index  this when coordinate variables(lattitude and longitude ) are not 1d array each but instead 2d arrays ?

To create a small sample let's suppose we have some netCDF file with the following variables:

lat(y,x) 
lon(y,x) 
mydataVariable(t,h,y,x) 

Suppose we know h (height), t (elevation)  and we want to find the value the array has in a position (lat,lon)

how can we find the index of x and y if we know lat and lon ?

I am trying to create a raster that has an equal cell size.

 

More info on Two-Dimensional Latitude, Longitude, Coordinate Variables : http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/ch05s02.html

Mar 10, 2011 at 9:21 AM
As a workable, albeit, slow solution I would use the array.index of function to scroll through all the members of both the lat(x,y) and lon(x,y) arrays until you get a match for the x any y values from both........this might be slow where you have large arrays but i don't know of a better method off-hand but then i'm not a programmer but a researcher who uses code to do specific tasks so my skills are limited