How to get data from 2-D into 1-D array

Jun 11, 2011 at 6:15 PM

Hi,

How do I get the data from a 2-dimentional array into a 1-dimensional array?  I have a 2-d data as follows:

using(var latlonDs = sds.DataSet.Open(latlonFilePath + "?openMode=open")
{
  //Open the source csv file
  var csvDs = sds.Dataset.Open(csvFilePath + "?openMode=open");

  //Get all data from the source dataset
  double[,] pcpval = csvDs.GetData<double[,]>("amountofprecip");

  //Put data into "Precip" column of destination dataset ( latlonDs)
  latlonDs.PutData<double[]>(1, pcpval);  //*********ERROR FLAG HERE**********/
}

Jun 14, 2011 at 6:55 AM

Hi,

SDS cannot do the conversion automatically, at least because there are two different ways (for 2d array) how it can be done: first index changes first, or second index changes first.

But this can be done using simple C# code:

using(var latlonDs = sds.DataSet.Open(latlonFilePath + "?openMode=open")
{
  //Open the source csv file
  var csvDs = sds.Dataset.Open(csvFilePath + "?openMode=open");

  //Get all data from the source dataset
  double[,] pcpval = csvDs.GetData<double[,]>("amountofprecip");

  // Convert 2d-array into 1d
  int n = pcpval.GetLength(0);
  int m = pcpval.GetLength(1);
  double[] pcp1d = new double[n*m];
  for(int i = 0, k = 0; i < n; i++)
   for(int j = 0; j < m; j++)
      pcp1d[k++] = pcpval[i,j];

  //Put data into "Precip" column of destination dataset ( latlonDs)
  latlonDs.PutData<double[]>(1, pcp1d); 
}

Regards,

Dmitry.

Jun 15, 2011 at 5:16 PM

Hi,

Thank you so much.  This works perfectly.  You wouldn't believe the strategies I took to get the precip values into the lat/long table.  The data are so huge that I'm either running out of memory or it was just taking too long to process due to the loops (from 11 minutes to over two hours for one file); I need to do this all at once for daily data (about 28-31 netcdf files; or for hourly (24 netcdf files for each day).  With MS Research SDS API, I can now process netcdf files and "bring it over" into ArcGIS.  For those who may be interested, these are the steps I followed:

1.  Using Microsoft Research SDS API, convert the NetCDF files into .csv files (this gives you the precipitation values).
2.  Using ArcGIS, export the AHPS polygon's attribute table (go to their website) into dbf and convert this into csv file (I used Excel for this).  You need the latitude and longitude fields for the entire Continental US and AHPS polygon has this data.
3.  Using the above code, insert the precip values into the polygon table's csv file.  (This is the fastest I've found at 17 to 20 seconds for one file.)  The polygon table's csv file will now become your working table.
4.  Remove the metadata from this table (so ArcGIS can recognize it).
5.  Using ArcToolBox, convert the csv file into dbf file; so you can get an Object ID for ArcGIS geoprocessing.
6.  Use "Make XY Event Layer" tool to get the X and Y fields.
7.  Use "Feature Class to Shapefile" tool to convert the layer into point shapefile.
8.  Define the projection (use Geographic Coordinate System first).
9.  Project to your desired projection (mine was Albers Equal Area).
10.  Use "Point to Raster" tool using Cell Assignment = Maximum; Cell Size = specify your cell size.  This took about 1 minute for each file.

And there you go.  You now have a raster to work with ArcGIS.

Again, thank you.

Marilyn