Better dataformat than .mat v7.3 for import from Matlab 2016 to Mathematica 11?

Question

I am trying to import data from Matlab 2016a/2016b-prerelease to Mathematica 11 with local storage

• preferable complete documentation
• expansible to big data so binary; now, 1-100 GBs, but TBs needed later
• to convert from Mathematica 11 to Matlab 2016 - is also a plus if the data format can be a bijection; so understanding to read the data format in Matlab 2016a/b and Mathematica 11
• ...

Data and Quality Assurance

• Full data is xy plane which has 650 000 x 650 000 points so dimensions are about a square matrix.
• Take any subset of the data etc 6500 x 6500 for your examples.
• Save datafile and .tiff image.

Steps

1. Specify any test -data (presenting a 2D-image), and Specify filenames of .tif and datafile files
2. Quality assurance that you get a figure in .tif
3. Store data in your chosen dataformat
4. Quality assurance that you can reload datafile back into Matlab
5. Import datafile in Mathematica
6. Open/process datafile in Mathematica

Differential solutions

1. Challenges with Matlab's v7.3

   • follows HDF5 unconventionally without complete documentation; so you cannot do directly Attempt 1 below
     • I do not understand how unconventiol the dataformat really is so please propose also a stable method to work with v7.3
   • not sure about binary and expansibility to big data
   • Related: How to get Matlab data imported with the same dimensions? , How does import data from matlab into mathematica? or how to import matlab code that can be run in mathematica? , MAT-File Level 5 File Format , ...

2. Challenges of automatic approaches

   • I think not suitable because I want local storage, etc here .

3. Matlab's v7 .mat deprecated against v7.3

4. Matlab's v5 and v4 .mat - I think deprecated against v7.3

My attempts of Work flow for Diff Condition (1)

Some specification: specify datafile as .mat file of v7.3

Attempt 1

• Matlab

   # Specify test data here time=0:0.001:1; potential=sin(time); C = spectrogram(potential); C=reshape(C,1,[]); C=nthroot( abs( C(1,1:1001) ), 1); hFig=figure(); hax=axes(hFig); imagesc(time,potential,C); filename=fullfile('/home/masi/Images/test'); filenameMat=fullfile('/home/masi/Images/test.mat'); export_fig(filename, '-tif', '-q101', '-a1', '-m1', '-RGB', '-nofontswap', '-nocrop', '-transparent', '-dpng', hax); save(filenameMat,'time', 'potential', 'C', '-v7.3'); 

• Mathematica where #1-2 both succeed with import of many variables

   (* http://mathgis.blogspot.fi/2010/09/tips-import-matlab-mat-files.html *) (* https://mathematica.stackexchange.com/a/10589/9815 *) (* #1 Succeeds; select specific data sets *) mma = Import["~/Images/test.mat", {"HDF5", "Datasets", "/time"}]; (* #2 Succeeds: Out {"/C", "/potential", "/time"} *) mma = Import["~/Images/test.mat", {"HDF5", "Datasets"}]; (* Output: {{1.}, {1.5}, {2.}} *) 

• Output: steps (1-4) succeeds but import of datafile (step 5) fails in Mathematica 11, see the error message above.

Reading the data in Mathematica where Flatten is used to remove one set of braces because one set too much

(* https://stackoverflow.com/a/16834090/54964 *)
SetDirectory["Desktop"]
a = Import["m.mat"] ;
(* https://mathematica.stackexchange.com/a/97252/9815 *)
a=Partition[Flatten[a], 5000]

(* Output fails: {} *)

Studying agentp's answer

He is using simply a square matrix. I have the data in three variables: time , potential and C , fitting imagesc() 's parameters.

1. Do square matrix of the vectors time mx 1 and potential nx 1. How can you apply the vector C in the square matrix A ? I do not understand the mathematics here sufficiently to answer the question myself.

    # time's dimensions mx 1 # (potential')'s dimensions 1 xn time=0:0.001:1; potential=sin(time); A = time' * potential; # Output: A is mxn matrix, which is as as expected. # C is vector 1 xm here. C = spectrogram(potential); C=reshape(C,1,[]); C=nthroot( abs( C(1,1:1001) ), 1); 

2. How can you convert the square matrix A(C) back to those three variables? - - A(C) is about the square matrix where the vector C has been applied on the square matrix A . I do not understand the mathematics behind it to create the result.

3. How can you keep those pieces of data separate as one binary? - - This may not possible but I want to understand the standards currently.

Matlab: 2016a, 2016b prerelease
Mathematica: 11
OS: Debian 8.5
Related: Is there a way to import the results or data from Matlab to Mathematica automatically?

Answer 1

an example of raw binary file exchange from matlab to mathematica:

matlab:

mat = [ pi 2*pi 3*pi ; 1 sqrt(2)  sqrt(3)  ]
f=fopen('out.bin','w')
fwrite(f,size(mat))
fwrite(f,mat,'double')
... # repeat for however many matrices we need to write
fwrite(f,size(mat2))
fwrite(f,mat2,'double')
...
fclose(f)

mathematica:

f = OpenRead["out.bin",   BinaryFormat -> True];
size = BinaryReadList[f, "Integer8", 2];
mat = Transpose@ArrayReshape[
         BinaryReadList[f, "Real64",Times@@size],
         Reverse@size];
(* repeat as needed to read multiple matrices *)
Close[f];
MatrixForm@mat

note the Reverse and Transpose are needed because matlab writes the data in in column major order. You could alternately do fwrite(f,transpose(mat),'double') when you write.

note also this assumes a square array. If you wanted to handle multidimensional arrays you'd also need to write length(size) to the file and so on.

for completeness, go back like this:

f = OpenWrite["out.bin",  BinaryFormat -> True];
BinaryWrite[f, Dimensions[mat], "Integer8"];
BinaryWrite[f, Transpose[mat], "Real64"];
Close[f]

..

f=fopen('out.bin','r')
sz=transpose(fread(f,2))
mat=fread(f,sz,'double')
fclose(f)