在C中读取具有许多变量的.dat文件：动态构建格式字符串？

Question

So I am seeking to convert .dat files from the Core Wave data of the Survey of Income and Program Participation. There are 15 waves with somewhere on the order of 300,000-400,000 observations each. Each of these waves has an identical layout with 1023 variables. The .dat file structure does not contain any delimiters, so each observation is a string of characters (mostly digits, but some "-" signs). I have parsed the data dictionary with Python to capture the variable names, start positions, and size. I also wrote a script that converts these files to DataFrames to facilitate use with pandas. The problem is, my script has been running for over 36 hours, and I need to speed this up dramatically.

Enter C. I am quite new to it, but all I want to do at this point is convert these .dat files to .csv. Yes, I could do a better job optimizing the Python using a couple packages that Anaconda has developed, but this seemed like a nice contained task to introduce myself to C. I have written a few small scripts to test this conversion, and here is the most relevant one:

/*This script tests a conversion from .dat to .csv*/
#include <stdio.h>

main()
{
  /*Declare variables*/
  int var1;
  int var2;
  int var3;
  int i;

  /*Initialize pointers for source (test) and desitination (test_out) files*/
  FILE *test; /* test = test.dat file pointer */
  FILE *test_out; /* test_out = test.csv file pointer */

  /*Attempt to open the source file, and if it doesn't work, tell me about it */
  if ((test=fopen("O:\\Analyst\\Marvin\\scrap\\test.dat","r"))==NULL)
    printf("Source file could not be opened\n");
  /*Attempt to open the source file, and if it doesn't work, tell me about it */
  else if ((test_out=fopen("O:\\Analyst\\Marvin\\scrap\\test.csv","w"))==NULL)
    printf("Destination file could not be opened\n");
  /*If it does open, initiate read*/
  else{
    /*Write the headings to disk in the destination file*/
    fprintf(test_out,"%10s%10s%10s\n","Var 1,","Var 2,","Var 3");
    /*Initialize variables with the first row of data in the source file*/
    fscanf(test,"%3d%4d%3d",&var1,&var2,&var3);
    /*Initialize line counter*/
    i=1;
    /*For the remaining data lines in the source file...*/
    while (!feof(test)) {
      /*...write the last line's values to destination file...*/
      fprintf(test_out,"%9d%1s%9d%1s%10d\n",var1,",",var2,",",var3);
      /*...load the current line's values into the variable addresses...*/
      fscanf(test,"%3d%4d%3d",&var1,&var2,&var3);
      /*...and print (stout) then iterate the counter*/
      printf("%20s %d\n","Writing line #",i++);
    }
    /*Once the EOF is reached, close the source and destination files*/
    fclose(test);
    fclose(test_out);
  }
  /*Return 0 if everything has gone smoothly*/
  return 0;
}

The source file contains the following values:

0123456789
0123456789
0123456789
0123456789
0123456789

It outputs the following to the destination:

Var 1,    Var 2,     Var 3
   12,     3456,       789
   12,     3456,       789
   12,     3456,       789
   12,     3456,       789
   12,     3456,       789

So, this is all well and good, but I am dealing with over 1000 variables. Not only is the idea of writing out a format string that long vomit-inducing, but it also strikes me as bad practice >> way to many keystrokes. Given that I have the layout in an easily parseable file, I figure there has got to be some programmatic solution to this issue on both the input and output sides.

There are just an unruly number of C-related input questions on SO. The sample I have reviewed never seems to get at this question of parsing such a large volume of variables with a known layout. Someone please enlighten me.

Answer 1

To read 1023 integers of various widths, suggest creating an array indicating the fixed width of each integer. Use that width as an index to the desired format. Use long long to handle digit width up to 19 or so.

#define MAX_WIDTH 4
int ReadLine(FILE *inf, const unsigned char *width, long long *data, size_t n) {
  static const char *format[MAX_WIDTH+1] = { 
      "%lld", "%1lld", "%2lld", "%3lld", "%4lld" };
  size_t i = 0;
  for (i = 0; i < n; i++) {
    if (width[i] > MAX_WIDTH) Handle_BadWidth(); 
    if (fscanf(inf, format[width[i]], &data[i]) != 1) {
      return 1;  // FAIL
    }
  }
  int eol = fgetc(inf);
  if (eol != '\n' && eol != EOF)
    return 2;  // FAIL
  return 0;
}

// Sample use - error handling omitted.
#define N (3)
int main(void) {
  FILE *inf = fopen("test.dat","r");
  long long data[N];

  unsigned char width[N];
  FormWidths("input.csv", width, N);
  return ReadLine(inf, width, data, N); 
}

OP explained that a .CSV file existed that the widths could be read in the form of "VarN,width"

int FormWidths(const char *fname, unsigned char *width, size_t n) {
  FILE *inf = fopen(fname, "r");
  if (!inf) return 1;  // error
  char buf[100];
  fgets(buf, sizeof buf, inf);  // Use this line if file has table header
  size_t i;
  for (i=0; i<n; i++) {
    if (fgets(buf, sizeof buf, inf) == NULL) break;
    int index, w;
    if (sscanf(buf, "Var%d ,%d", &index, &w) != 2) break;
    if (index != i) break;
    // Adjust MAX_WIDTH to larger values like 19 and then update the format table
    if (w < 1 || w > MAX_WIDTH) break; 
    width[i] = w;
  }
  fclose(inf);
  if (i != n) return 1; // Did not read .csv as expected
  return 0;
}