如何处理stan中缺失的数据？

Question

I am a newbie to stan and I am implementing the probabilistic matrix factorization model.

Given a user-item rating matrix:

                       item
 user     1    3   NA   4     5    NA
          2    0    3   NA    1     5
          1    1    NA  NA    NA    0
          ....

How should I represent the observable data in the data block and the missing data for prediction in the parameter block?

Thank you in advance!

EDIT:

Now I am implementing the model as below:

pmf_code = """
data {

int<lower=0> K; //number of factors
int<lower=0> N; //number of user
int<lower=0> M; //number of item
int<lower=0> D; //number of observation
int<lower=0> D_new; //number of pridictor 
int<lower=0, upper=N> ii[D]; //item 
int<lower=0, upper=M> jj[D]; //user
int<lower=0, upper=N> ii_new[D_new]; // item
int<lower=0, upper=N> jj_new[D_new]; // user
real<lower=0, upper=5> r[D]; //rating
real<lower=0, upper=5> r_new[D_new]; //pridict rating

}

parameters {
row_vector[K] i[M]; // item profile
row_vector[K] u[N]; // user profile
real<lower=0> alpha;
real<lower=0> alpha_i;
real<lower=0> alpha_u;

}
transformed parameters {
matrix[N,M] I; // indicator variable
I <- rep_matrix(0, N, M);
for (d in 1:D){
    I[ii[d]][jj[d]] <- 1;
}
}
model {
for (d in 1:D){
    r[d] ~ normal(u[jj[d]]' * i[ii[d]], 1/alpha);
}

for (n in 1: N){
    u[n] ~ normal(0,(1/alpha_u) * I);
}
for (m in 1:M){
    i[m] ~ normal(0,(1/alpha_i) * I);
}
}
generated_quantities{
for (d in 1:D_new){
    r_new[d] <- normal(u[jj_new[d]]' * i[ii_new[d]], 1/alpha);
}
}
"""     

but got an No matches for: real ~ normal(matrix, real) error in this line of code:

for (d in 1:D){
    r[d] ~ normal(u[jj[d]]' * i[ii[d]], 1/alpha);
}

But the jj[d] should be a integer, denoting the id of user . And u[ int ] should be a row_vector has k factors and so is i[ii[d]] . The product of them should be a single real value, why stan said it was a matrix ?

Answer 1

There's a chapter in the Stan manual on how to deal with missing or sparse data. In this case, it's missing data. What you want to do is put it in long form (what R's reshape package calls melted form):

  int<lower=0> I;               // number of items
  int<lower=0> J;               // number of users
  int N;                        // number of observations
  int<lower=1, upper=I> ii[N];  // item 
  int<lower=1, upper=J> jj[N];  // user
  int<lower=0, upper=5> y[N];   // rating

Then, for each observation n , you have user jj[n] assigning the rating y[n] to item ii[n] .

There's an example of this in the IRT models in the regression section of the manual. But you have an ordinal outcome, which is a bit trickier. You could either do a direct ordinal logistic of some kind, probably hierarchical, or you could try to do something like a factor model (like the partial SVD everyone used for Netflix). There are also example of factor models in the manual --- you'd use those to generate the linear predictor for an ordinal regression.

Then, if you want to predict y[m] for some new combination of item i and user j , you can do that in the generated quantities block as a posterior predictive quantity. And you can do that either via sampling or via an expectation; there's an example of that in the change-point model in the latent discrete parameter chapter and also in the regression chapter on prediction.

Answer 2

Stan has neither a missing data symbol nor the ability to estimate discrete unknowns, so what you are proposing is almost impossible and not a great entry point for learning Stan. This is explained in the Stan User Manual.

In principle, you could pass in the non-missing data and a two-dimensional integer array that is 0 if the item is missing for a user and 1 if the item is observed for that user. Then you need to declare a latent utility for each user and item, constrain them to fall between the right two cutpoints if the data point is observed, and adjust for the absolute value of the derivative of the transformation you use to get the latent utility between the cutpoints. If the data point is missing, then the corresponding latent utility is unconstrained. This is essentially the data augmentation approach used by Gibbs samplers, although Stan is not a Gibbs sampler. Then, you specify your model for the latent utilities (constraining the scale of the errors to be 1) and hope for the best. Most likely there will be a lot of divergent transitions, which will require that you set adapt_delta quite close to 1 to eliminate them.

The closest thing we have to an example of this approach is a multivariate probit model but that is for the simpler case of binary outcomes.