MongoDB projection on an @Aggregation spring data

Question

I've been struggling with this for some days now. We've just started working with mongoDB and therefore my knowledge of it is extremely limited.

Anyhow here's what I want to do: we have a class called Loan like this:

@Document(collection = "Loans")
public class Loan{

    @Id
    private String id;
    private String type;

    private LoanApplication loanApplication;
    private LoanDecision loanDecision;

    private String loanAppId;
    private String loanDecisionId;

    private LocalDateTime receivedDate;
    private LocalDate leadExpiry;

    private Integer previousStatus;
    private Integer status;

    private Boolean isEligibleForRemove;

    private String timeStamp;

    private List<String> loanDecisions;

    private String activeLoanDecision;
    private String lenderReferenceNumber;
    private String pdfName;

Because we have to perform various operations when querying this collection we therefor created an aggregation (the aggregation works for we created in Mongo Compass and then exported it into our code) in the LoanRepository:

@Repository
@EnableMongoRepositories
public interface LoanRepository extends MongoRepository<Loan, String> {

@Aggregation(pipeline = {"{$match: {\r\n" + 
        "  'status':{$in: ?0},\r\n" + 
        "  'loanApplication.mortgage.requested.lenderSubmission.lenderProfile.lenderCode': ?1\r\n" + 
        "  }},{\r\n" + 
        "    $project: {\r\n" + 
        "        'applicationId': '$loanApplication.deal.applicationId',\r\n" + 
        "        'cominedLtv': '$loanApplication.deal.combinedLtv',\r\n" + 
        "        'receivedDate': '$receivedDate',\r\n" + 
                 //other fields to project
        "        }\r\n" + 
        "    }\r\n" + 
        "}, {\r\n" + 
        "    $unwind: {\r\n" + 
        "        path: '$applicants'\r\n" + 
        "    }\r\n" + 
        "}, {\r\n" + 
        "    $match: {\r\n" + 
        "        'applicants.primaryApplicantFlag': 'Y'\r\n" + 
        "    }\r\n" + 
        "}
         //other aggregate methods
"})
    public List<LoanProjection> findLeads(final List<Integer> leadsStatus, final String lenderCode);
}

The ideea is that after all the above operations we need to return just some fields from the whole Loans table (as can be seen in the $project step). I know that the return type of all the methods from this repository must be Loan and one solution would be to utilise spring projection. Thus i created the necessary projection interfaces (bellow you will find only the interfaces for the fields in the $project step):

public interface LoanProjection {   
    DealType getDealType();
    LocalDate getReceivedDate();
}

public interface DealTypeProjection {
    String getApplicationId();
    String getcombinedLtv();
}

The problem is: when the findLeads() repository method is called the following error occurs:

java.lang.NullPointerException: null
    at org.springframework.data.mongodb.core.mapping.MongoSimpleTypes$1.isSimpleType(MongoSimpleTypes.java:110) ~[spring-data-mongodb-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.data.mongodb.repository.query.StringBasedAggregation.isSimpleReturnType(StringBasedAggregation.java:119) ~[spring-data-mongodb-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.data.mongodb.repository.query.StringBasedAggregation.doExecute(StringBasedAggregation.java:81) ~[spring-data-mongodb-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.data.mongodb.repository.query.AbstractMongoQuery.execute(AbstractMongoQuery.java:101) ~[spring-data-mongodb-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:618) ~[spring-data-commons-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:605) ~[spring-data-commons-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80) ~[spring-data-commons-2.2.5.RELEASE.jar:2.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at com.sun.proxy.$Proxy130.findLeadsTest(Unknown Source) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_231]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_231]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_231]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_231]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at com.sun.proxy.$Proxy130.findLeadsTest(Unknown Source) ~[na:na]
    at com.filogix.lg.service.impl.LoanServiceImpl.retrieveLeads(LoanServiceImpl.java:261) ~[classes/:na]
    at com.filogix.lg.api.LoanController.getLeads(LoanController.java:49) ~[classes/:na]
    at com.filogix.lg.api.LoanController$$FastClassBySpringCGLIB$$54fe4fe8.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:120) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at com.filogix.lg.api.LoanController$$EnhancerBySpringCGLIB$$f692749f.getLeads(<generated>) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_231]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_231]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_231]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_231]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at com.filogix.lg.config.DecodingFilter.doFilter(DecodingFilter.java:35) ~[classes/:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.4.RELEASE.jar:5.2.4.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_231]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_231]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.31.jar:9.0.31]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_231]

I first started trimming my Aggregation logic thinking that maybe there is a problem with it (the aggregation works fine in Mongo Compass and Atlass) and i ended up having only the $match operator (something very basic) and still the error persisted.

@Aggregation(pipeline = {"{$match: {\r\n" + 
            "  'status':{$in: ?0},\r\n" + 
            "  'loanApplication.mortgage.requested.lenderSubmission.lenderProfile.lenderCode': ?1\r\n" + 
            "  }}"})
    public List<LoanProjection> findLeadsTest(final List<Integer> leadsStatus, final String lenderCode);

Then I wanted to see if by utilising the @Query annotation i would receive the same error. Funny enough with this method the projection worked!!!

@Query(value = "{$and: [{'status': {$in: ?0}}, {'loanApplication.mortgage.requested.lenderSubmission.lenderProfile.lenderCode': ?1}]}")
    public List<LoanProjection> findLeads(final List<Integer> leadsStatus, final String lenderCode);

Therefore my question is: is somehow spring projection not compatible with @Agregation or am i using it wrong (most likely)?

Answer 1

I think you would need to use a class instead of an interface and you need to add the @Id annotation in order to match the result from the aggregation

In your case is very hard to follow exactly what would be the id , because you didn't added any result from running the aggregation manually in mongo from CLI.

But if I try to guess then something like this probably would help you:

import org.springframework.data.annotation.Id;
// other imports

public class LoanProjection {

    @Id
    private DealType dealType;
    private LocalDate receivedDate;

    DealType getDealType() {
        return dealType;
    }

    LocalDate getReceivedDate() {
        return receivedDate;
    }
}

public class DealTypeProjection {
    @Id
    private String applicationId;
    private String combinedLtv;

    public String getApplicationId() {
        return applicationId;
    }

    public String getCombinedLtv() {
        return combinedLtv;
    }
}

Usually the result from an aggregation is very different than the Document used so you would need to have some unique identifier in order to map it to a new Projection . You could look at the result from aggregation like a new Document because this is the way spring matches the result with actual DTO.

More details here: https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongodb.repositories.queries.aggregation