如何使用SpringBoot从~100个客户数据库中查询和检索结果？

Question

I have the scenario where I have to run 10 different queries on 100 customer databases with similar structure and then push the results to an ElasticSearch cluster for analysis. All the database connections are configured inside my applications.properties file. I decided to use Spring-Boot for the project and Java High Level Rest Client as the ElasticSearch API. However I found that in Spring-Boot I have to create an entity class for every entity and create a separate class and methods for each database connection. I am new to Spring Boot and am also not understanding the concepts of entitymanager or rowmapper. It is quite different from Java connection-statement-query-resultset format. Kindly help me

I have tried creating this Database configuration class where I tried to configure a single database reading from the properties file. I have created the basic datasource() and jdbctemplate() methods

package elasticsearch;

import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;

import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;

@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(
    entityManagerFactoryRef = "entityManagerFactory",
    basePackages = { "elasticsearch" }
)
public class DatabaseConfig {


@Bean(name = "dataSource")
@ConfigurationProperties(prefix = "primary.datasource")
public DataSource dataSource() {
    return DataSourceBuilder.create().build();
}


@Bean(name = "jdbcTemplate")
public JdbcTemplate jdbcTemplate(@Qualifier("dataSource") DataSource         dataSource){
    return new JdbcTemplate(dataSource);
}


@Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean entityManagerFactory(
        EntityManagerFactoryBuilder builder,
        @Qualifier("dataSource") DataSource dataSource) {
    return builder
            .dataSource(dataSource)
            .packages("elasticsearch")
            .persistenceUnit("elasticsearch")
            .build();
}


@Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(
        @Qualifier("entityManagerFactory") EntityManagerFactory
                entityManagerFactory
) {
    return new JpaTransactionManager(entityManagerFactory);
}

}

I do not want to create an entity class for every object because the queries and also the results can vary. Also I do not want to create rowmappers because I have already written Json mappers for the rows retrieved to push it to elasticsearch

Answer 1

I am new to Spring Boot and am also not understanding the concepts of entitymanager or rowmapper.

You don't show any of the details of your schema, but I don't think you need both JPA and JdbcTemplate . I'd recommend one or the other.

My preference would be JdbcTemplate .

JPA/Hibernate is overkill and complexity that you don't need. JdbcTemplate will be fine if you are comfortable with writing SQL SELECTs.

It is quite different from Java connection-statement-query-resultset format.

Not really. JdbcTemplate helps you with the boilerplate, but it's still JDBC underneath.

100 client databases will require 100 sets of URL and credentials, one for each. That is a lot of configuration. You can't get around that.

The problem is intractable if the schemas are not identical for all customers.

I would separate the two problems: querying for customer data and pushing to Elastic Search.

You only need a single RowMapper per query if the schema and query are identical for all customers.

I think a single repository/data access object can be used. You only need write and test it once, but you need to instantiate a new instance at runtime for each database connection.