简体   繁体   English

使用Java 8流时的性能问题(列表中的大量数据)

[英]Performance Issue when using Java 8 Stream (Bulk of Data in a List)

I have the following 2 codes/logics in a single method: 我在单一方法中有以下2个代码/逻辑:

Code 1: Returns around 10K++ of data 代码1:返回大约10K ++的数据

    //myObjectRepository extends the spring CrudRepository (hibernate)
    final List<MyObject> myObjectList =
            myObjectRepository.getObjectsByIdAndReportDate();

Code 2: Filters the 10k++ of data 代码2:过滤10k ++数据

    //Only return valid objects
    final List<MyObject> objectsWithStrings = myObjectList
            //parallelStream() randomly throw an exception related to hibernate
            .stream()
            .filter(d -> d.getMyObjectStrings() != null && !d.getMyObjectStrings().isEmpty())
            .collect(Collectors.toList());

Note : MyObject is a hibernate entity with the following structure: 注意 :MyObject是具有以下结构的休眠实体:

@JsonBackReference
//Jackson – Bidirectional Relationships. To avoid "Could not write content: Infinite recursion (StackOverflowError)"
@OneToMany(mappedBy = "myObject")
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
private List<MyObjectString> myObjectStrings;

It is working as it is but I am encountering the following in Code 2: 它按原样工作,但是我在代码2中遇到以下问题:

  • a performance problem in my Code 2 . 我的代码2中性能问题
  • a random hibernate problem whenever I used the parallelStream(), thus, I opted to use the sequential stream() instead 每当我使用parallelStream()时,都会出现一个随机休眠问题 因此,我选择使用顺序stream()代替

In this case, is there a way for me to optimize code 2 so it can process the data or filter the data faster? 在这种情况下,我是否有办法优化代码2 ,使其可以处理数据或更快地过滤数据?

Snippet of the hibernate error I was encountering: 我遇到的休眠错误代码段:

2018-03-05 15:35:28.431 ERROR 10056 --- [nio-8082-exec-1] e.b.MyServiceExceptionControllerAdvice : An generic error is encountered [400] [Error accessing field [private java.util.Date my.entity.AnotherObjectPK.reportDate] by reflection for persistent property [my.entity.AnotherObjectPK#reportDate] : null]

org.hibernate.property.access.spi.PropertyAccessException: Error accessing field [private java.util.Date my.entity.AnotherObjectPK.reportDate] by reflection for persistent property [my.entity.AnotherObjectPK#reportDate] : null
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:43) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.tuple.component.AbstractComponentTuplizer.getPropertyValue(AbstractComponentTuplizer.java:58) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.ComponentType.getPropertyValue(ComponentType.java:419) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.ComponentType.getHashCode(ComponentType.java:242) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.CacheKeyImplementation.calculateHashCode(CacheKeyImplementation.java:57) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.CacheKeyImplementation.<init>(CacheKeyImplementation.java:53) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.DefaultCacheKeysFactory.staticCreateEntityKey(DefaultCacheKeysFactory.java:50) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.redis.hibernate5.strategy.ReadWriteRedisEntityRegionAccessStrategy.generateCacheKey(ReadWriteRedisEntityRegionAccessStrategy.java:54) ~[hibernate-redis-2.4.0.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.getFromSharedCache(DefaultLoadEventListener.java:644) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:595) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.returnNarrowedProxy(DefaultLoadEventListener.java:310) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:270) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1129) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1022) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:639) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.resolve(EntityType.java:431) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.nullSafeGet(EntityType.java:262) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.persister.collection.AbstractCollectionPersister.readElement(AbstractCollectionPersister.java:839) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.PersistentBag.readFrom(PersistentBag.java:95) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.CollectionReferenceInitializerImpl.finishUpRow(CollectionReferenceInitializerImpl.java:77) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:121) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:122) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:122) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:86) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:88) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:688) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1991) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:570) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:252) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:566) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:135) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:164) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:149) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:252) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:148) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.PersistentBag.isEmpty(PersistentBag.java:266) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at my.service.impl.MyObjectServiceImpl.lambda$getObjectsWithObjectStringByKeyAndReportDate$0(MyObjectServiceImpl.java:126) ~[classes/:na]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[na:1.8.0_121]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_121]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_121]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_121]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) ~[na:1.8.0_121]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) ~[na:1.8.0_121]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316) ~[na:1.8.0_121]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[na:1.8.0_121]
Caused by: java.lang.NullPointerException: null
at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57) ~[na:1.8.0_121]
at sun.reflect.UnsafeObjectFieldAccessorImpl.get(UnsafeObjectFieldAccessorImpl.java:36) ~[na:1.8.0_121]
at java.lang.reflect.Field.get(Field.java:393) ~[na:1.8.0_121]
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:39) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
... 53 common frames omitted

I have the following hibernate entities structure: 我有以下休眠实体结构:

MyObjectEntity : MyObjectEntity

@Data
@Entity
@Table(name = "MY_OBJECT_TABLE", schema = "MYSCHEMA")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
@IdClass(MyObjectEntityPK.class)
@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class MyObjectEntity implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@ManyToOne(fetch = FetchType.LAZY)
@Fetch(FetchMode.SELECT)
@JoinColumns(value = {
        @JoinColumn(name = "OBJECTID", referencedColumnName = "OBJECTID", insertable = false,
                updatable = false),
        @JoinColumn(name = "REPORT_DATE", referencedColumnName = "REPORT_DATE", insertable = false,
                updatable = false),
        @JoinColumn(name = "LASTGENERATION", referencedColumnName = "GENERATION", insertable = false,
                updatable = false)},
        foreignKey = @ForeignKey(value = ConstraintMode.CONSTRAINT))
private AnotherObject anotherObject;

@JsonBackReference
//Jackson – Bidirectional Relationships. To avoid "Could not write content: Infinite recursion (StackOverflowError)"
@OneToMany(mappedBy = "myObject")
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
private List<MyObjectString> myObjectStrings;

EmbeddedPrimary Key: MyObjectEntityPK 嵌入式主键: MyObjectEntityPK

@Data
class MyObjectEntityPK implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@ManyToOne(fetch = FetchType.LAZY, targetEntity = AnotherObject.class)
@JoinColumns(value = {
        @JoinColumn(name = "OBJECTID", referencedColumnName = "OBJECTID", insertable = false,
                updatable = false),
        @JoinColumn(name = "REPORT_DATE", referencedColumnName = "REPORT_DATE", insertable = false,
                updatable = false)},
        foreignKey = @ForeignKey(value = ConstraintMode.CONSTRAINT))
private AnotherObject anotherObject;

} }

AnotherObject entity: AnotherObject实体:

@Data
@Entity
@Table(name = "ANOTHER_OBJECT", schema = "MYSCHEMA")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
@IdClass(AnotherObjectPK.class)
@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class AnotherObject implements Serializable {
...
 }

Embedded Primary key: AnotherObjectPK 嵌入式主键: AnotherObjectPK

@Data
class AnotherObjectPK implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@Column(name = "OBJECTID", nullable = false, unique = true)
private Integer id;

@Id
@Column(name = "REPORT_DATE")
private Date reportDate;
}

In my opinion the problem is that you are trying to initialize a Lazy collection in your case myObjectStrings from multiple threads at the same time. 在我看来,问题是您试图同时从多个线程初始化myObjectStrings的Lazy集合。 In my opinion you need to initialize both 我认为您需要同时初始化

List<MyObject> myObjectList 

and

d.getMyObjectStrings()

In advance before any paralelism kicks in. In my first answer I asked you to initialize in advance myObjectList now I am telling you to initialize in addition d.getMyObjectStrings() 在任何并行处理开始之前提前。在我的第一个答案中,我要求您提前初始化myObjectList,现在我告诉您另外初始化d.getMyObjectStrings()

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM