I am working on a Kafka Streaming service using Apache Spark 2.1 and Java 8. I use a nested for
loop to populate an ArrayList
with Topic/Partition pairs.
Is it possible to reduce this nested for
loop from O(N^2) using another methodology?
Here is the code:
private static List<TopicAndPartition> createTAndPList(List<String> topics, List<String> partitions)
throws ConsumerException {
if (topics.size() != partitions.size()) {
throw new ConsumerException("Cannot create list with unequal number of topics and parititons,");
}
List<TopicAndPartition> topicsAndPartitions = new ArrayList<>();
for (int t = 0; t < topics.size(); t++) {
for (int p = 0; p < Integer.parseInt(partitions.get(t)); p++) {
topicsAndPartitions.add(new TopicAndPartition(topics.get(t), p));
}
}
return topicsAndPartitions;
}
FYI: I am prevented from using above Kafka 8 due to powers beyond my control (management.)
With your given code, it does not look possible to reduce the order.
However, there may be two small optimizations you can make.
Move topics.get(t) out of the inner for loop,
Do not recalculate the inner for loop termination condition every loop.
for (int t = 0; t < topics.size(); t++) { String topic = topics.get(t); int count = Integer.parseInt(partitions.get(t)); for (int p = 0; p < count; p++) { topicsAndPartitions.add(new TopicAndPartition(topic, p));
You are calling topics.get and Integer.parseInt(partitions.get(t)) t*p times instead of just t times. The change of topics.get() will probably not do much, but moving something out of the inner loop like this is a pretty common optimization.
Finally, do you really need them all in a list? Or can you generate them dynamically where you actually need them?
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.