需要使用lambda映射多个文件中的减少行

Question

I have a number of files that I was to read line by line. Each line contains a url followed by a timestamp, followed by a number of tags

I have a class called Link that parses each line and provides static methods to get

Link::url
Link::timestamp
Link::tags  where this returns a List of tagstrings

The urls can be duplicated in the file along with the tags. I need to read the lines from all the files, collect the tags for each url and eliminate the duplicates Then write the results to an output file in the format url tag1, tag2, tag3

I am able to do this with Java 7 using map/reduce but cannot figure out how to do this using lambda expression. I am told that it can be done in one line of code?

This is what I have. I am stuck after the filter. I think what I want to do is create a map with a key that is the url and a TreeMap where the TreeMap would contain all the unique tags. I just don't know how to write this any help would be appreciated.

public static void tagUnion() throws Exception {   
    Stream<Path> fstream = Files.list(Paths.get(indir));
    fstream.forEach(path -> {
        Stream<String> lines;
        try (Stream<String> entry = Files.lines(path)) {
            entry
            .filter(s -> !s.isEmpty())
            .map(Link::parse)
            .filter(map -> inDate(map.timestamp()));
            // this is where I’m stuck
        } catch (IOException e) {
            e.printStackTrace();
        }
    });
}

Answer 1

I would suggest using Stream::flatMap instead. this method maps a each object inthe stream to different stream, all of the same type, and combines them into a single stream you can continue working on. For example:

Files.list(somePath)
        .flatMap(Files::lines)
        .filter(s -> !s.isEmpty())
        .map(Link::parse)
        .filter(map -> inDate(map.timestamp()));

Now to do what you are asking requires writing a method that will handle the link and parse it into the line you want it to be.

Finally, to collect a stream of strings into one string with a delimiter( be it newline or comma), there is a method for that:

String csvLine = stream.collect(Collectors.joining(",");

Answer 2

I'm not sure there is enough information here to confidently answer your question, but here is a stab at it anyway.

Given that you have something similar to this:

@FunctionalInterface
interface IOFunction<T, R>
{
  R apply(T t) throws IOException;

  public static <T, R> Function<T, R> unchecked(IOFunction<T, R> f)
  {
    return v -> {
      try {
        return f.apply(v);
      } catch (IOException e) {
        throw new UncheckedIOException(e);
      }
    };
  }
}

You might be able to get what you want with something like this:

  public static Map<String, Set<String>> tagUnion(String indir)
      throws IOException {
    try (Stream<Path> fstream = Files.list(Paths.get(indir))) {
      return fstream
          .flatMap(IOFunction.unchecked(Files::lines))
          .filter(s -> !s.isEmpty())
          .map(Link::parse)
          .filter(link -> inDate(link.timestamp()))
          .collect(Collectors.toMap(Link::url, link -> new TreeSet<>(link.tags())));
    } catch (UncheckedIOException e) {
      throw e.getCause();
    }
  }

The complication here is that Files.lines(...) throws a checked IOException which precludes its use directly in a stream pipeline.

---

OK, based on your comments, you want a groupingBy(...) operation. It's a little more code to collect the contents of a bunch of List<String> into a Set<String> .

  return fstream
      .flatMap(IOFunction.unchecked(Files::lines))
      .filter(s -> !s.isEmpty())
      .map(Link::parse)
      .filter(link -> inDate(link.timestamp()))
      .collect(Collectors.groupingBy(Link::url,
          Collectors.mapping(Link::tags,
              Collector.of(
                  () -> new TreeSet<>(),
                  (s, l) -> s.addAll(l),
                  (s1, s2) -> {
                    s1.addAll(s2);
                    return s1;
                  }))));

For Java 9, this could be simplified to something like:

  return fstream
      .flatMap(IOFunction.unchecked(Files::lines))
      .filter(s -> !s.isEmpty())
      .map(Link::parse)
      .filter(link -> inDate(link.timestamp()))
      .collect(Collectors.groupingBy(Link::url,
          Collectors.flatMapping(link -> link.tags().stream(), Collectors.toSet())));

Answer 3

Thanks for the help. I was able to solve the problem a different way using a TreeMap

    // create array of files in the directory
    // make sure the files are json files only
    File[] files = new File(indir).listFiles(new FileFilter() {
        @Override
        public boolean accept(File pathname) {
            //System.out.println(pathname.getName());
            return pathname.getName().toLowerCase().endsWith(".json");
        }
    });

    // exit if no json were found
    if (files.length == 0) {
        System.out.println("No JSON files found in directory " + indir);
        System.exit(0);
    }

    // map each line to a String(url), Set(tags)
    Map<String, Set<String>> tagMap = new TreeMap<>();


            lines.filter(s -> !s.isEmpty())
                    .map(Link::parse).forEach(l -> {
                    HashSet hs = new HashSet(l.tags());
                    if (tagMap.containsKey(l.url())) {
                        tagMap.get(l.url()).addAll(hs);
                    } else {
                        tagMap.put(l.url(), hs);
                    }
            });
        }

    }


        // write the output to the specified file
        writeOutput(tagMap, false);