Stream of millions of objects takes too much memory

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. JaffaKetchup wants to to this question: ：

Explain and direct to a structure that appropriatley handles large amounts of small objects efficiently, as stated in the question.

I'm generating a load of coordinates (made of 3 numbers) within a geographical area. However, using Streams (which should be much more efficient than Lists), fills up the app's memory very quickly, as can be seen in this screenshot from Observatory.

I need a structure where events can go in, and be read out one by one, and when this happens, removed from the structure. As far as I understand, that is what a Stream is.

Here's the function that starts the Isolate that returns the stream of coordinate tiles. I'll omit the actual generator, as it's not important: it just sends a Coord to the SendPort .

static Stream<Coords<num>> _generateTilesComputer(
    DownloadableRegion region,
  ) async* {
    List<List<double>> serialiseOutline(l) => (l as List)
        .cast<LatLng>()
        .map((e) => [e.latitude, e.latitude])
        .toList();

    final port = ReceivePort();

    final tilesCalc = await Isolate.spawn(
      region.type == RegionType.rectangle
          ? rectangleTiles
          : region.type == RegionType.circle
              ? circleTiles
              : lineTiles,
      {
        'port': port.sendPort,
        'rectOutline': region.type != RegionType.rectangle
            ? null
            : serialiseOutline(region.points),
        'circleOutline': region.type != RegionType.circle
            ? null
            : serialiseOutline(region.points),
        'lineOutline': region.type != RegionType.line
            ? null
            : (region.points as List<List<LatLng>>)
                .chunked(4)
                .map((e) => e.map(serialiseOutline)),
        'minZoom': region.minZoom,
        'maxZoom': region.maxZoom,
        'crs': region.crs,
        'tileSize': region.options.tileSize,
      },
    );

    await for (final Coords<num>? coord in port
        .skip(region.start)
        .take(((region.end ?? double.maxFinite) - region.start).toInt())
        .cast()) {
      if (coord == null) {
        port.close();
        tilesCalc.kill();
        return;
      }
      yield coord;
    }
  }
}

This method's results get sent through a bunch of other methods in some scenarios, but for now, I just use .length to get the number of tiles in the stream.

How can I force the deletion of these values when they are read out?

Answer 1

Does this help a bit? It's your bottom bit.
The.batch method is used to read values in batches of 500, which can be changed to a different value if it is needed.
The count variable is used to keep track of the number of values processed, and when it reaches the limit, the port is closed and the isolate is killed.

    int count = 0;
    final limit = ((region.end ?? double.maxFinite) - region.start).toInt();
    await for (final Coords<num> coord in port
        .skip(region.start)
        .batch(500)) {
      if (count >= limit) {
        port.close();
        tilesCalc.kill();
        return;
      }
      count += coord.length;
      yield coord;
    }
  }
}