What's the best way to play video in SceneKit?

Question

I've tried a number of different techniques to play videos in SceneKit, but none of them seem to work correctly.

I've tried assigning an AVPlayerLayer to an SCNMaterial.diffuse.contents and that simply fails to display any video frames at all, though the sounds plays fine.

I've also tried assigning a SpriteKite SKScene containing an SKVideoNode child node to an SCNMaterial.diffuse.contents, and that plays well, though ( as noted by others on StackOverflow ) it leaks megabytes of memory every time the video plays, until finally iOS kills the app.

Finally, since its now supported in iOS 11.0+ and recommended by @mnuages and others, I've tried assigning an AVPlayer directly to an SCNMaterial.diffuse.contents and while that plays and doesn't leak memory, unfortunately it has a number of bad visual artifacts (even on iOS 12.1.2). In particular, as other developers have noted , SceneKit logs the following numerous times to the console every time an AVPlayer instance is assigned to a material and it starts playing:

[SceneKit] Error: Could not get pixel buffer (CVPixelBufferRef)

The log messages might be livable, except with each error, the video frame in the scene is displayed as all white, which causes flashing for a fraction of a second each time a new video plays.

A worse visual artifact of this technique is what I'll call the "psychedelic rainbow" effect which randomly adds a glittering color noise patern to adjacent nodes while the AVPlayer is assigned to the material (regardless of whether it's playing or paused). It's kinda cool looking (see the attached screen capture), but not quite what we'd like for our app.

Does anyone have know how to make one of these techniques work reliably?

Here's some code I've used for testing assigning an AVPlayer directly to a material:

class GameViewController: UIViewController {

var plane: SCNPlane!
var planeNode: SCNNode!
var playerObserver: NSKeyValueObservation?
var playerCompletionObserver: NSObjectProtocol?

override func viewDidAppear(_ animated: Bool) {
    super.viewDidAppear(animated)

    guard let scnView = self.view as? SCNView else {return}

    guard let scene = SCNScene(named: "art.scnassets/scene.scn") else {return}
    scnView.scene = scene

    plane = SCNPlane(width: 10, height: 10)
    planeNode = SCNNode(geometry: plane)
    planeNode.position.y = 10.0
    scene.rootNode.addChildNode(planeNode)

    scnView.isPlaying = true
    scnView.loops = true
    addPlayer()
}

func addPlayer() {

    guard let url = Bundle.main.url(forResource: "Clover", withExtension: "mov") else {return}
    let asset = AVURLAsset(url: url)
    let playerItem = AVPlayerItem(asset: asset, automaticallyLoadedAssetKeys: [#keyPath(AVAsset.tracks), #keyPath(AVAsset.duration)])
    let player = AVPlayer(playerItem: playerItem)

    print("\(url.lastPathComponent) \(player.error?.localizedDescription ?? "loaded")")

    playerObserver = playerItem.observe(\.status, options:  [.new, .old], changeHandler: {[weak self] (playerItem, change) in
        guard let strongSelf = self else {return}

        if playerItem.status == .readyToPlay {

            strongSelf.playerObserver?.invalidate()
            strongSelf.playerObserver = nil

            if let material = strongSelf.plane.firstMaterial {

                material.diffuse.contents = player
                let videoSize = playerItem.asset.tracks(withMediaType: .video).first?.naturalSize ?? CGSize(width: 640, height: 480)
                material.diffuse.contentsTransform = SCNMatrix4Translate(SCNMatrix4MakeScale(Float(videoSize.height / videoSize.width), 1, 1), 0.25, 0, 0)
                material.isDoubleSided = true
            }

            strongSelf.playerCompletionObserver = NotificationCenter.default.addObserver(forName: NSNotification.Name.AVPlayerItemDidPlayToEndTime, object: playerItem, queue: nil) {[weak self](Notification) in
                guard let strongSelf = self else {return}
                NotificationCenter.default.removeObserver(strongSelf.playerCompletionObserver!)
                strongSelf.addPlayer()
            }

            player.play()
            }
        })

}

}

Update : I've now tried dozens of combinations of automaticallyLoadedAssetKeys and .preroll, but so far, none of them did anything to minimize the white flashing problem. I did however, come up with something that almost eliminates the problem.

I noticed that if the opacity of the node was zero, the SceneKit video renderer wouldn't do anything, so I changed this line of code:

player.play()

to add a little delay:

planeNode.opacity = 0
DispatchQueue.main.asyncAfter(deadline: .now() + 0.3) {
     player.play()
     self.planeNode.opacity = 1
}

Adding delays like this always seems like a hack to me, but I just haven't found anything else that works at all. Doe anyone have a better fix?

Answer 1

I used SKVideoNode and SKScene then set plane.firstMaterial?.diffuse.contents = skScene. I also set a observer for the AVPlayer when it play to end time, it will seek to the begin and play again.