Creating texture in Vertex shader and passing to Fragment to achieve smudging brush with Metal?

Question

I am trying to add a smudge effect to my paint brush project. To achieve that, I think I need to sample the the current results (which is in paintedTexture ) from the start of the brush stroke coordinates and pass it to the fragment shader.

I have a vertex shader such as:

vertex VertexOut vertex_particle(
                 device Particle *particles [[buffer(0)]],
                 constant RenderParticleParameters *params [[buffer(1)]],
                 texture2d<half> imageTexture [[ texture(0) ]],
                 texture2d<half> paintedTexture [[ texture(1) ]],
                 uint instance [[instance_id]])
{
    VertexOut out;

Drawing a fragment shader such as:

fragment half4 fragment_particle(VertexOut in [[ stage_in ]],
               half4 existingColor [[color(0)]],
               texture2d<half> brushTexture [[ texture(0) ]],
               float2 point [[ point_coord ]]) {

Is it possible to create a clipped texture from the paintedTexture and send it to the fragment shader?

paintedTexture is the current results that have been painted to the canvas. I would like to create a new texture from paintedTexture using the same area as the brush texture and pass it to the fragment shader.

The existingColor [[color(0)]] in the fragment shader is of no use since it is the current color, not the color at the beginning of a stroke. If I use existingColor, it's like using transparency (or a transfer mode based on what math is used to combine it with a new color).

If I am barking up the wrong tree, any suggestions on how to achieve a smudging effect with Metal would potentially be acceptable answers.

Update: I tried using a texture2d in the VertexOut struct:

struct VertexOut {
    float4 position   [[ position ]];
    float  point_size [[ point_size ]];
    texture2d<half> paintedTexture;
}

But it fails to compile with the error:

vertex function has invalid return type 'VertexOut'

It doesn't seem possible to have an array in the VertexOut struct either (which isn't nearly as ideal as a texture, but it could be a path forward):

struct VertexOut {
    float4 position   [[ position ]];
    float  point_size [[ point_size ]];
    half4 paintedPixels[65536];
}

Gives me the error:

type 'VertexOut' is not valid for attribute 'stage_in'

Answer 1

It's not possible for shaders to create textures. They could fill an existing one, but I don't think that's what you want or need, here.

I would expect you could pass paintedTexture to the fragment shader and use the vertex shader to note where, from that texture, to sample. So, just coordinates.