SceneKit painting on texture with texture coordinates

Question

I have a Collada model that I load into SceneKit. When I perform a hittest on the model I am able to retrieve the texture coordinates of the model that was hit.

With these texture coordinates I should be able to replace texture coordinates with a color. So this way I should be able to draw on the model

Correct me if I am wrong so far.

I read a lot of articles till now but I just don't get my shaders right. ( Though I did get some funky effects ;-)

My vertex shader :

precision highp float;

attribute vec4 position;
attribute vec2 textureCoordinate;
attribute vec2 aTexureCoordForColor; //coordinates from the hittest

uniform mat4 modelViewProjection;

varying vec2 aTexureCoordForColorVarying; // passing to the fragment shader here
varying vec2 texCoord;

void main(void) {
    // Pass along to the fragment shader
    texCoord = textureCoordinate;
    aTexureCoordForColorVarying = aTexureCoordForColor; //assigning here
    // output the projected position
    gl_Position = modelViewProjection * position;
}

my fragment shader

precision highp float;

uniform sampler2D yourTexture;

uniform vec2 uResolution;
uniform int uTexureCoordsCount;

varying vec2 texCoord;
varying vec2 aTexureCoordForColorVarying;

void main(void) {

    // ??????????? no idea anymore what to do here
    gl_FragColor = texture2D(yourTexture, texCoord);
}

If you need more code please let me know.

Answer 1

First, shaders aren't the only way to draw onto an object's material. One other option that might work well for you is to use a SpriteKit scene as the material's contents — see this answer for some help with that.

If you stick to the shader route, you don't need to rewrite the whole shader program just to paint on top of the existing texture. (If you do, you lose things that SceneKit's program provides for you, like lighting and bump mapping. No sense reinventing those wheels unless you really want to.) Instead, use a shader modifier — a little snippet of GLSL that gets inserted into the SceneKit shader program. The SCNShadable reference explains how to use those.

Third, I'm not sure you're providing the texture coordinates to your shader in the best way. You want every fragment to get the same texcoord value for the clicked point, so there's little point to passing it into GL as an attribute and interpolating it between the vertex and fragment stages. Just pass it as a uniform, and set that uniform on your material with key-value coding. (See the SCNShadable reference again for info on binding shader parameters with KVC.)

Finally, to get at the main point of your question... :)

To change the output color of the fragment shader (or shader modifier) at or near a particular set of texture coordinates, just compare your passed-in click coordinates to the current set of texcoords that'd be used for the regular texture lookup. Here's an example that does that, going the shader modifier route:

uniform vec2 clickTexcoord;
// set this from ObjC/Swift code with setValue:forKey:
// and an NSValue with CGPoint data

uniform float radius = 0.01; 
// change this to determine how large an area to highlight

uniform vec3 paintColor = vec4(0.0, 1.0, 0.0);
// nice and green; you can change this with KVC, too

#pragma body

if (distance(_surface.diffuseTexcoord.x, clickTexcoord.x) < radius) {
    _surface.diffuse.rgb = paintColor
}

Use this example as a SCNShaderModifierEntryPointSurface shader modifier and lighting/shading will still be applied to the result. If you want your paint to override lighting, use a SCNShaderModifierEntryPointFragment shader modifier instead, and in the GLSL snippet set _output.color.rgb instead of _surface.color.rgb .