How to define which diagonals belong to concave polygon

Question

I have problem with triangulation algorithm:

In concave polygons some diagonals are not in polygon. I have double connected list of edges,vertex and know which vertex have concave angle.

There is 2 cases:

A) diagonal intersection with edge or another diagonal. B) diagonal dont intersection any, but is out of polygon.

How i can define diagonals in A and B cases?

Answer 1

Consider this sample, where vertices are sorted in clockwise order :

Look at the red diagonals, outside the polygon. Vertex '4' is at right of line '3-5'. Vertices '3, 4' are at right of line '2-5'.

We can infer a law: If a line ij has some vertex k with i < k < j that is at right of the line, then that line is external to the polygon.

Notice that if you reverse the order of the vertices, then the you must also use "at left" instead of "at right".