Is there a method to detect if the mouse position is inside a simple custom vertex shape.
Let’s take this image as an exemple:
The shape is in blue and the point we want to know if it is inside the shape is in red.
One possible solution is to triangulate your shape: the white dotted lines inside the shape on the figure. Now, the only thing remaining is to check if the point is inside either one of those triangles. If that’s the case then the point is also inside the shape.
You will find plenty of resources online to check whether a point is inside a triangle or not.
Another option is to cast a ray from outside the shape to the point and count how many times it intersect the shape. In the figure, I used an horizontal ray drawn in yellow.
For each segment of the shape, we can check if it intersects the ray (again there are plenty of resources online for this).
If the number of intersection is odd, then the point is inside the shape, else it’s outside.
Note that there are few edge cases to consider (intersection on the edge of the segment, segment parallel to the ray, …).
In both cases, you can do a quick check first to see if the point is inside the bounding box of the shape (in purple). If it is not, no need to check in more detail.