Almost,
d (distance) = (herbivor.x - water.x)² + (herbivor.y - water.y)²
That is simply the pythagorean theorem but without the square-root :
To compute the smallest one you need a variable to store the current smallest. In macro code in would be something like this :
int currentSmallest = height * height + width * width; // You initialize that variable with the biggest distance you can found AKA the diagonal length of your canvas
for each waterPixel in waterEdgePixels { //You go through all your water edge pixels
d = (creature.x - waterPixel.x)² + (creature.y - waterPixel.y)²; //You compute the distance to your creature
if (d < currentSmallest) { // If the distance is smaller than the smallest that you found before then you found a new smallest
currentSmallest = d;
}
}