I am using a recursive flood fill algorithm in javascript and I am not sure how to avoid exceeding the max call stack size. This is a little project that runs in the browser.
Flood fill is a problematic process with respect to stack size requirements (be it the system stack or one managed on the heap): in the worst case you will need a recursion depth on the order of the image size. Such cases can occur when you binarize random noise, they are not so improbable.
There is a version of flood filling that is based on filling whole horizontal runs in a single go (https://en.wikipedia.org/wiki/Flood_fill#Scanline_fill). It is advisable in general because it roughly divides the recursion depth by the average length of the runs and is faster in the "normal" cases. Anyway, it doesn't solve the worst-case issue.
There is also an interesting truly stackless algorithm as described here: https://en.wikipedia.org/wiki/Flood_fill#Fixed-memory_method_(right-hand_fill_method). But the implementation looks cumbersome.