"What constitutes derivative work?" is not a question a programmer can answer. See for example What is the difference between an “aggregate” and other kinds of “modified versions”? and the GNU Licenses FAQ in general.
Regarding your specific case, Algorithms are usually not subject to copyright law, but (sadly) to patent law.
The way that these gray areas typically work is, that you are open about it, talk to the original authors, maybe work out a deal. It is very unlikely that such a case would end up in court, since neither party can gain very much from it. If it does end up in court, lawyers and judges will be the ones to decide, not programmers.
