This may be subjective, I don\'t know: I have this problem, which I\'m kind of equating to the \"what language for this project?\" question, since I can\'t seem to solve it.
Ultimately I would look at it this way: a DSL (like any programming language) is a set of lexical tokens and idioms that you can use to explain how to do something. Implicit within that set of tokens is a set of idioms.
The idea of any language is that you select an appropriate one that makes it easier to express expected algorithms.
A good example are reporting languages (eg Natural). That's because reports have common idioms like groups, breaking conditions and so on. Anyone who has done that in a general purpose language would tell you that it can be really tedious to do things like break conditions but it's certainly possible.
So the question you should be asking is this: are there any common idioms that would be tedious, difficult or extremely verbose to express in a general purpose language? You should also be asking: is there a particular subset of functionality that you are most interested in that would benefit by making it a language concept? This could be things like vectors and matrices for linear algebra (with the appropriate operations) or the same for integral/differnetial calculus or differential equations.
What you'd really like to have in the end with a DSL is where, say, a 500 line program could be express as 50-100 lines of your DSL and this applies for most "expected" algorithms. I guess it's a little like normalization in data modelling where the goal is to remove repeating groups. Writing the same 10-20 lines of code with a little variation suggests a common idiom.
As for a library of functions, that is a more limited view of the same thing (ultimately) except that a library of function is merely a package of different behaviours. That ssumes you define a library of functions as just that and don't instead include complex object hierarchies, closures and so on. If you do then the line between a "library of functions" and a DSL is blurred.
I would think of a library of functions as, say, the set of math functions that exist in most modern languages (sin, cos, square root and so forth). So I guess it comes down to:
I would think that the purpose of a DSL was to abstract away details and enable a programmer (or implementor) to be productive by keeping their head in the domain, so to speak.
If you want to explain the details of a collection of algorithms, I would stick with one of the usual suspects (C++?).
An important consideration is whether your audience already use a single language predominately. If so, the choice has been made for you!
Another thing you might want to consider, is that by choosing a popular mainstream language, your book is applicable to a wider audience, without first having to 'learn' a DSL.
I think it will depend on writing a DSL and functional programming languages, as, in my experience, physicists tend to be fairly competent in Fortan or Matlab.
You may want to try using those languages for your examples, as it will enable you to get into algorithms as it may help those that are more technical.
As Mitch Wheat mentioned, DSLs are good when you want to hide the details from the user.
You can use a DSL to decouple a set of related operations from the calling language.
If you want to shield your users from the complexity of the general-purpose language (or restrict their access to that language because they could abuse it), design a DSL.
But if you want the operations to be used in a manner integrated with the calling language, then use a library of functions (or other operators).
You have to weigh the degree to which you can provide a cleaner notation against the fact that nobody will start out knowing the notation you use. In theory, you might be able to follow the notation this group uses in general so well that it's "intuitively obvious" -- but unless they're different from most physicists, that probably won't work well. Many use things like Greek and/or Hebrew characters that aren't present on a typical keyboard.
If you do use a DSL, are you prepared not only to implement it, but spend a good part of your life maintaining it? Do you have the resources to go beyond the language proper, and provide the kind of infrastructure for it to become a useful development tool? If it's going to be used for real development, it'll probably need a debugger. You might also want to consider either a new IDE (if it's really different) or tailoring an existing product to work (e.g. create a specialized mode for emacs or a plug-in for Eclipse).
If you do invent a DSL, how likely is it that people will really use it? If there's already something thoroughly entrenched, displacing it will probably be quite difficult. Creating a new language that fills a vacuum is a lot easier than replacing something that's already in use. On the other hand, if there really is a vacuum, this may indicate that there's just not much market -- your audience may not be particularly interested in becoming software developers.
You could choose a DSL to match exactly the domain you're dealing with. Programmers can be comfortable with a general purpose language and applying it in many situations. Your non-programmers may benefit from a language that is directly relevant to them. It can also be easier for them to understand if keywords and concepts match their expertise.
Like most things, it's a trade-off. A DSL should be easier for your audience, but it could well be more work for you.