In SQL Server 2008, you can create a filtered index, like:
CREATE UNIQUE INDEX indexName ON tableName(isbn) WHERE isbn IS NOT NULL
In earlier versions, create a calculated column which is the ISBN when it exists, and the primary key when the ISBN is null:
CREATE TABLE tableName (
pk int identity(1,1) primary key,
isbn int NULL,
nullbuster as (case when isbn is null then pk else 0 end),
CONSTRAINT dupNulls_uqX UNIQUE (isbn)
)
Copied from SQL Server UNIQUE constraint with duplicate NULLs, so this is in wiki mode :)