We are trying to have a transactional table that only takes new records inserted on a regular basis.
This simple table requires us to continuously add new records to it
I also ran into this issue. After investigation, it seemed to be that it was not the data that was locked per se, but the clustered index that was being modified as a result of the insert. Since the clustered index resides on the data pages, the associated data rows are locked as well.
this locking behavior is a feature of SQL Server. With 2005 and above, you can use row level versioning (which is what is used by default on Oracle) to achieve the same result & not block your selects. This puts extra strain on tempdb because tempdb maintains the row level versioning, so make sure you accommodate for this. To make SQL behave the way you want it to, run this:
ALTER DATABASE MyDatabase
SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE MyDatabase
SET READ_COMMITTED_SNAPSHOT ON
This is completely standard behaviour in SQL Server and Sybase (at least).
Data changes (Insert, update, delete) require exclusive locks. this causes readers to be blocked:
With an exclusive (X) lock, no other transactions can modify data; read operations can take place only with the use of the NOLOCK hint or read uncommitted isolation level.
With SQL Server 2005 and above, they introduced snapshot isolation (aka row versioning). From MS BOL: Understanding Row Versioning-Based Isolation Levels. This means a reader has latest committed data but if you then want to write back, say, then you may find the data is wrong because it changed in the blocking transaction.
Now, this is why best practice is to keep transactions short. For example, only process what you need to between BEGIN/COMMIT, don't send emails from triggers etc.
Edit:
Locking like this happens all the time in SQL Server. However, locking for too long becomes blocking that reduce performance. Too long is subjective, obviously.
This exists in MySQL too. Here's the logic behind it: if you perform a SELECT on some data, you expect it to operate on an up-to-date dataset. That's why INSERT results in a table-level lock (that is unless the engine is able to perform row-level locking, like innodb can). There are ways to disable this behaviour, so that you can do "dirty reads", but they all are software (or even database-engine) specific.