SELECT item_id, item_name, catid
FROM ( SELECT item_id, item_name, catid,
CASE WHEN @catid != catid THEN (@rownum := 1) + !(@catid := catid) ELSE @rownum := $rownum + 1 AS rownum
FROM ( SELECT item_id, item_name, i.catid
FROM items AS i
INNER JOIN categories AS c
ON i.catid = c.catid
ORDER BY i.catid ASC) AS h, (SELECT @catid := NULL, @rownum := NULL) AS var
HAVING rownum <= 5
LIMIT 20) AS h2
ORDER BY item_id
Devart's solution looks right, but I have a bit of a pickle understanding it fully. Mine may not make much more sence to others, but it does to me.
EDIT: Understand Devart's solution now, and it is quite brilliant, actually. :)