One of the definitive libraries to do panorama stitching is Panorama Tools. You can either port or call from Perl.
Note that your specification is at odds. Unless you images are 100% rectilinear (i.e., taken 1:1 by an imager the same size as the image) you MUST compensate for the lens distortion. To accurately stitch photos together (pixel by pixel) the image needs compensating distortion.
