I am getting OutofMemoryException at runtime with the message \"Insufficient memory to continue the execution of the program.\". I am loading the images at the start of program.
You could probably try to allow your program to access more memory, but it would be a struggle. For a .NET application, the amount of memory is controlled by the processModel/memoryLimit setting in your machine.config file. Microsoft recommends that you set it no higher than 60%.
However, you are loading 50+ MB images, all at once, to display thumbnails (which are probably tiny in size). I suggest that this is where you make your change. You could load your images one by one, then generate the thumbnail and free the memory straight away.
In any case, having 50 MB+ images is not efficient when you are trying to show a thumbnail. Can't you just save the thumbnails and not have to generate them every time?
Is there any possibility to use more memory of the system or somehow other solution.
Switching to 64bit is the only simple option.
There is an underlying .NET per object 1GB limit (also applies to 64bit), but this isn't your problem. To create an object .NET needs there to be enough contiguous free memory in the process. Once you have a few very large (>250MB) large objects in process it is increasingly unlikely enough continuous address space is going to be available.
Options:
Also have a read of "Windows Internals" on how Windows manages memory, then on how the .NET GC manages memory for background. There will be no substitute for knowing what is going on when you are pushing so much data around. (Tools like VMMap will help, but only if you have a core understanding of how it all works.)