I can already get a batch file to run when a user right clicks on a file type. How can I make it so that only one instance runs per highlighted group and gets all the files
The way it works seems to have been imposed on you by the shell, and there doesn't seem to be an easy way to solve this.
Some application would add its own menu item that would allow the app to be invoked differently (i.e. just once for the group) from how it is done generally (repeatedly for every selected item), while another one would employ the API to check its own presence and would redirect the 'open' request to its already running copy.
Batch files aren't meant for either of these. You would probably need a different tool. Even if you would like the main job to be done by the batch file, you'd still need a way to call the batch file for processing of the item list.