Google Maps Using Touch Events That Don't Target It

Question

I am trying to build a web application that has true multi touch capabilities. For example, I want to be able to drag a container around the screen while also zooming/panning on a google map at the same time. The problem that I am having is that if I am touching somewhere else on the page (not on the map) and then I touch on the map with another finger, the map acts as if both fingers were touching the map. This results in trying to pan on the google map with one finger turning into zooming in/out of the map because I am touching somewhere else on the page.

I am using Hammer.js for my other multi touch events on the page. I have tried putting an overlay over the top of the map and manually calling google maps api functions to move the map, which worked okay, but there are many features of the map that I will be missing if I use an overlay and manually pan and zoom.

I know that there is a list of all the different touch events that occur on the webpage, but I am not sure how to implement that list to solve this issue, if it is even possible.

EDIT: For more reference, you can recreate my problem by going to the google maps documentation. https://developers.google.com/maps/documentation/javascript/examples/map-simple

If you are on a touchscreen device, touch the map to start panning like normal, then touch some white-space on the page with another finger. If you remove your finger that was on the map, you can still control the position with the finger that is not even touching it.

Someone asked for code for reference, this is as close as I got to finding where the issue was. This prints out 'not correct' if I first touch in the map with one finger then off the map with another, but I am not sure where to go from there. Also, it does not print out if I touch outside of the map first, then in the map.

this.map.addListener('mousedown', function(e) {
  if(e.va.target.className == 'company-header') {
    console.log('not correct')
  }
});