It seems like there may be a serious issue in the Gmail REST API.
This is not a bug in the API and is documented at https://developers.google.com/gmail/api/v1/reference/users/history/list
Specifically "History IDs increase chronologically but are not contiguous with random gaps in between valid IDs. Supplying an invalid or out of date startHistoryId typically returns an HTTP 404 error code. A historyId is typically valid for at least a week, but in some circumstances may be valid for only a few hours. If you receive an HTTP 404 error response, your application should perform a full sync."
So you are likely just using a historyId that is outside the range of history stored (it's not stored indefinitely, that would be very expensive). It's only stored long enough for syncing clients to need it (e.g. a week or so).
If you just need to sync from a certain point forward then just use the historyId returned from getProfile. The historyId on a message was when the message was last updated, which could be months or years ago, longer than history is stored.
See also the sync'ing guide: https://developers.google.com/gmail/api/guides/sync