From the share tab on the openstreetmap page, I can export a map view as HTML e.g.:
Embedding an <iframe>
into GitHub-flavored Markdown (GFM) is not supported. Here's the official stance on it from their specs:
6.11 Disallowed Raw HTML (extension)
GFM enables the tagfilter extension, where the following HTML tags will be filtered when rendering HTML output:
<title>
<textarea>
<style>
<xmp>
<iframe>
<noembed>
<noframes>
<script>
<plaintext>
[...] These tags are chosen in particular as they change how HTML is interpreted in a way unique to them [...], and this is usually undesireable in the context of other rendered Markdown content.
All other HTML tags are left untouched.
Similar to solutions mentioned for videos in other answers, you could embed a screenshot of your map, and make it a link to the URL of your OSM map section: