Is there an API to modify the general browser UI in Chrome extensions or WebExtensions? For example I would like to modify the tab bar so that it can display multiple rows o
No, there is nothing in either Chrome extensions or WebExtensions that allows you to modify the browser UI to the extent you describe. The ability of such add-ons to modify the browser UI is significantly less than Firefox add-on developers (and users) are used to.
In the browser UI, you can change:
Those are the extent of the changes which you can make to the Browser UI.
You can also create a UI that exists within pages (using content scripts (Chrome)), within its own window (windows.create({type:'panel'}) (Chrome)), or a tab (tabs.create() (Chrome)), but such things are not actually a part of the browser UI.
If you are coming from a Firefox add-on background, this will often require you to re-think how you present your add-on's user interface in order to fit within this much more limited paradigm.
You could create an API using a WebExtensions experiment which would do what you desire. If adopted, such an API would be migrated into Firefox and become a standard part of WebExtensions. This should allow you to make whatever changes you desire.
For Firefox, you can add significant styling using a theme (a different type of add-on, which is just styles).
A theme I use, LittleFox, changes the stock browser bar (and many other changes throughout the browser) from looking like:
to look like:
Complete themes are being removed from Firefox with Firefox 57 (scheduled for release on 2017-11-14). The basics of the new methodology have been released, but it's still being developed (as is all of WebExtensions). However, as with the change from legacy extensions to WebExtensions, the new themes will be vastly less capable.
Alternately, you could create a different type of extension (e.g. Add-on SDK) with which you could make arbitrary changes. While a much wider range of browser UI modifications are available through the Add-on SDK, to make completely arbitrary changes requires accessing the DOM for the browser directly, which you can do from an Add-on SDK extension. As you are probably aware, these other extension types are scheduled for elimination from release versions of Firefox with Firefox 57, which is scheduled for 2017-11-14.
If you are wanting it for personal use, you can add styles to your profile's userChrome.css, which will allow you to style the browser UI. There is also a userContent.css file which will apply styles to content.
In my answer to "Firefox how to get dark themed searchbar-results" I show modifications to userChrome.css which will make the searchbar results be dark themed to match the Developer Edition theme.
Chrome also has themes, but they appear to be much more limited than what was available for Firefox. However, they are comparable to the new methodology for Firefox themes.