Just as a round up of my Rotation adventure.

First thing was to set the config.txt. in boot to adjust the entire system to my needs. My config as it stands right now is bellow:

disable_splash=1
hdmi_ignore_edid_audio=1
disable_audio_dither=1
hdmi_force_hotplug=1
config_hdmi_boost=7
hdmi_drive=1
disable_overscan=1
display_rotate=1
framebuffer_width=1200
framebuffer_height=1600
hdmi_mode=87
hdmi_group=2
hdmi_cvt=1920 1200 60

Some of this reduces my display to 3:4. Note that I had to max my HDMI output because my display wouldn't have it any other way. It's a bit old I think.

Next. I just found information on controlling advMame:

http://advancemame.sourceforge.net/doc-advmenu.html

Now to confuse the hell out of everything! Adjusting the game system settings using the TAB control within a game has revealed something to me. It looks like my issue is that my games are 4:3 in a vertical orientated 3:4 area. For games in a 4:3 ratio this isn't really an issue. But as soon as we get a tall 3:4 game things don't behave as I want. When adjusting the games VIDEO settings I realised that what I'm looking at is a 3:4 display area (with black edges left/right) being displayed within a 4:3 display area (with black edge top/bottom) which is displayed within a 3:4 display area (with black top/bottom); the last being set by my config file. Get all that?

In other words. The system is using 3:4 area of the display. But the emulator is using 4:3 area of that.

So far I am able to set the display area (3:4 in this case) using the config file, and the games ratio/resolution; but not the middle one. I have tried adjusting some of the obvious settings (to me anyway) in the two .advance directories. Each containing two files. Through tests I found one to have a handful of settings that seemed to affect the emulator but others that didn't. Changing the x/y to 3/4 on all of them had no effect.

Oh maybe I'm wrong... Looks like the 3:4 games are themselves rendering in a 4:3 area with black edges left/right.

Is someone more familiar with these configs be of help?

Another prevailing issue is having the rotated display i'm getting lower quality video performance. Specifically diagonal video tearing. Looks a bit like vsync. I read this is to do with memory and I did try changing this but either I failed to understand what I needed to do or it didn't help.

I've had the exact same issue in a cocktail table build I have. The dirty solution I have found is to not rotate the entire view via config.txt but to rotate each emulator individually. This makes Advance Mame run as it should... BUT... the piplay game lists are all 90 degrees off.
Not what you want to hear and surely something you've tried... just wanted you to know your woes are not alone.

Yea. Meant to ask this.