REALLY "sluggish" video drawing on X10SRL-F via IPMI VGA

ljwobker

New Member
Oct 12, 2017
22
4
3
44
I've put together a Linux/KVM system based around an X10SRL-F motherboard, and I'm stumped on what I can only describe as "extremely slow or sluggish video". What I mean by this is that when the system boots, the console messages that appear on the screen scroll excruciatingly slowly, and this adds something like 90-120 seconds to the boot process. My background is in routers and switches that use serial consoles, and the only way I can describe it is like watching a 2400 baud terminal scroll up and down -- it's so slow that you can almost read every line as it appears. Once the machine boots into the graphical desktop, the video is STILL very sluggish, you can visually see that it takes a long time to redraw or resize windows or paint the initial screens of any application that loads.

I don't *think* it's related to the boot/console parameters in any way, because it behaves the same way regardless of whether it's booting (i.e. "console") or already in the graphical desktop.

This is my first IPMI capable board, so I'm pretty sure there is something in between the main system and the IMPI video that's slowing things down, but I don't have the first idea where to look for this. I couldn't come up with a better way to describe in words the difference between the "fast" video and the "slow" video, so I just recorded a couple of short video clips and put them up on google drive: it's very obvious by watching and comparing them how much difference there is in how fast the video is drawn.

X10SRL-F video - Google Drive

Interestingly enough, once I get the system booted, I can do a kexec-reboot (error10/kexec-reboot) and once the system comes back via that approach, it's WAY more responsive. I can't figure out if there's some kind of driver/accelerator that doesn't get loaded the first time around or something like that? I could almost convince myself that if it needed to get booted into the desktop to load the driver this would make sense, but even the first shutdown (i.e. once you've gotten into the Linux desktop) it's still very slow.
To break it down step by step with speeds, it's something like this:

  1. cold boot the machine. IPMI, BIOS, all that stuff operates at normal speed/responsiveness.
  2. once the main OS (Ubuntu 18.04.1) starts loading, the boot console messages get very sluggish/laggy and the system crawls along through the boot process. It's important for me to note that the video is synchronous, it's NOT like the system is booting normally and the video is just dropping a bunch of frames along the way... because the video is so slow it bogs down the entire boot process -see logs below... (google drive video link: you can really see the impact at about the 25 second point)
  3. The main desktop loads, but the video is still super laggy and windows take forever to paint. (video)
  4. if I do a regular reboot (or shutdown) the behavior stays the same... even the shutdown console is still really slow (video)
  5. BUT, if I do a kexec-reboot (bounce just the kernel) then it comes back a lot faster. (video after kexec-reboot)
  6. after the kexec-reboot when the desktop loads, it's quite responsive and snappy! (video)

On this machine, one of the last messages to hit syslog as the system boots is the various ports in the bridge going into forwarding state... you can see there's almost a three minute difference between when the video is "good" and things move quickly compared to when it's "bad" and they move really slowly...


Jan 25 12:02:12 lwobker-vms kernel: [ 15.804309] bridge1: port 1(eno1) entered forwarding state
Jan 25 12:02:16 lwobker-vms kernel: [ 19.868160] bridge1: port 2(ens2) entered forwarding state
Jan 25 12:02:18 lwobker-vms kernel: [ 21.640799] bridge1: port 3(vnet0) entered forwarding state

Jan 25 12:03:16 lwobker-vms kernel: [ 16.784198] bridge1: port 1(eno1) entered forwarding state
Jan 25 12:03:19 lwobker-vms kernel: [ 20.284144] bridge1: port 2(ens2) entered forwarding state
Jan 25 12:03:21 lwobker-vms kernel: [ 22.265671] bridge1: port 3(vnet0) entered forwarding state

Jan 25 14:52:47 lwobker-vms kernel: [ 12.746976] bridge1: port 1(eno1) entered forwarding state
Jan 25 14:52:53 lwobker-vms kernel: [ 19.062700] bridge1: port 3(vnet0) entered forwarding state
Jan 25 14:52:54 lwobker-vms kernel: [ 19.900282] bridge1: port 2(ens2) entered forwarding state




Jan 25 14:28:08 lwobker-vms kernel: [ 154.754752] bridge1: port 1(eno1) entered forwarding state
Jan 25 14:28:08 lwobker-vms kernel: [ 160.380175] bridge1: port 2(ens2) entered forwarding state
Jan 25 14:28:39 lwobker-vms kernel: [ 193.922273] bridge1: port 3(vnet0) entered forwarding state

Jan 25 14:36:09 lwobker-vms kernel: [ 153.044693] bridge1: port 1(eno1) entered forwarding state
Jan 25 14:36:09 lwobker-vms kernel: [ 158.652181] bridge1: port 2(ens2) entered forwarding state
Jan 25 14:36:40 lwobker-vms kernel: [ 194.898956] bridge1: port 3(vnet0) entered forwarding state

Jan 25 14:49:21 lwobker-vms kernel: [ 146.123308] bridge1: port 1(eno1) entered forwarding state
Jan 25 14:49:21 lwobker-vms kernel: [ 152.252138] bridge1: port 2(ens2) entered forwarding state
Jan 25 14:49:44 lwobker-vms kernel: [ 187.832669] bridge1: port 3(vnet0) entered forwarding state


Help!?!?!
 

dandanio

Active Member
Oct 10, 2017
141
48
28
I am not surprised at all. Not sure where you got the idea of using a KVM/ipmi for anything close to production, even worse in a graphical environment. Switch to a regular video and you will have great performance.
 

RageBone

Active Member
Jul 11, 2017
295
77
28
@ljwobker did you update the ipmi fw to the newest version?

Is it the same if you aren't watching over iKVM? Boottimes whise I mean.
 

ljwobker

New Member
Oct 12, 2017
22
4
3
44
I am not surprised at all. Not sure where you got the idea of using a KVM/ipmi for anything close to production, even worse in a graphical environment. Switch to a regular video and you will have great performance.
So I never said it needed to be super fast, nor would I plan on doing anything like this in a "real" production environment. I was just curious as to whether this was a known / expected thing and/or if it could be mitigated. And I wouldn't consider it a graphical environment, it's just the boot console (which I believe in modern linux distros is usually actually graphical, but I think you get the point?)
 

ljwobker

New Member
Oct 12, 2017
22
4
3
44
@ljwobker did you update the ipmi fw to the newest version?

Is it the same if you aren't watching over iKVM? Boottimes whise I mean.
ARGH. Upgrading the IPMI firmware fixed it. I was told by the person I bought the board from that it had been updated to the latest, but I never actually verified this. Sorry for the spam ;-)
 
  • Like
Reactions: RageBone

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,242
420
83
Sorry for the spam ;-)
T'was at least an interesting read for me - and an interesting failure mode (although I didn't watch the videos). I've never heard of a failure mode like this that actually apparently blocked the OS from functioning properly.

Out of curiosity, do you know which IPMI version you running that suffered from this problem before you upgraded?
 

ljwobker

New Member
Oct 12, 2017
22
4
3
44
Yeah it was really odd, like there was some horrible block/sync call in the frame buffer. As far as the IPMI versions, I think from something like 3.27 up to 3.65.