Category: Uncategorized

When RamDisks Get Full

I thought it might be worth sharing the outcome of some troubleshooting of some ESXi 5 servers running off SD cards, in case anyone else encounters similar issues.

When vSphere 4 proved capable of booting ESXi from a 4Gb SD card or USB stick it became a tempting option for saving money on storage with new deployments.  The HP Proliant G6s even came with an onboard SD card slot to make it easy to leave those disk slots empty.

However, ESXi 5 moved the goalposts, with USB or SD storage no longer suitable for persistent storage:  hosts without local storage now mount a 32Mb RamDisk to mount the scratch partition to.

After upgrading to ESXi 5 we saw several instances of host instability as the Ramdisk ram out of space, symptoms including :-

  • Changes to VM settings failing
  • VMotion to or from the server failing (so we couldn’t put the server into Maintenance Mode for a safe reboot)
  • Local logs not updating
  • Host reporting as disconnected status within vCenter

The VMs themselves continued to run on the host, just not able to manage the host properly.

As a short term fix to bring the server under control again – providing the Shell is enabled – we logged onto the shell, navigated to the /var/log folder and deleted any archived log files (ending .gz) and created a new, empty wtmp file (the file recording shell access).  That recovered enough Ramdisk space for the server to become manageable again.

In our case the issue was caused by trying to monitor the server with PRTG, which gathers information by logging onto the shell over SSH every few minutes, each time writing to wtmp and rapidly growing the file (see link below for more detail), though we also managed to fill the Ramdisk through normal logging after disabling PRTG monitoring.

The long term fix for this is to repoint the scratch partition to persistent storage, as per KB article 1033696 (link below).   We deployed a single 50Gb LUN called “Logs” just for the purpose of storing log data for our 5 hosts, then changed the Advanced Settings parameter ScratchConfig.ConfiguredScratchLocation to /vmfs/volumes/Logs/.locker-servername and rebooted the server.   You can browse the Logs datastore and see the timestamp on the contents of the locker folder to verify it’s working.  Just make sure each server gets a unique folder.

We tested booting the ESXi 5 host without access to the datastore (by disabling the iSCSI switch ports) and proved that the host still comes up OK – it reverts to the Ramdisk as described in the KB.  If lost you need to configure the parameter again and bounce the server.

A related change we also made was to set up syslogging for a larger timeframe (full details via link below) :-

  • We installed the VMware Syslog Collector application onto our vCenter server  (available on the vCenter ISO setup menu)
  • for each ESXi 5 server change the Advanced Settings > Syslog > Global parameters to forward logging to the vCenter server
    • syslog.global.defaultRotate  =  number of logs to keep (0-100)
    • syslog.global.defaultSize = size of each log before rotate (0-10240KiB)
    • syslog.global.logHost = IP address of vCenter or syslog server

The Syslog server creates a subfolder under ..\VMware Syslog Collector\Data\ for the Management IP of each host sending Syslog data.

 

Ramdisk is full:  http://communities.vmware.com/message/2026032

Investigating disk space :  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003564

Configuring Syslog on ESXi 5.0:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003322

Configure persistent scratch location:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033696

ESXi 5.0 monitoring with PRTG:  http://www.paessler.com/knowledgebase/en/topic/32963-esxi-5-vsphere5-with-prtg

 

What has Troyer Done for you?

Once long ago, on a Saturday Afternoon, a little bit before dinner time some vEXPERTs were plotting.

vEXPERT1: We get in through the underground heating system here… up through to the main audience chamber here… and John’s Wookie Costume is here. Having grabbed the Wookie, we inform John that its is in our custody and forthwith issue our demands. Any questions?
vEXPERT4: What exactly are the demands?
vEXPERT2 We’re giving John two days to Deliver the entire Public vEXPERT and Free licenses for all VMware products and if he doesn’t agree immediately we burn it.
vEXPERT3: Cut its head off?
vEXPERT1: Cut all its bits off, send ’em back every hour on the hour… show him we’re not to be trifled with.
vEXPERT3: Also, we’re demanding a ten foot mahogany statue of the Emperor Paul Maritz with burning a Microsoft Flag.
vEXPERT5: What? They’ll never agree to that, Reg
vEXPERT3: That’s just a bargaining counter. And of course, we point out that John bears full responsibility when we chop it up, and… that we shall not submit to blackmail.
all vEXPERTS (Applause) No blackmail!
vEXPERT3: John’s bled us white, the *********. He’s taken All our free time, not just from us, from our Techies and from our Techies’ Techies’.
vEXPERT5: And from our Techies’, Techies’, Techies.
vEXPERT3: Yes
vEXPERT5: And from our Techies, Techies, Techies’, Techies .And from our Techies’, Techies’, Techies’, Techies’, Techies’.
vEXPERT3 All right, vEXPERT5. Don’t labour the point. And what have John ever given us in return?
vEXPERT2: The vEXPERT program.
vEXPERT3: Oh yeah, yeah he gave us that. Yeah. That’s true.
Masked vEXPERT: And the Blogger Lounge!
vEXPERT: Oh yes… The Bloggers Lounge, vEXPERT1, you remember what VMworld used to be like.
vEXPERT3: All right, I’ll grant you that the vEXPERT program and the Bloggers Lounge are two things that John has done…
vEXPERT6: And the Community…
vEXPERT3: (sharply) Well yes obviously the Community… the Community goes without saying. But apart from the vEXPERT program, the Bloggers Lounge and the Community…
Another Masked vEXPERT: Access to Beta Programs…
Other Masked vEXPERTs: Licenses… Emgbargo’d Breifings… Opportunities for Betterment…
vEXPERT3: Yes… all right, fair enough…
vEXPERT Near Front: And the Party…
All vEXPERTs: Oh yes! True!
vEXPERT1: Yeah. That’s something we’d really miss if Jonh left, Reg.
Masked vEXPERT at Back: Our Private Forum section!
vEXPERT5: And it’s safe to walk in talk in the Forums at night now.
vEXPERT1: Yes, John certainly know how to keep order… (general nodding)… let’s face it, he’s the only ones who could in a place like that.
  (more general murmurs of agreement)
vEXPERT3: All right… all right… but apart the Bloggers Lounge and Licenses and Embargo’d Breifings and Access to Beta Programs and Opportunities for Betterment and
the Community and the vEXPERT Program and our Private Forum Section  and The ability for Free Speech… what has the John done for us?
vEXPERT2: Brought US ALL Together!
 
HAPPY BIRTHDAY JOHN AND THANKS FOR ALL THE THINGS THAT YOU HAVE
DONE FOR US
 
 
 
Apologies for the vastly Talented Monty Python team for my pathetic attempt at a spoonerism.