Exchange 2007 Disk Thresholds

First computer post....

This hit us at the office today, internal and outgoing mail was working fine but incoming was not reaching us. No bounce backs, no queues showing messages; it was just.... quiet. There were a couple of patches outstanding from last week and I figured, there was probably some minor exchange glitch that a reboot would fix (the app log looked pretty clean).

So I let it apply the outstanding patches, and in the process it bounced the exchange services and we started getting mail again. However, after system reboot we were back to square one. So I went digging around a bit more and determined that we had just under a gig of free space on the C:\. Usually when an exchange server hard drive fills up I go through and look at the directory exchange writes logs to, which in this case was C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group.

We had what I will call a heroic amount of log files there; roughly 14 gigs worth. I love the lack of Exchange 2007 support on our Sonicwall CDP...

Best way to clear these out that I've seen on systems without backup software such as Symantec's Backup Exec is to run a system state and exchange backup with NTbackup; in the process of running the backup it'll flush the logs for you. There are two caveats to this, first, you need sufficient space SOMEWHERE on the network to write the backup, and two, NTbackup has been severely gutted in Server 2008 and you may be up a creek if you're running Server 2008 and Exchange 2007.

While I was waiting for the backup to process, I went through the logs again and found some entries in the app log describing the problem.

First we got:

Event Type: Warning Event Source: MSExchangeTransport Event Category: ResourceManager Event ID: 15004 Date: 6/17/2008 Time: 3:09:31 PM User: N/A Computer: MERLOT Description: Resource pressure increased from Normal to Medium. Resource utilization of the following resources exceed the normal level: Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\ TransportRoles\data\Queue\") = 96% [Medium] [Normal=94% Medium=96% High=98%] Physical memory load = 92% [limit is 94% before message dehydration occurs.] Back pressure caused the following components to be disabled: Inbound mail submission from the Internet Mail submission from the Pickup directory Mail submission from the Replay directory The following resources are in the normal state: Queue database and disk space ("C:\Program Files\ Microsoft\Exchange Server\TransportRoles\data\ Queue\mail.que") = 95% [Normal] [Normal=94% Medium=96% High=98%] Version buckets = 0 [Normal] [Normal=40 Medium=60 High=100] Private bytes = 14% [Normal] [Normal=71% Medium=73% High=75%]

Followed immediately by:

Event Type: Error Event Source: MSExchangeTransport Event Category: ResourceManager Event ID: 15006 Date: 6/18/2008 Time: 10:53:54 AM User: N/A Computer: MERLOT Description: The Microsoft Exchange Transport service is rejecting message submissions because the available disk space has dropped below the configured threshold. Resource utilization of the following resources exceed the normal level: Queue database logging disk space ("C:\Program Files\ Microsoft\Exchange Server\TransportRoles\data\Queue\") = 96% [Medium] [Normal=94% Medium=96% High=98%] No components were disabled because of back pressure. The following resources are in the normal state: Queue database and disk space ("C:\Program Files\Microsoft\ Exchange Server\TransportRoles\data\Queue\mail.que") = 95% [Normal] [Normal=94% Medium=96% High=98%] Version buckets = 0 [Normal] [Normal=40 Medium=60 High=100] Private bytes = 0% [Normal] [Normal=71% Medium=73% High=75%] Physical memory load = 37% [limit is 94% before message dehydration occurs.]

When service was restored the app log wrote:

Event Type: Information Event Source: MSExchangeTransport Event Category: ResourceManager Event ID: 15005 Date: 6/18/2008 Time: 11:22:55 AM User: N/A Computer: MERLOT Description: Resource pressure decreased from Medium to Normal. No components were disabled because of back pressure. The following resources are in the normal state: Queue database and disk space ("C:\Program Files\ Microsoft\Exchange Server\TransportRoles\data\Queue\ mail.que") = 90% [Normal] [Normal=94% Medium=96% High=98%] Queue database logging disk space ("C:\Program Files\ Microsoft\Exchange Server\TransportRoles\data\Queue\") = 91% [Normal] [Normal=94% Medium=96% High=98%] Version buckets = 0 [Normal] [Normal=40 Medium=60 High=100] Private bytes = 10% [Normal] [Normal=71% Medium=73% High=75%] Physical memory load = 62% [limit is 94% before message dehydration occurs.]

Technet has nothing on this error at this point and our monitoring software, HyBlue didn't catch it, albeit I am DEFINITELY sending them a note! I thought I'd pop a note out there for others who might run into his error; it's a pretty easy fix, just free up some drive space or set your threshold down lower and risk filling your hard drive completely. Haven't found any documentation on HOW to do that yet, personally I think it's better to watch your free space and make sure those logs get cleared out!