Wednesday, June 18, 2008

Exchange 2007 Disk Thresholds

First computer post....

This hit us at the office today, internal and outgoing mail was working fine but incoming was not reaching us. No bounce backs, no queues showing messages; it was just.... quiet. There were a couple of patches outstanding from last week and I figured, there was probably some minor exchange glitch that a reboot would fix (the app log looked pretty clean).

So I let it apply the outstanding patches, and in the process it bounced the exchange services and we started getting mail again. However, after system reboot we were back to square one. So I went digging around a bit more and determined that we had just under a gig of free space on the C:\. Usually when an exchange server hard drive fills up I go through and look at the directory exchange writes logs to, which in this case was C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group.

We had what I will call a heroic amount of log files there; roughly 14 gigs worth. I love the lack of Exchange 2007 support on our Sonicwall CDP...

Best way to clear these out that I've seen on systems without backup software such as Symantec's Backup Exec is to run a system state and exchange backup with NTbackup; in the process of running the backup it'll flush the logs for you. There are two caveats to this, first, you need sufficient space SOMEWHERE on the network to write the backup, and two, NTbackup has been severely gutted in Server 2008 and you may be up a creek if you're running Server 2008 and Exchange 2007.

While I was waiting for the backup to process, I went through the logs again and found some entries in the app log describing the problem.

First we got:





Event Type: Warning
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15004
Date: 6/17/2008
Time: 3:09:31 PM
User: N/A
Computer: MERLOT
Description:
Resource pressure increased from Normal to Medium.

Resource utilization of the following resources exceed
the normal level: Queue database logging disk
space ("C:\Program Files\Microsoft\Exchange Server\
TransportRoles\data\Queue\") = 96% [Medium] [Normal=94%
Medium=96% High=98%] Physical memory load = 92% [limit
is 94% before message dehydration occurs.]

Back pressure caused the following components to be disabled:
Inbound mail submission from the Internet
Mail submission from the Pickup directory
Mail submission from the Replay directory

The following resources are in the normal state:
Queue database and disk space ("C:\Program Files\
Microsoft\Exchange Server\TransportRoles\data\
Queue\mail.que") = 95% [Normal] [Normal=94% Medium=96%
High=98%]
Version buckets = 0 [Normal] [Normal=40 Medium=60 High=100]
Private bytes = 14% [Normal] [Normal=71% Medium=73% High=75%]



Followed immediately by:





Event Type: Error
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15006
Date: 6/18/2008
Time: 10:53:54 AM
User: N/A
Computer: MERLOT
Description:
The Microsoft Exchange Transport service is rejecting
message submissions because the available disk space has
dropped below the configured threshold.

Resource utilization of the following resources exceed
the normal level:
Queue database logging disk space ("C:\Program Files\
Microsoft\Exchange Server\TransportRoles\data\Queue\")
= 96% [Medium] [Normal=94% Medium=96% High=98%]

No components were disabled because of back pressure.
The following resources are in the normal state:
Queue database and disk space ("C:\Program Files\Microsoft\
Exchange Server\TransportRoles\data\Queue\mail.que") = 95%
[Normal] [Normal=94% Medium=96% High=98%]
Version buckets = 0 [Normal] [Normal=40 Medium=60 High=100]
Private bytes = 0% [Normal] [Normal=71% Medium=73% High=75%]
Physical memory load = 37% [limit is 94% before message
dehydration occurs.]



When service was restored the app log wrote:



Event Type: Information
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15005
Date: 6/18/2008
Time: 11:22:55 AM
User: N/A
Computer: MERLOT
Description:
Resource pressure decreased from Medium to Normal.

No components were disabled because of back pressure.
The following resources are in the normal state:
Queue database and disk space ("C:\Program Files\
Microsoft\Exchange Server\TransportRoles\data\Queue\
mail.que") = 90% [Normal] [Normal=94% Medium=96% High=98%]
Queue database logging disk space ("C:\Program Files\
Microsoft\Exchange Server\TransportRoles\data\Queue\")
= 91% [Normal] [Normal=94% Medium=96% High=98%]
Version buckets = 0 [Normal] [Normal=40 Medium=60 High=100]
Private bytes = 10% [Normal] [Normal=71% Medium=73% High=75%]
Physical memory load = 62% [limit is 94% before message
dehydration occurs.]



Technet has nothing on this error at this point and our monitoring software, HyBlue didn't catch it, albeit I am DEFINITELY sending them a note! I thought I'd pop a note out there for others who might run into his error; it's a pretty easy fix, just free up some drive space or set your threshold down lower and risk filling your hard drive completely. Haven't found any documentation on HOW to do that yet, personally I think it's better to watch your free space and make sure those logs get cleared out!

3 comments:

Eric said...

Wow, thank you for this post. I am new to exchange. I have a test / almost production environment server I have been testing for the last couple weeks. I was almost ready to bring it 100% live / production but then email stopped working in the middle of the night. I could send out from Exchange, but nothing comming inbound and no bounce msgs. This worked. Cleared out the logs, created a new disk to store them and back on track!

Thanks.

Eric

Bill Davis said...

I had a similar issue.

Check this link.

http://technet.microsoft.com/en-us/library/bb397220.aspx

Magnus said...

Thanks you! came in very useful. Other people consider me to be quite experienced... because i don't use google in from of them! Great post, can't believe it's Jan 2010 and still nothing on Technet about this...

P.S> i hate your captcha! :)