In my test lab, I wanted to determine the best way to recover a CCR node in the event the server fails due to corruption, hardware failure or the OS just tanks.  Here are the steps I have found to work: 

(Taken from: http://technet.microsoft.com/en-us/library/bb124095.aspx)

1. Evict the corrupted node from the Windows 2008 Cluster.  Delete any DNS entries or Computer objects relating to the failed machine from AD and DNS.  You may have to use ADSIEDIT.
2. Rebuild the Physical machine with the same OS, Patches and VLAN (network) configurations
3. Add the node to the existing Windows 2008 Cluster
4. Install Exchange 2007.  During the installation Select the Passive Mailbox Role
5. Fail the services over to the new Node.  Take the services offline.
   • Note: This will break service while the node is being recovered or the SGs initially will not mount.
6. Once the Exchange Passive Node installation completes, open a command prompt at type:
   • Setup.com /recoverCMS /CMSName:<Exchange Virtual Cluster Name> /CMSIPaddress:<IP Address of the Exchange Virtual Cluster Name> (An Example would be: Setup.com /recoverCMS /CMSName:MSX-MAIL01 /CMSIPaddress:192.168.0.50)
   • Note: The Storage configuration should match that of the lost clustered mailbox server prior to running Setup.com.  After restoring a CCR environment, replication is suspended. Replication must be enabled before the CCR environment is fully operational. In addition, you may need to reseed the passive copy of one or more databases.
7. After the clustered mailbox server has been recovered, the Microsoft Exchange System Attendant service will start and then stop. You must then manually bring the System Attendant resource online
8. The expected behavior for this operation is that the databases will not be mounted at the end of recovery. The databases will stay dismounted throughout the move and failovers until you explicitly mount the databases.
9. Install the passive clustered mailbox role on other passive nodes, if necessary.

Since the inception of Windows 2008 Server and Exchange 2007, 2008 Server is no longer able to perform Exchange aware backups using NTbackup or now known as Windows Server Backup.  In my opinion, the disconnect between the Microsoft's Exchange and Windows Dev teams, is now even more obvious.  I suspect they have been getting hammered by the Exchange community who upgraded their systems and were unaware of this major flaw.  (Hey MS, don't you guys have internal news groups???  :) )  Anyways, the fix is to either go buy some type of 3rd party app, be it software or SAN initiated, or use Microsoft's Data Protection Manager 2007.  Either is likely not cheap, especially if you have a large Exchange environment.

This is what I have installed from my Virtual Server environment.

1. 1 - Windows 2008 DC\GC
2. 2 - Windows 2008 Clustered Machines running Exchange 2007 in a CCR Cluster Configuration
3. 1 - DPM Server running SQL 2005 and DPM 2007

• Fully Patch all of your Windows 2008 servers, including the latest Exchange 2007 roll up.  I am running Roll up 6.
• For simplicity sake, disable the Windows Firewalls on each server.  (Hey, its a test lab in a NAT'd environment :) )
• All Servers and Software are ran in 64 Bit

Note: Before this part is completed, the assumption here is you already have Active Directory and your CCR cluster configured.  I will have another blog shortly describing these steps.

On the DPM Windows 2008 Server:

1. Add the Windows Powershell feature
2. Install IIS role

   In the Add features required for Web Server (IIS)? message box, click Add Required Features.

Ensure that you select the following Role service:

1. HTTP Redirection
2. Application Development
3. ASP.net
   • .NET Extensibility
   • ISAPI Extensions
   • ISAPI Filter
   • Server Side Includes
4. IIS 6 Management Compatibility
   • IIS 6 Metabase Compatibility
   • IIS 6 WMI Compatibility
   • IIS 6 Scripting Tools
   • IIS 6 Management Console
5. Security (Installed)
   • Windows Authentication (Installed)
6. Install the Single Instance Store Optional Component (type “ocsetup.exe SIS-Limited” in the command prompt)
7. Reboot the DPM Server
8. Install the Data Protection Manager 2007.  I placed the DPM database on the C: drive and added iSCSI drives later with Solar Winds.
9. Install SQL 2005 SP3
10. Reboot the DPM Server
11. Install the DPM 2007 Feature Pack
12. Install the DPM Hotfix Rollup 2 Fix 
13. Reboot the DPM Server
14. Copy the ESE.DLL and ESEUTIL.EXE files from one of your Exchange 2007 Cluster nodes to C:\Program Files\Microsoft DPM\DPM\bin on the DPM Server
15. Start the IIS Manager and navigate to Report$MS$DPM2007$, Open Handler Mappings and click Edit Feature Permissions, Make sure the “Script” option is checked.

You should now be able to go into the DPM Management Console and add Disks and Protected Groups.

Happy Hunting...

Ed McKinzie

With multiple Domain Controllers in AD, it makes little sense to hard-code any DC within programming code, applications and user profiles.  The main reason being a single point of failure, such as during maintenance windows or if a physical machine unexpectedly dies off.  The best way to mediate this is to create a DNS Round Robin for a friendly name, such as LDAP.Domain.com, and list several Domain Controllers as possible end points.  The trick however, you must configure the Domain Controllers with a certificate from either your internal PKI or a 3rd party CA, such as VeriSign.  Installing this cert is a requirement to talk SSL\TLS LDAP using subject alternate names.

Here are two KB articles that outline the process:

• How to add a Subject Alternate Name to a secure LDAP certificate
• How to enable LDAP over SSL with a third-party Certificate Authority

There are 4 key steps to follow:

1. Create the INF file, which determines which attributes to use
2. Use the INF to create a request file
3. Submit the request file to be signed by the Certificate Authority
4. Accept and install the new certificate in the local computer store

• Create the INF file.  Save the blue text to a file and name it certnew.inf

==========================================================================================

• From the same directory you saved the certnew.inf file issue this command (On the DC). :

Certreq -new certnew.inf certnew.req

• Issue this command:

Certreq -submit certnew.req certnew.cer

Note: Certreq.exe will prompt you for which CA to use if you have multiple CA’s in your environment.

• Issue this command:

Certreq -accept certnew.cer

 

Note: This command copies the private key with the certificate and automatically places it in the private computer store.  If you try to use the Certificate Authority Web enrollment form, and do not select store in the local computer store, it will place the cert in the current user cert store on the DC.  This breaks the SSL\SAN capability. 

 

You can and should delete any other certs in the DC computer store, as they will cause problems due to caching of certs within SCHANNEL as there is no way to know which cert the LDAP client binds to.

There was a problem connecting to Microsoft Office Outlook. Your Outlook profile is not configured correctly. Contact your system administrator with this information."

I have read several blogs and forums regarding this topic.  I have ran the gauntlet of uninstalling Communicator 2007, Outlook 2007, Exchange System Manager 2003, and several hotfixes, but nothing has seemed to fix the issue.  Until now.  The simple fix was to run FIXMAPI from a command prompt.  The file itself is located in the "C:\Windows\System32" directory.  I opened a command prompt and ran the FIXMAPI executable.   It returns no errors or output stating it ran. 

My assumption here is it replaces or removes a lock on the mapi32.dll that is shared among the Office and OCS products.

Once you run it, restart Outlook and Communicator 2007.  I now no longer get the Outlook Integration errors. 

Fixmapi is described here: http://msdn.microsoft.com/en-us/library/bb927655.aspx and http://support.microsoft.com/kb/228457.

-E

Here is the Technet Article explaining the process. http://technet.microsoft.com/en-us/library/cc738780.aspx

Copy this to a batch file and run it.  I typically schedule the job to run once daily, as the Local System Account. 

Note, the local computer account will need write access to the directory or share you are copying the database to.  You should also place a secure password in the command.

certutil -backup -p "password" -f -gmt -seconds -v \\ServerName\CABackup\

The directory should look similar to this after the script runs.

This issue may occur when Active Directory has not been updated with the Windows Server 2003 R2 schema extensions.

To resolve this issue, run the adprep.exe /forestprep command from the Windows Server 2003 R2 installation disk 2 on the schema master. To do this, insert the Windows Server 2003 R2 installation disk 2, and then type the following command:

Drive:\CMPNENTS\R2\ADPREP\adprep.exe /forestprep

Once, complete, rerun your DCPROMO and the issue should be resolved.

I have run into a few issues trying to get Networking to work properly with Virtual hosts and Virtual PC 2007 SP1.  I can get them to talk within the Local Network (Using the Local only settings) or installing a Microsoft Loopback Adapter, but I need the Virtuals to be able to talk to each other and access the Net.  I have found a couple of good Microsoft whitepapers describing some of the nuances and work arounds to make it happen.  Hopefully if you read this, you'll find it helpful.

Ed McKinzie

Scenario: Using a single network subnet

I have a series of applications I needed to test audio, namely Microsoft Live Meeting 2007 (Audio\Video).  However, as most of you know, audio drivers or emulators are not installed by default with Microsoft Virtual PC 2007.  In order to get it to work on Windows Server 2003, you must copy two files off of a Windows XP cd and place them on the virtualized system.  The two driver files you need are wdma_ctl.inf and ctlsb16.sys.   I want to note that this does not work for Virtual Server 2005, only Virtual PC 2007.

Here are the steps:

From a command prompt on a Windows XP system, run this command:

"%windir%\driver cache\i386\driver.cab" -F:ctlsb16.sys c:\

This will extract the ctlsb16.sys to your root c:\.  If it does not, from the extracted CAB driver window that popped up, copy the ctlsb16.sys file manually to the root of C:\.

The wdma_ctl.inf file is also located at C:\WINDOWS\inf folder on the Windows XP system.

Copy these two files over to the Virtual PC.  From the Device Manager on the Virtual PC instance, perform a Have Disk during the device installation and direct the install to *.inf file you copied over.  Reboot the workstation as necessary.  You may have to enable the sound device from the control panel after the server is booted.  You may also want to place an audio icon on the task bar....which is also done from the control panel. 

Ed McKinzie

Scenario: Installing Exchange 2007 SP1 on Windows Server 2008 Server, in a new Forest, new Domain deployment.  During the HUB Transport Server Role install, the E2K7 install failed with this error:

Hub Transport Role
Failed Error: Service 'MSExchangeTransport' failed to reach  status 'Running' on this server.

After some digging and Google research, it appears the problem stems from IPv6 being disabled on the servers NIC.  I had disabled IPv6 when the server was first installed on the network, knowing it was not necessary for my networks' topology.  Evidently Exchange 2007 requires otherwise.  Re-enabling IPv6 and booting the server resolved the issue.

Ed McKinzie

I have had numerous complaints about users having problems either seeing other users' Free/Busy information, having problems updating their Delegation settings and\or users' having difficulty Accepting a meeting request on behalf of another user.  (Awarded by way of Delegated rights).  There are a varying degree of error messages, such as: "Unable to update your free/busy information" or "The Delegates settings were not saved correctly", etc.  This has been an ongoing issue, that I believe we have made progress on understanding.

I always give props, when props are due, so I wanted to mention a great article I found that was written by JD Wade.  The article can be found here: Free/Busy Article.  

How Free/Busy works: Free/Busy information is stored in a dedicated system folder called SCHEDULE+FREE BUSY on the Public Folder Server. This folder contains a separate sub-folder for each Exchange administrative group. When a user publishes his Free/Busy data, Exchange posts this information in a message in the appropriate sub-folder that now functions in a manner similar to the offline address book folders.  A hidden message is also added into the actual Exchange Mailbox to keep track of the Free/Busy data.  Every time a user receives a meeting or calendar request, this message gets updated.   If that file becomes corrupt, any subsequent files are also corrupt.  Deleting the Free/Busy message off of the Public Folder server should force the removal of the hidden message in the Exchange mailbox.

For initial troubleshooting of Free/Busy issues you can try one of 2 things:
1.  Run the /cleanfreebusy switch againt Outlook Profile
2.  Manually remove the user’s Free/Busy folder

You can also use the MAPI Editor Tool to manually delete the Free/Busy message off of the server.  You will need to create an Outlook profile for the mailbox you want to repair:

1.  Choose Session and Logon and Display Store Table
2.  Choose the Outlook profile you created
3.  Double click the Instance that starts with “Mailbox -”
4.  Expand Root - Mailbox
5.  Click on IPM_SUBTREE
6.  In the right window, if there is a property name of PR_FREEBUSY_ENTRYIDS, right click on it and choose Delete Property (if it exists here…it may not)
7.  Expand IPM_SUBTREE
8.  Click on Inbox
9.  In the right window, if there is a property name of PR_FREEBUSY_ENTRYIDS, right click on it and choose Delete Property
10.  Exit out of MAPI Editor.
11.  Re-run the /cleanfreebusy on the mailbox. 

Here is the hidden message in Exchange Mailbox:

Here is the Free/Busy Message on the Public Folder Server.  Each user mailbox is represented by ExchangeLegacyDN attribute, which is highlighted below.

Ed McKinzie

How to configure a CA to accept a SAN attribute from a certificate request

Situation: We have an EFS deployment where we can encrypt Files and folders based on the Domain GPO policy.  Unfortunately, the Recovery Agent certificates expired causing the encrypted services to begin to fail.  Here is what we did to fix the issue.  Keep in mind we use an internal PKI infrastructure to issue certificates.

Securing the Default Recovery Key for the Domain
As with the stand-alone computer, a default recovery policy is configured for the
domain when the first domain controller is set up. The default recovery policy uses
a self-signed certificate to make the domain Administrator account the recovery
agent.
Note: To change the default, log on as Administrator on the first domain controller
of the domain, and follow the steps above to secure the recovery key for the domain.

THIS WAS NOT AN OPTION FOR ME, AS MY FIRST DC HAS LONG BEEN RETIRED, BUT I WANTED PEOPLE TO BE AWARE OF THIS.

Requesting a File Recovery Certificate
If you decide to use the default recovery policies, you never need to request a file
recovery certificate. However, in circumstances where multiple recovery agents are
needed for the domain or where the recovery agent needs to be different from the
domain administrator due to legal or corporate policy, you may need to identify certain
users as recovery agents, and these users must be issued file recovery certificates.
To accomplish this, the following procedures must be completed:
•  An Enterprise certification authority (CA) must be set up, if one isn’t available.
•  The policy on the Enterprise CA must allow the designated user/agents to request and obtain a file recovery Certificate.  (Enroll rights must be granted on the EFSRECOVERY Template)
•  Each user must request a file recovery certificate.

I am assuming you already have a CA installed and have already created a user account(s) to be used as the recovery Agents.  They can be simple user accounts in the domain, if you have not. 

In my example I am using EFSUSER1 and EFSUSER2.

Add the Domain Recovery Agents group to the EFS Recovery Template.
This procedure allows users in that group to request recovery certificates.
1. Click Start, point to Programs, point to Administrative Tools, and click Active Directory Sites and Services.
2. On the View menu, click Show Services.
3. Click the + next to Services in the left pane. Use this method to expand the Public Key Services folder.
4. Click Certificate Templates (in the left pane).

5. Double-click EFSRecovery in the right pane.
6. Click the Security tab.
7. Click Add. Scroll and find the EFSUSER1 and EFSUSER2 accounts and click Add.

8. With the two accounts selected in the top pane, select the Enroll check box in the bottom pane.
9. Click OK, and close the Active Directory Sites and Services snap-in.

Assign a Certificate to the EFS Accounts.

1. Add the EFSUSER1 and EFSUSER2 accounts to the local administrators group on a workstation in the UM-USERS domain.
2. Login to the workstation as one of the accounts. Note, if you cannot remember the password, reset it.
3. Once logged in, open the local certificates as “My User Account”. (NOT Computer). Either by way of an MMC or within the Administrator tools.  Expand Personal certificates.   Request a “New Certificate”. Select EFS Recovery Agent. Give it a friendly name of the user account.

1. Once complete, export the certificate with both the private key and one without the private key to a directory on the local workstation or server.
2. Repeat these steps for both accounts.
3. Once the certificates are exported, login to a workstation\server with Domain Admin rights. Make sure the account has access to the directory and exported certs you just created. Launch AD Users and Computers, right click on the Domain and Select Properties. Edit the Default Domain Policy.

• Expand the Group Policy --> Windows Settings --> Security Settings --> Public Key Policies --> Encrypting File System.

• Right click in the EFS Panel and select Add Data Recovery Agent.

• Browse to the Certificate with no private key or the *.CER file you just created in the exported Cert Directory. Hit Next and Finish. Refresh the EFS Panel.

1. You should now see the proper certificates.
2. Now burn the contents of the exported EFS directory to a CD or Media, place in the vault and delete the cert files\directories from your workstation\server.  Also remove the accounts from the local administrators group on the workstation.

In Exchange 5.5, 2000, 2003, or 2007 there many instances where a user has defined a delegate to their mailbox.  This is due to many reasons, such as Secretary or Assistant permissions for Calendaring, Inbox, Task and Journaling management.  There is a whole plethora of reason to do it.  However, along with it comes lots of potential issues.  We'll get to those later.

From within Outlook, Select Tools --> Options --> Delegates.   From this panel you can add Delegates as desired.  However, the Outlook client is the only way for a non-administrator to know any Delegates are in fact assigned.

You can however, poll Active Directory by using custom scripts or via ADSIedit and attain the Delegate settings.  There are two attributes in AD that hold this information.

From the command prompt type Ipconfig /all to retrieve the current server name.

Type this command:netdom renamecomputer <ComputerName> /NewName:<NewComputerName>

To reboot the server, type: shutdown /r /t 0

The name change will not take affect until the server is rebooted.