Exchange

 

Federation always fun as you may have read in one of my previous blogs. Starting this week we had some issues at a customer where the Free/busy lookups across environments stopped working. Not only to other Exchange environments but also Office 365.

Both the Outlook logging and event logs on the CAS provided the same error. The complete error message is displayed below:

Process Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]: <johan.veldhuis@domain.com>SMTP:johan.veldhuis@domain.com failed. Exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.AutoDiscoverFailedException: Autodiscover failed for e-mail address <johan.veldhuis@domain.com>SMTP:johan.veldhuis@domain.com with exception System.Web.Services.Protocols.SoapHeaderException: An error occurred when processing the security tokens in the message.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
   at Microsoft.Exchange.SoapWebClient.AutoDiscover.DefaultBinding_Autodiscover.EndGetUserSettings(IAsyncResult asyncResult)
   at Microsoft.Exchange.InfoWorker.Common.Availability.UserSoapAutoDiscoverRequest.EndGetSettings(IAsyncResult asyncResult)
   at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.<>c__DisplayClass4.<EndInvoke>b__3()
   at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.ExecuteAndHandleException(ExecuteAndHandleExceptionDelegate operation). —> System.Web.Services.Protocols.SoapHeaderException: An error occurred when processing the security tokens in the message.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
   at Microsoft.Exchange.SoapWebClient.AutoDiscover.DefaultBinding_Autodiscover.EndGetUserSettings(IAsyncResult asyncResult)
   at Microsoft.Exchange.InfoWorker.Common.Availability.UserSoapAutoDiscoverRequest.EndGetSettings(IAsyncResult asyncResult)
   at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.<>c__DisplayClass4.<EndInvoke>b__3()
   at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.ExecuteAndHandleException(ExecuteAndHandleExceptionDelegate operation)
   — End of inner exception stack trace —
. Name of the server where exception originated: EXC01. This event may occur when Availability Service cannot discover an Availability Service in the remote forest.

The bold part marks the error in this case An error occurred when processing the security tokens in the message. This error tells you there is an issue with the security token which your server has received from the Microsoft Federation Gateway. So probably this customer is not the only customer facing this issue.

Microsoft has acknowledged there were some issues earlier this week which not only affected this functionality but also some Office 365 services.

Office 365 status

Currently a work around is available to fix this issue. To fix the issue you will need to refresh the federation trust metadata. This can be done by using the following Powershell cmdlets:

Get-Federationtrust|Set-Federationtrust -Refreshmetadata

It is important to know that this cmdlet will need to be run on both sides else it may not fix the issues. Once the cmdlet have been executed it may take some time before the free/busy lookups start to work again.

In the first two parts of the blog series about troubleshooting federated sharing we had a look at the infrastructure and configuration which is required. Besides this we did some basic troubleshooting on the components involved during federated sharing. In this part we will look at some examples which I gathered during troubleshooting a federated sharing issue.

Below you will see an example of an error which was received when trying to retrieve the free/busy information from a user hosted on another Exchange environment.

Process 1212: ProxyWebRequest FederatedCrossForest from S-1-5-21-1671710892-3805255249-3875359145-102309 to https://mail.domain.com/ews/exchange.asmx/WSSecurity failed. Caller SIDs: WSSecurity. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)

   at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)

   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)

   — End of inner exception stack trace —

   at System.Web.Services.Protocols.WebClientAsyncResult.WaitForResponse()

   at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)

   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, Service service, IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling():. The request information is ProxyWebRequest type = FederatedCrossForest, url = https://mail.domain.com/ews/exchange.asmx/WSSecurity

Mailbox list = <Johan@domain.com>SMTP:Johan@domain.com, Parameters: windowStart = 10/1/2013 10:00:00 AM, windowEnd = 10/31/2013 10:00:00 AM, MergedFBInterval = 30, RequestedView = Detailed

. —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)

   at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)

   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)

   — End of inner exception stack trace —

   at System.Web.Services.Protocols.WebClientAsyncResult.WaitForResponse()

   at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)

   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, Service service, IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()

   — End of inner exception stack trace —

. Name of the server where exception originated:CAS01. Make sure that the Active Directory site/forest that contain the user’s mailbox has at least one local Exchange 2010 server running the Availability service. Turn up logging for the Availability service and test basic network connectivity.

When looking at the marked text you will see the actual cause, a certificate issue. So how to deal in this case? The first step you can take is try to access the Exchange Web Services of the other Exchange environment. In this case we can do it by browsing to https://mail.domain/com/ews/exchange.asmx/WSSecurity what will probably happen is that you receive a certificate warning. And that is exactly why the lookup fails. The certificate from the remote Exchange environment is not valid according to the validation procedure. However when you open it in a browser you will see the reason why the certificate is not trusted. This can be caused by several things among them:

  • Certificates is signed by a root CA which is not trusted
  • Name on the certificate is incorrect

In this case the root CA was not trusted by the Exchange environment. By importing the root CA in the Enterprise Trusted Root folder of the CAS the problem was solved.

The second one was pretty hard to troubleshoot but the solution to solve it was pretty easy. Again the error is marked in the text below. The error tells you that the other side did close the connection. OK nice and now what? In this case you will need to search in the IIS logs on the CAS of the target Exchange environment to see what happens when traffic from your Exchange environment arrives at the CAS.

Process 1212: ProxyWebRequest FederatedCrossForest from S-1-5-21-1671710892-3805255249-3875359145-102309 to https://mail.domain.com/ews/exchange.asmx/WSSecurity failed. Caller SIDs: WSSecurity. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

   — End of inner exception stack trace —

   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)

   at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)

   at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)

   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)

   — End of inner exception stack trace —

   at System.Web.Services.Protocols.WebClientAsyncResult.WaitForResponse()

   at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)

   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, Service service, IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling():. The request information is ProxyWebRequest type = FederatedCrossForest, url = https://mail.domain.com/ews/exchange.asmx/WSSecurity

Mailbox list = <Johan@domain.com>SMTP:Johan@domain.com, Parameters: windowStart = 9/29/2013 12:00:00 AM, windowEnd = 11/10/2013 12:00:00 AM, MergedFBInterval = 30, RequestedView = MergedOnly

. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

   — End of inner exception stack trace —

   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)

   at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)

   at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)

   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)

   — End of inner exception stack trace —

   at System.Web.Services.Protocols.WebClientAsyncResult.WaitForResponse()

   at System.Web.Services.Protocols.WebClientProtocol.EndSend(IAsyncResult asyncResult, Object& internalAsyncState, Stream& responseStream)

   at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, Service service, IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)

   at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()

   — End of inner exception stack trace —

. Name of the server where exception originated: CAS01. Make sure that the Active Directory site/forest that contain the user’s mailbox has at least one local Exchange 2010 server running the Availability service. Turn up logging for the Availability service and test basic network connectivity.

Search for traffic destined to /ews/exchange.asmx/WSSecurity and you will probably find the error that did occur. Normally when everything works a 200 will be displayed. If you do receive a 4XX error then verify that the federation with the Microsoft Federation Gateway works correctly as explained in the first part. Besides this verify that WSSecurity is enabled on the Autodiscover and EWS directory.

However you might get other errors, in this case it was a 500 error. What this means and that it doesn’t know how to deal with the traffic which arrives and will close the connection. If this happens make sure WSSecurity is enabled on the virtual directories for Autodiscover and EWS. When this is confirmed verify that the svc-integrated handler is assigned to both the Autodiscover and EWS. If this is both configured correctly everything should be OK but why doesn’t it work?

In some occasions it may happen that EWSSecurity is correctly enabled but for some reason IIS doesn’t pick this up. If this happens an iisreset will fix your issue and you will be able to retrieve the free/busy information from the other Exchange organization.

Here ends the series of troubleshooting federated sharing. I am aware there might be other solution for the issues you might find during the implementation but these were just two examples of issues I found.

I hope you liked this series and if you have any questions use the contact form on the site to send me a message or ask your question by posting a comment.

Troubleshooting federated sharing – part II

In the first part of the blog series about troubleshooting federated sharing we had a look at the server side. We discussed which parameters you should verify during troubleshooting and if they are configured incorrectly how to fix them.

Now we will continue our journey in this second part. In this part we will have a look at the reverse proxy side followed by the client side.

Let’s start with having a look at the reverse proxy.

Because you don’t want your Exchange environment is directly accessible via the internet most companies will us a reverse proxy to “publish” their Exchange environment. This can either be done via a Microsoft product such as TMG or ARR. The first one will be end-of-support shortly so if not having one in place don’t start implementing it but use ARR or a 3rd party solution. When looking at the third party solutions there are a lot of solutions you can use. For example BlueCoat, Kemp Technologies or F5 have products which offer this functionality.

So what can be the problem with the reverse proxy which is used for publishing the Exchange environment? Well there can be many problems among them:

  • Authentication issues
  • Name resolving issues
  • Routing issues

This are just three of the sort of issues you can experience when using a reverse proxy. I am sure there are a lot more sorts. In this blog we will focus on the authentication issues. The reason for this is that these are the issues you will probably encounter during your journey in configuring federated sharing.

The authentication issue is one of those things you will see a lot. Especially when you are using pre-authentication. Using pre-authentication your session will be authenticated before it is send to the CAS/CAS Array which is located behind the firewall.

For federated sharing it is required that all the federated sharing sessions will be passed-through to the CAS without having to authenticate. Authentication in this case will take place on the CAS/CAS Array. So one thing to make sure is that you will create the rules required for federated sharing above the existing rules. As mentioned in the first part the URL’s which are required are as follows:

  • /EWS/Exchange.asmx/WSSecurity
  • /Autodiscover/Autodiscover.svc
  • /Autodiscover/Autodiscover.svc/WSSecurity

If you are using the TMG to publish these URL’s ensure the following options are select:

  • Authentication Delegation: No delegation, but client may authenticate directly
  • Users: all users

Most companies will use Form Based Authentication (FBA) to publish their Exchange Services to the internet. If this is the case you will need to change the listener. This because by default the listener will required all users to authenticate before they can connect which makes sense.

However for the URL’s used for federated sharing the authentication takes on the CAS so we also want to disable this specific option.

To do this get the properties of the listener and select the authentication tab. Open the advanced authentication options by pressing the advanced button and uncheck the option Require all users to authenticate.

Once configured you will need to find out if the rule really works the way you want. For example if configured in the wrong order the traffic will not even hit configured rule. To monitor this piece some logging will need to be enabled on the TMG side. I personally recommend you to create a filter using the following parameters:

  • Filter by: rule
  • Condition: equals
  • Value: name of the publishing rule used to publish these services

Now logging is configured try to do a free/busy from another Exchange environment or from Office 365. If the rule is on the correct place you should see that the traffic hits the rule.

If you do not see any logging the traffic will not hit the rule. In that case verify the already discussed parameters and try again. In some cases it might take a few minutes before the TMG has picked up the new config so if it does not work directly wait a few minutes and try again.

Another issue you might see are authentication issues in the TMG log. This will be either 401.x or 403.x errors in most cases. In this case verify that when you browse to the following pages no form will be displayed:

If a form is displayed please verify the delegation settings on the rule and the authentication settings on the listener.

OK enough about TMG let’s continue with the client side. By default Outlook has disabled logging for troubleshooting purposes so before starting the troubleshooting you will need to enable it. Microsoft has published a knowledge base article on how to enable the troubleshooting logging which can be found here.

When you have reproduced the issue it is time to look at the logs. Depending on the version of Outlook you the log are named different and are located in different locations. Below an overview of Outlook version, the log location and log file name:

Version Location Filename
Outlook 2007 %temp%\OLKas date-time -fb.log
Outlook 2010 %temp%\OLKas date-time -AS.log
Outlook 2013 %temp%\Outlook logging Outlook-########.etl

So if you are on Outlook 2007/2010 your good investigate the logs yourself. If you are using Outlook 2013 you will have to send the etl file to Microsoft support for research.

Below you will find an example of a log file create with Outlook 2010. Let’s analyse it:

First Outlook will determine the URL for the availability service. Outlook will use autodiscover to do determine the availability URL.

2013/09/25 11:38:40.041: Getting ASURL

As you can see Outlook already will use the URL from a previous autodiscover request:

2013/09/25 11:38:40.041: URL returned from cached autodiscover: https://mail.domain.com/ews/exchange.asmx

The URL in cache is the one that is used for requesting the availability service:

2013/09/25 11:38:40.041: Request to URL: https://mail.domain.com/ews/exchange.asmx

Next step is to send the actual request which is send as an XML to the target CAS. As you can see this XML request will be used to request the free/busy information from a specific user:

2013/09/25 11:38:40.041: Request action: http://schemas.microsoft.com/exchange/services/2006/messages/GetUserAvailability

Once the type of request has been determined the parameters will be provided in the body. You will see the following parameters to be used in every free/busy request:

  • ex12t:Timezone: the timezone which is being used: in this case 60
  • ex12t:StandardTime: the default value of the time for the request being send, I guess if no specific time is specified it will use this one
  • ex12t:Time: the standard time for which we want to know the availability: in this case 03:00:00
  • ex12t:DayOrder: the standard day (date) you want to know the availability for: in this case 5
  • ex12t:Month: the standard month: in this case 3
  • ex12t:DayOfWeek: the day for which you want to know the availability: in this case Sunday
  • ex12t:DaylightTime: I haven’t figured out this one but it looks like it has something to do with the winter/summer time
  • ex12t:Address: the smtp address of the user for which you want to know the availability: in this case johan.@johanveldhuis.nl
  • ex12Routingtype: the transport method which is used for sending the actual request once you hit the send button and your CAS will deliver it to the mailbox of the user determined in the ex12t:Address: in this case smtp
  • ex12t:AttendeeType: the type of attendee this can be either required or optional: in this case required
  • ex12t:FreeBusyViewOptions: contains the meeting specific details
  • ex12t:Starttime: the start time of the meeting: in this case 2013-09-09T18:00:00
  • ex12t:Endtime: the end time of the meeting: in this case 2013-10-09T18:00:00
  • ex12t:MergedFreeBusyIntervalInMinutes: the intervals for which you would like to display the availability by default per 30 minutes
  • ex12t:RequestedView: which info we want to see for the user: in this case detailed

request

Now the XML has been generated the actual request can be send:

2013/09/25 11:38:40.041: Sending request

Next we will receive the response which will tell us the request has been successfully send or that it failed. In this case the HTTP status code will tell us the request has been send successfully:

2013/09/25 11:38:44.135: Request sent
2013/09/25 11:38:44.135: Response error code: 00000000
2013/09/25 11:38:44.135: HTTP status code: 200

Once the request has been send we will receive an answer from the source CAS which tries to route the request to the remote CAS. It starts with some generic XML which contains the server version information. This is the version number our own CAS. In this case we can see it runs Exchange 2010 SP3.

Next information contains the actual response the source CAS received from the remote CAS. This part will start with FreeBusyResponseArray this part contains the ResponseMessage as you can see there is a security related issue. This issue will prevent Outlook from receiving the free/busy for the other user.

After the message we will see some more detailed information which will show you which request the source CAS tries to send to the remote CAS.

response

In the final part you will see the actual error message and from which CAS it did receive this error. In this case error 5016 id received from CAS02. Outlook will display the 5016 error when hovering over the free/busy info from the specific user.

Here ends the second part of the blog series about troubleshooting federated sharing. In this part we had a look at the requirements for the reverse proxy and some basic troubleshooting and continued with troubleshooting from the client side.

In the third and last part of the article we will discuss some issues I have seen happening when setting up federated sharing.

If you have any comments/questions please let me know.

 By using federated sharing it is possible to exchange free/busy informative between different Exchange organizations. This can be done by using the Microsoft Federation Gateway (MSFG) when no direct trust exists between the Active Directories. The MSFG is in this case responsible for providing a ticket which is used for authentication. By using a ticket a CAS can contact the CAS from the other organization to retrieve the free/busy information.

To use this feature several things will need to be configured:

  • trust with the Microsoft Federation Gateway
  • organizational relationship must be configured
  • autodiscover and EWS must allow WS Security authentication
  • the reverse proxy needs to allow unauthenticated traffic to the following url’s:
    • /EWS/exchange.asmx/WSSecurity
    • /autodiscover/autodiscover.svc/WSSecurity
    • /autodiscover/autodiscover.svc

Several sites contain a step-by-step plan on how to configured this. An overview of those sites can be found on the end of this article.

When you setup these things everything should work, indeed should. In most cases it will work but in some cases you will need to perform some troubleshooting. In this serie of blogs we will have a look how you can validate that it works and perform some troubleshooting in case something doesn’t work.

But how can you troubleshoot these issues? First of all it is very useful if you have a contact person who has access to the other Exchange organization. In most cases this isn’t a big issue but when using Office 365 or another form of hosting this can be very hard sometimes.

To start the troubleshooting process you mostly would like to verify your own Exchange organization first. Things that could be checked are:

  • verify if autodiscover allows WS Security for authentication
  • verify the external EWS url
  • verify if Exchange Web Services will allow WS Security for authentication

If your Exchange organization contains multiple CAS then Powershell is your friend and you can use several cmdlets to verify the steps above:

Get-AutodiscoverVirtualDirectory|select server, WSSecurityAuthentication

Get-AutodiscoverVirtualDirectory

You will get an output like above. Verify if the value of the column WSSecurityAuthentication is set to true

If WSSecurity is not true then use the following cmdlet to enable it:

Get-AutodiscoverVirtualDirectory|Set-AutodiscoverVirtualDirectory -WSSecurityAuthentication:$true

Using this cmdlet the authentication method will be configured but to offer this authentication type you will need to perform an IISReset. Additional you can verify if the svc-integrated handler is attached to the autodiscover virtual directory:

IIS

Next step is to verify the Exchange Web Services, this can be done by using the Get-WebServicesVirtualDirectory cmdlet:

Get-WebServicesVirtualDirectory|select server, ExternalUrl, WSSecurityAuthenticationcmdle

Get-WebServicesVirtualDirectory

Again the same rule apples WSSecurityAuthentication needs to be set to true. Besides this the ExternalUrl needs to contains a valid value. This url needs to accessible from the internet. If this is not the case it simply won’t work.

In case that WSSecurity is not true enabled it by using the following cmdlet:

Get-WebServicesVirtualDirectory|Set-WebServicesVirtualDirectory -WSSecurityAuthentication:$true

The same applies for EWS as it did for autodiscover perform an IISReset to activate the authentication method. Besides this verify if the ws-security handler is attached to the EWS virtual directory.

If no external url is configured you will need to configure one. Before you do this make sure the certificate contains all the correct names. If you will configure a value but it is not part of your certificate you will get errors.

To verify if the certificate contains the correct names we will use Powershell again:

Get-ExchangeCertificate|? {$_.Services -like “*IIS*”}|select Subject, CertificateDomains|FL

Verify if the CertificateDomains contain the FQDN you are planning to use for EWS, for example mail.domain.com of ews.domain.com. If this name is not on the certificate you will need to renew your certificate.

If the certificate contains the name for the external URL then you can continue configuring it:

Get-WebServicerVirtualDirectory|Set-WebServicesVirtualDirectory -Externalurl “https://ews.domain.com/EWS/exchange.asmx”

Using the cmdlet above the external URL on all Client Access Servers will be configured the same. REMARK: in some scenarios this is not what you want. Please verify if this is the case in your scenario before configuring the external url.

When this has been configured and validated it is time to verify the Federation Trust and the Organization Relationship.

This can be done by starting with Test-FederationTrustCertificate which will verify if the certificate used for the trust is correct and is installed on all CAS. During the creation of the trust the self-signed certificate will normally be distributed to all CAS in your environment. But in some cases this may not happen. If this is the case the CAS will be unable to authenticate against the Federation Gateway of Microsoft which will eventually result the autodiscover process to fail.

Test-FederationTrustCertificate

Test-FederationTrustCertificate

Verify that the State column for all CAS contain the value installed.

Additionally you can run the Test-FederationTrust cmdlet to verify if the Federation Trust really works. By default this cmdlet will use the extest account:

Test-FederationTrust

If you don’t have an extest account or you would like to use another user add the UserIdentity parameter:

Test-FederationTrust -UserIdentity user@domain.com

This cmdlet will perform several tests and will output the results on the screen, verify if no errors did occur.

As final step of the process you can verify the Organization Relationship. This is only applicable if the other organization has issues when looking up the free/busy information for your mailboxes. By using an organization relationship you will give the other organization permission to retrieve free/busy information of your organization. If not all domains are listed here it can cause strange issues such as it works for domain A but not for domain B while they are located in the same Exchange environment.

To troubleshoot these kind of issues you can use two cmdlets:

  • Get-OrganizationRelationShip,  retrieves the current configuration
  • GetFederationInformation, will send an autodiscover request to the other organization to retrieve the configuration

The following parameters are important in this case:

  • DomainNames
  • TargetApplicationUri
  • TargetAutoDiscoverEpr

One remark must be made about the DomainNames parameter. In some cases this parameter doesn’t have to be the same for both cmdlets. Some organizations only want to share free/busy information with a specific domain and not all domains hosted by the other Exchange organization.

In this part of the series we did have a look which configuration items you will need to verify in your Exchange organization. Besides this we did had a look on how to fix them if they are configured incorrectly.

In the second part we will have a closer look at the reverse proxy and client part of troubleshooting.

Below are some pages which contain use full information about federated sharing:

Office 365 Community: How to configure TMG for Office 365: open
TechNet: How does Federated Calendar sharing work in Exchange 2010?: open
TechNet: Exchange 2013: Sharing: open

Several companies have published their Exchange environment by using TMG. As you may know Microsoft has announced to discontinue the product but when working in the field you will still see that customers are using TMG.

One of the features of the Lync 2013 mobile client is to retrieve your contacts and free/busy information using Exchange Web Services (EWS). It depends on your TMG config if this will work. You may wonder why? For this we will need to have a closer look at the listener. For those who do not work much with TMG using the listener we can configure:

  • Which certificate is used to provide HTTPS
  • What kind of authentication types do we accept

So the item we need to have  closer look at is the authentication types. Using the authentication types we can configure how clients can authenticate against our Exchange environment. There are various options which you can configure for authentication amongst them:

  • HTTP form
  • Basic

Let’s assume you created one rule to publish OWA, ECP, ActiveSync, EWS and Autodiscover. In this case the listener is probably configured to offer form based authentication. Which will perfectly work for Outlook, OWA, ECP and ActiveSync. But it doesn’t work for Lync 2013 mobile clients. Normally when clients try to authenticate they will hit the form. Some clients however can’t authenticate using the form and will fall back to basic authentication. ActiveSync devices are an example of clients which work like this.

But the Lync 2013 mobile client doesn’t contain the option to fall back to basic authentication which results in authentication to fail. When you have enabled the logging on your device and examine it after trying to authenticate you will see this:

First the form will be displayed (below a small part of the code):

<em><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"></em>
<em><!-- {57A118C6-2DA9-419d-BE9A-F92B0F9A418B} --></em>
<em><html></em>
<em><head></em>
<em>    <title>Microsoft Forefront TMG</title></em>
<em>    <meta http-equiv="Content-Type" content="text/html; CHARSET=utf-8"></em>
<em>    <meta content="NOINDEX, NOFOLLOW" name="Robots"></em>
<em>    <link href="/CookieAuth.dll?GetPic?formdir=3&image=logon_style.css" type="text/css" rel="stylesheet"></em>
<em>    <script src="/CookieAuth.dll?GetPic?formdir=3&image=flogon.js" type="text/javascript"></script></em>
<em><script type="text/javascript"></em>

Followed by the following error:

2013-09-12 11:45:31.020 Lync[4120:5520] ERROR APPLICATION CEwsAutoDiscoverOperation.cpp/652:All possible ews autodiscover urls exhausted.Failing autodiscover operation

So how to solve the issue? Well the resolution is pretty simple allow basic authentication for autodiscover and EWS. This can be realized via two methods:

  • Create a new publishing rule for EWS and Autodiscover and select the option No delegation, but client may authenticate directly. Ensure the rule may be used for the All users group instead of the Authenticated users. This rule must be placed above the existing publishing rules which are used for publishing Exchange
  • Create a new web listener and new publishing rule for EWS and Autodiscover. The remark which must be made to this method is that it requires new IP-address

If you want to have more information about publishing your Exchange environment either using TMG or UAG then  good starting point is the document of Greg Taylor which contains some guidelines on how to publish your Exchange environment using TMG/UAG. This document can be found on the site below:

TechNet: Publishing Exchange Server 2010 with Forefront UAG and TMG: open

Using OWA and Internet Explorer 11

Now Windows 8.1 RTM is available via Technet and MSDN it is installed by several ITpro’s. And then it is just a matter of time befor someone finds  an issue. This because the code of Exchange doesnt recognize Internet Explorer 11 correctly and this will result in OWA to be displayed in light mode. As you may know OWA light doesn’t connect all the options which are available in the full blown OWA. Microsoft will probably fix this in a future update so you can use Internet Explorer 11 in full blown mode. This is confirmed by something I discovered when using Office 365. It looks like my tenant is already upgraded because Interne Explorer 11 works OK for me. The on-premise environments will need to wait till Microsoft releases an update which contains the correct code to recognize Internet Explorer 11.

20130821-172919.jpg

A few weeks ago the Exchange 2013 version of the Powershell Cookbook was published by Packt Publishing. This version also introduced a additional writer for the book: Jonas Andersson. A frequent Exchange blogger with quality articles so before I started reading I was convinced the book should be from high quality also. The second writer is the one who was also responsible for the first part Mike Pfeiffer. Those who missed the first part and are working with Exchange 2010, please read the book it contains very usefull info.

Enough about the writers and the first part . Let’s start reviewing the secondary part which is mostly decicated to Exchange 2013. The book starts from the beginning so if you are not familiar with Powershell the book will introduce how Powershell will work and you can benefit from it. And trust me I also found some nice tips for future use my scripts although I use Powershell almost every deployment.

Once the writers discus some basics which you really want to know they describe the abilities Powershell will give your per part of Exchange. For example they discuss which Powdershell cmdlets you can use for managing the Exchange Transport part. Per Exchange part a very clear description is given including some really nice examples.

As last and bonus part the writers describe how you can use EWS via Powershell which extends Powershell again. For those who are familiar with basic Powershell this is a must read chapter.

So what’s my conclusion about the book: a really nice book both for Exchange Powershell starters but also for people that are more experienced with Powershell for Exchange.

You can buy the book via the site below:

Packt Publishing

Exchange 2010 SP3 Rollup 2 released

Microsoft has released Exchange 2010 SP3 Rollup 2 yesterday. This rollup will fix the following issues:

  • 2837926 Error message when you try to activate a passive copy of an Exchange Server 2010 SP3 database: “File check failed”
  • 2841150 Cannot change a distribution group that contains more than 1,800 members by using ECP in OWA in an Exchange Server 2010 environment
  • 2851419 Slow performance in some databases after Exchange Server 2010 is running continuously for at least 23 days
  • 2853899 Only the first page of an S/MIME signed or encrypted message is printed by using OWA in an Exchange Server 2010 environment
  • 2854564 Messaging Records Management 2.0 policy can’t be applied in an Exchange Server 2010 environment
  • 2855083 Public Folder contents are not replicated successfully from Exchange Server 2003 or Exchange Server 2007 to Exchange Server 2010
  • 2859596 Event ID 4999 when you use a disclaimer transport rule in an environment that has Update Rollup 1 for Exchange Server 2010 SP3 installed
  • 2860037 iOS devices cannot synchronize mailboxes in an Exchange Server 2010 environment
  • 2861118 W3wp.exe process for the MSExchangeSyncAppPool application pool crashes in an Exchange Server 2010 SP2 or SP3 environment
  • 2863310 You cannot send an RTF email message that contains an embedded picture to an external recipient in an Exchange Server 2010 SP3 environment
  • 2863473 Users cannot access Outlook mailboxes that connect to a Client Access server array in an Exchange Server 2010 environment
  • 2866913 Outlook prompts to send a response to an additional update even though the response request is disabled in an Exchange Server 2010 environment
  • 2870028 EdgeTransport.exe crashes when an email message without a sender address is sent to an Exchange Server 2010 Hub Transport server
  • 2871758 EdgeTransport.exe process consumes excessive CPU resources on an Exchange Server 2010 Edge Transport server
  • 2873477 All messages are stamped by MRM if a deletion tag in a retention policy is configured in an Exchange Server 2010 environment

Especially when you are planning to migrate from Exchange 2010 from Exchange 2003 or Exchange 2007 this rollup is recommended. The rollup fixes an issue which can occur with Public Folder replication between Exchange 2003/2007 and Exchange 2010. Because replicating the public folder content is one of the steps you might see this issue if you won’t deploy Exchange 2010 SP3 rollup 2.

The second interesting thing in the release notes is a fix for the MsExchangeSyncAppPool which crashes. This issue occurs only when offering ActiveSync services to your end-users and you will use Apple devices. In this case it can happen that the CPU usage on the CAS is high (between 80% and 100%). Users might in this case not be able to sync their mailbox content because throttling is applied. Cause of this issue is the throttling mechanism. Exchange 2010 SP3 Rollup 2 contains a fix for this issue which should prevent this issue from occurring again.

Looking at the third issue you may think what? Slower performance which is experienced by users when their mailbox is hosted in a database which his created after Exchange 2010 SP3. This problem occurs only if the database is online for more then 23 days.

You can download Exchange 2010 SP3 rollup 2via the link below:

download

Deploy Exchange 2013 CU’s

This weeks the second version of the second cumulative update has been released for Exchange 2013. As you may already know Microsoft has changed his update process completely with the introduction of Exchange 2013. No more rollups and only cumulative updates (CU’s).

A pretty big change as the CU’s require you to make changes to the AD also as step of the update process. For those not familiar with rollups of Exchange 2010, the rollups where just bit updates and did not include AD changes. In Exchange 2010 the AD changes where only included in service packs.

CU’s are a complete install of Exchange 2013. So if you plan to install Exchange 2013 in the future you will only need to install the latest CU and you’re done.

But what has changed from deployment standpoint? Well pretty much because we now have to think of the Managed Availability feature of Exchange 2013. This feature will continually monitor the environment and takes action if necessary.

You can compare it with placing a node in maintenance mode in a load balancer. Although with placing a node in maintenance mode in a load balancer only ensures that it doesn’t offer any traffic anymore to the specific server.

Managed availability goes steps further, it can do this because the monitors contain knowledge about the product something load balancers don’t have.

For changing the status of the components on a server you will need to use the Set-ServerComponentState cmdlet. Using this cmdlet with the necessary parameters you can influence the components of a server.

Which components are active on a server depends on which roles are deployed to  server. To find out which components are available on a server you can run Get-ServerComponentState. In the example below you will see an output example of a multi-role Exchange 2013 server:

get-servercomponentstate

As you can see all components are up and running. Let’s assume that we have prepared our AD and are now ready to deploy the bits on our Exchange 2013 servers.

Our environment consists of two Exchange 2013 multirole servers. A third server is used as the File Share Witness (FSW) to maintain quorum.

get-servercomponentstate serverwideoffline

The first step will me to ensure that the transport queues will be empty before continuing the process. Using the Set-ServerComponentState cmdlet we can set the status of a component. Besides the status we will need to provide the requestor. In this case the requestor we are going to use is maintenance.

Set-ServerComponentState –identity ex02 –component –state draining –requestor  maintenanceget-servercomponentstate serverwideoffline

hub_draining

By using the cmdlet above we will put the transport component in draining state. This means it will take care of the messages which are currently in the queue but will not accept any new messages.

You can speed up the process by moving the messages to a transport component hosted on another server. To do this you will need to use the move-messages cmdlet:

Redirect-Messages -server ex02 –targetserver ex01.corp.local

redirect-message

To check if the queues are empty we can use the good old get-queue cmdlet:

Get-Queue –server ex02

queue

If all queues are clean we can continue with disabling all the other components which are active on the server:

Set-ServerComponentState –identity ex02 –component serverwideoffline –state inactive –requestor maintenance

set-servcomponentstat serverwideoffline restore hub

To verify if all components are set to the status inactive we could run the Get-ServerComponentState cmdlet again:

get-servercomponentstate serverwideoffline

Now all components are inactive it’s time to handle the database availability group (DAG) part.

There are a two ways to put a DAG member in maintenance mode. Those who are familiar with Exchange 2010 will remember the start-dagmaintenance script which will help you in putting s DAG member in maintenance model Well good news the script is still available and can be found in the same location as in Exchange 2010, the Exchange scripting directory.

Another method to put  DAG member in maintenance mode is to set the status of the cluster component using the Suspend-ClusterNode, for example:

Suspend-ClusterNode

suspend-clusternode

Once the cluster node has been stopped you will need to prevent the databases from being moved d on the member:

Set-MailboxServer  -Identity ex02-DatabaseCopyActivationDisabledAndMoveNow $trueSet-MailboxServer  -DatabaseCopyAutoActivationPolicy Blocked

set-mailboxserver activation

And as final step we want to set the auto activation policy temporarily to blocked:

Set-MailboxServer  -Identity ex02 -DatabaseCopyAutoActivationPolicy Blocked

Now all prerequired steps have been taken it is time to kick of the setup. In this blog we will use the cmdline to update our Exchange 2013 server. To update the server start the setup using the following parameters:

Setup.exe /m:upgrade /IacceptExchangeServerLicensing

This will start the upgrade to CU2 as you can see below:

setup.exe /m:upgrade

Once the setup has finished you might decide to reboot the server. Once the server is back online it is time to restore the services.

To restore the services we will need need to perform the steps mentioned earlier only then in reverse order.

So we will start with putting the DAG member back online:

First we will make the DAG member and active of part of the cluster again:

Resume-ClusterNode

Once this has been done we need to remove the limits for database failovers which we configured earlier:

Set-MailboxServer  -Identity ex02-DatabaseCopyActivationDisabledAndMoveNow $trueSet-MailboxServer  -DatabaseCopyAutoActivationPolicy Blocked

Set-MailboxServer  -Identity ex02 -DatabaseCopyAutoActivationPolicy Blocked

resume-clusternode

Now our DAG member is back online we need to change the component states to active:

Set-ServerComponentStatus –identity ex02 –component serverwideoffline –status active –requestor maintenance

set-servcomponentstat serverwideoffline restore

When running the Get-ServerComponentStatus cmdlet you will see that the status of the transport component is still draining.

To stop the draining process for the transport component we will need to run the Set-ServerComponentStatus cmdlet again only using a different action parameter:

Set-ServerComponentStatus –identity ex02 –component transport –status active –requestor maintenance

set-servcomponentstat serverwideoffline restore hub

Now when running the Get-ServerComponent cmdlet again the status of each component should show active.

get-servercomponentstate_fullrestore

Here ends the blog about how to install a CU on an Exchange 2013 server. If you hate to type the cmdlets manually then have a look at this script from Michael van Hoorenbeeck:

Micheals script: open

The script will perform the same cmdlets only with less typing for your and will simplify the proces of putting a server in maintenance and restore the services again.

Review ScanMail Suite for Microsoft Exchange

Exchange 2013 is available for a few months now and people may start to consider to implement it either in greenfield or in their current Exchange environment. The last one is an option which became an option with the release of Exchange 2013 CU1.

A lot has changed in Exchange 2013 the server which contains the Mailbox role does all the work and the CAS does only proxying and redirection. Redirection is only used for SIP request to the UM server.

From antivirus perspective also a lot has changed. This since Microsoft has announced that it will discontinue Forefront Protection for Exchange. So what options do we have for AV then in Exchange 2013? There are two options

  • Use the build-in AV solution from Exchange 2013
  • Use a 3rd party solution which supports Exchange 2013

The first one might be an option for organizations who can live with the limited functionality. However I think most organizations will start to use 3rd party solutions.

Starting from Exchange 2013 can’t hook in on the VSAPI which was available in previous releases and was used to perform scanning.

Currently all vendors are working hard or have already released their new versions of the product which is compatible with Exchange 2013.

Among those vendors is Trend Micro, they have just release the RTM version of ScanMail™ Suite for Microsoft® Exchange™  , also known as SMEX. You may think that it took long before they released the new version for Exchange 2013 but keep in mind Exchange 2013 is totally different then Exchange 2010 as discussed earlier.

SMEX console

With SMEX 11 Trend Micro does continue to build on previous releases of the product only then available for Exchange 2013. Starting from SMEX 11 you can’t install the product anymore on Exchange 2003. This is a logical result of the support for Exchange 2003 will end in April 2014.

Just like the previous releases SMEX will give organizations the possibility to perform both transport and mailbox scanning. It does contain several scan filters which can be used to perform scanning

As this blog would be to long too describe all of the filters in detail I have out them in the table below. In this table the scan filters are listed including a short description:

Scan Filter description
Attachment Blocking Allows you to scan for attachment types and block them
Content Filtering Allows you to verify messages for specific content and block them
Data Loss Prevention Solution to scan messages for certain content to prevent the leakage of data
Spam Prevention Rules Filter which allows you to perform basic antispam filtering
Web Reputation Filter which allows you to scan content for malicious url’s

All these methods are built on the years of experience Trend Micro has in the world of antivirus for Exchange.

Web Reputation Services

Now we know which filters are available let’s have a look at some new/enhanced features of SMEX 11. From the documentation there are 3 featured where they performed a lot of work to include these in the product.

At first the web reputation service (WRS), as already described in the table this feature will check url’s to websites in messages. For each url it sends a query to check the reputation of a url. If the url has a value which passes the threshold SMEX will perform the configured action.

Queries can be either send directly to the Smart Protection Network (a Cloud based solution) or a Smart Protection Server which is located on the local network. You can limit the Smart Protection Server to prevent queries to the Smart Protection Network but this will restrict the web reputation security level to low. The reason for this is that the Smart Protection Servers cannot maintain the complete repository of the Smart Protection Network.

New functionality introduced in WRS is the integration of the Trend Micro Command & Control (C&C) Contact Alert Services. Using this services you can benefit from the Global Intelligence list. This list is compiled by Trend Micro Smart Protection Network from sources all over the world and test and evaluates the risk level of each C&C callback address. The Web Reputation security level determines the action that needs to be taken on malicious websites or C&C servers bases on the assigned risk levels. Besides the C&C service integration it is also available on other Trend layered defense such as network, endpoint and server security. It will give you a holistic review of entire organization cyber security and targeted attack visibility.

Search and Destroy

The next feature is the search and destroy feature, I think most people will know what it does. Indeed search for items and destroy them. However this is the short story.

The search and destroy functionality is only available when SMEX is installed on an Exchange 2010 SP1 or Exchange 2013 server. When looking under the hood you will find out that SMEX does use the e-discovery functionality from Exchange for this.

So before you can use the functionality you will need to ensure that the service account you are going to use for search and destroy has the required permissions. For those who are unfamiliar with Exchange, since Exchange 2010 Microsoft has implemented Role-Based Access Control (RBAC) to specify what users can do. Exchange contains a default RBAC role called Discovery Management which is attached to the security group Exchange Discovery Management. So in this case add the service account to the group and you should be able to perform search and destroy tasks.

The search and destroy functionality uses two types of accounts which must be configured in addition to the Exchange part:

  • Search & Destroy Administrators, which can search for, monitor and delete undesirable content;
  • Search & Destroy operators, which can search for and monitor undesirable content;

At least one search & destroy administrator needs to be assigned, this since “normal” administrators won’t be able to perform the search & destroy tasks unless assigned this role.

Search & Destroy

The Search & Destroy feature has the ability to create PST files before removing the content. The content in that case will be exported to PST and stored in a folder on the local server. So if performing large search & destroy operations make sure you have enough space left on the volume where you install SMEX. This option does require the Exchange Mailbox Import Export role to be assigned to the user account.

So what happens when performing a search & destroy search:

  1. User specifies the search criteria;
  2. SMEX creates a new mailbox search in Exchange;
  3. Exchange performs the mailbox search, places the result in the discovery mailbox of Exchange and returns the results to SMEX;
  4. User can view the results and either chose to directly delete the content from the mailbox or first export it to PST;

I had a look at this specific feature of SMEX for a while during a co-existence scenario where both Exchange 2010 and Exchange 2013 were implemented. If you do plan to do this make sure to discovery mailboxes exist one for each environment at least else you won’t be able to perform searches. Despite this an additional throttling policy has to be assigned to the service account. This because Exchange will limit the amount of concurrent mailbox searches to a maximum of 2 by default.

Every step which need to be configure for search & destroy is very clear documented in the PDF which is available for download from the Trend Micro website.

Deep Discovery Advisor

New in SMEX 11 is the integration with Trend’s Deep Discovery Advisor. A new product which is currently only available as hardware appliance. Using Deep Discovery Advisor you will get a sort of virtual virus doctor in your network which offers:

  • Centralized location for aggregate, manage and analyze logs;
  • Advanced visualization and investigation tool which monitors, explores and diagnoses security events on your network;
  • Custom signature and custom defense against targeted attack.

To configuration in SMEX consists of a few steps:

  1. Enable the Advanced Threat Scan Engine;
  2. Configure the pickup directory on the Exchange Server;
  3. Specify the IP address of the appliance in SMEX;

Once configured SMEX will forward content that meets the criteria configured in the AV scanning method to the Deep Discovery Advisor. The Deep Discovery Advisor will analyze the content and will report back the results back to SMEX.

While they might have waited a bit longer than expected Trend Micro did a lot of good work which resulted in a new version of SMEX with a lot of new features.  Especially the search & destroy and enhancements in WRS are a great addition to the product. So if you decide to migrate to Exchange 2013 or have a current Exchange 2007 or Exchange 2010 environment and you are looking for an antivirus product for Exchange make sure you have a look at Scanmail.

A trail version of SMEX can be downloaded via the link below:

download